Media Capture and Streams API (Media Streams)

The Media Capture and Streams API, often called the Media Stream API or the Stream API,  is an API related to WebRTC which supports streams of audio or video data, the methods for working with them, the constraints associated with the type of data, the success and error callbacks when using the data asynchronously, and the events that are fired during the process.

Basic concepts

The API is based on the manipulation of a MediaStream object representing a flux of audio- or video-related data. Typically a MediaStream object is as a simple URL string which can be used to reference data stored in a DOM File, or a Blob object created with window.URL.createObjectURL(), as described in Get the video.

A MediaStream consists of zero or more MediaStreamTrack objects, representing various audio or video tracks. Each MediaStreamTrack may have one or more channels. The channel represents the smallest unit of a media stream, such as an audio signal associated with a given speaker, like left or right in a stereo audio track.

MediaStream objects have a single input and a single output. A MediaStream object generated by getUserMedia() is called local, and has as its source input one of the user's cameras or microphones. A non-local MediaStream may be representing to a media element, like <video> or <audio>, a stream originating over the network, and obtained via the WebRTC PeerConnection API, or a stream created using the Web Audio API MediaStreamAudioSourceNode.   The output of the MediaStream object is linked to a consumer. It can be a media elements, like <audio> or <video>,  the WebRTC RTCPeerConnection API or a Web Audio API MediaStreamAudioDestinationNode.

Capabilities and constraints

Historically, writing scripts for the Web that work intimately with Web APIs has had a well-known challenge: often, your code needs to know whether or not an API exists and if so, what its limitations are on the user agent it's running on. Figuring this out has often been difficult, and has usually involved looking at some combination of which user agent (or browser) you're running on, which version it is, looking to see if certain objects exist, trying to see whether various things work or not and determining what errors occur, and so forth. The result has been a lot of very fragile code, or a reliance on libraries which figure this stuff out for you, then implement polyfills to patch the holes in the implementation on your behalf.

The twin concepts of constraints and capabilies let the browser and Web site or app exchange information about what constrainable properties the browser's implementation of an interface supports and what values it will accept for each one. The interface provides methods which let scripts work with capabilities and constraints. The process works like this (using MediaStreamTrack as an example):

1. MediaDevices.getSupportedConstraints() is called to get a set of supported constraints, which lets the script determine which constrainable properties are supported by the browser.
2. Once the script knows whether the property or properties it wishes to use are supported, it can then check the capabilities of the API and its implementation by examining the object returned by the track's getCapabilities() method; this object lists each supported constraint and the values or range of values which are supported.
3. Finally, the track's applyConstraints() method is called to configure the API as desired by specifying the values or ranges of values it wishes to use for any of the constrainable properties about which it has a preference.
4. The track's getConstraints() method returns the set of constraints passed into the most recent call to applyConstraints(). This may not represent the actual current state of the track, due to properties whose requested values had to be adjusted and because platform default values aren't represented. For a complete representation of the track's current configuration, use getSettings().

In the Media Stream API, both MediaStream and MediaStreamTrack have constrainable properties.

Determining if a constraint is supported

Before using any of the constrainable properties on the MediaStreamTrack interface, you begin by calling navigator.mediaDevices.getSupportedConstraints() to get a list of the constrainable properties which the browser recognizes at all. Only properties included in the returned list should be used. You can use a pattern like this:

let supported = navigator.mediaDevices.getSupportedConstraints();

let constraints = {
  width: 640,
  height: 480
};

if (supported["frameRate"]) {
  constraints.frameRate = { ideal: 30 };
}

In this example, the supported constraints are fetched, and then the frameRate constraint is only added to the set of requested constraints when the browser supports it. That way, the requirement of a minimum 30 frames per second is used whenever possible, but otherwise the code accepts whatever it can get.

How constraints are defined

A single constraint is an object whose name matches the constrainable property whose desired value or range of values is being specified. This object contains zero or more individual constraints, as well as an optional sub-object named advanced, which contains another set of zero or more constraints  which the user agent must satisfy if at all possible. The user agent attempts to satisfy constraints in the order specified in the constraint set.

The most important thing to understand is that most constraints aren't requirements; instead, they're requests. There are exceptions, and we'll get to those shortly.

Requesting a specific value for a setting

Most simply, each constraint may be a specific value indicating a desired value for the setting. For example:

let constraints = {
  width: 1920,
  height: 1080,
  aspectRatio = 1.777777778
};

myTrack.applyConstraints(constraints);

In this case, the constraints indicate that any values are fine for nearly all properties, but that a standard high definition (HD) video size is desired, with the standard 16:9 aspect ratio. There's no guarantee that the resulting track will match any of these, but the user agent should do its best to match as many as possible.

The prioritization of the properties is simple: if two properties' requested values are mutually exclusive, then the one listed first in the constraint set will be used. As an example, if the browser running the code above couldn't provide a 1920x1080 track but could do 1920x900, then that's what would be provided.

Simple constraints like these, specifying a single value, are always treated as non-required. The user agent will try to provide what you request but will not guarantee that what you get will match. However, if you use simple values for properties when calling MediaStreamTrack.applyConstraints(), the request will always succeed, because these values will be considered a request, not a requirement.

Specifying a range of values

Sometimes, any value within a range is acceptable for a property's value. You can specify ranges with either or both minimum and maximum values, and you can event specify an ideal value within the range, if you choose.

let supports = navigator.mediaDevices.getSupportedConstraints();

if (!supports["width"] || !supports["height"] || !supports["frameRate"] || !supports["facingMode"]) {
  // We're missing needed properties, so handle that error.
} else {
  let constraints = {
    width: { min: 640, ideal: 1920 },
    height: { min: 400, ideal: 1080 },
    aspectRatio: 1.777777778,
    frameRate: { max: 30 },
    facingMode: { exact: "user" }
  };

  myTrack.applyConstraints(constraints).then(function() => {
    /* do stuff if constraints applied successfully */
  }).catch(function(reason) {
    /* failed to apply constraints; reason is why */
  });
}

Here, after ensuring that the constrainable properties for which matches must be found are supported (width, height, frameRate, and facingMode), we set up constraints which request a width no smaller than 640 (but ideally 1920), a height no smaller than 400 (but ideally 1080), an aspect ratio of 16:10 (1.777777778), and a frame rate no greater than 30 frames per second. In addition, the only acceptable input device is a camera facing the user (a "selfie cam"). If the width, height, frameRate, or facingMode constraints can't be met, the promise returned by applyConstraints() will be rejected.

Advanced constraints

So-called advanced constraints are created by adding an advanced property to the constraint set; this property's value is an array of additional constraint sets which are considered optional. There are few if any use cases for this feature, and there is some interest in removing it from the specification, so it will not be discussed here. If you wish to learn more, see section 11 of the Media Capture and Streams specification, past example 2.

Checking capabilities

You can call MediaStreamTrack.getCapabilities() to get a list of all of the supported capabilities and the values or ranges of values which each one accepts on the current platform and user agent. This function returns a MediaTrackCapabilities object which lists each constrainable property supported by the browser and a value or range of values which are supported for each one of those properties.

Applying constraints

Retrieving current constraints and settings

Reference

Browser compatibility

Currently using WebRTC for accessing the camera is supported in Chrome, Opera and Firefox Nightly 18. Enabling WebRTC in Firefox Nightly requires you to set a flag in the configuration:

• Type "about:config" in the address bar and say yes that you want to make changes
• Find the "media.navigator.enabled" entry and set it to true

See Also

• WebRTC - the introductory page to the API
• getUserMedia()
• Taking webcam photos - a tutorial on using getUserMedia()