Media Source Extensions API
This is an experimental technology
Because this technology's specification has not stabilized, check the compatibility table for the proper prefixes to use in various browsers. Also note that the syntax and behavior of an experimental technology is subject to change in future versions of browsers as the spec changes.
Concepts and usage
Playing video and audio has been available in web applications without plugins for a few years now, but the basic facilities offered have only really been able to cater for playing single whole tracks at a time — we can't combine/split arraybuffers. Streaming media has up until recently still been the domain of Flash, with technologies like Flash Media Server serving video streams up using the RTMP protocol.
The MSE standard
With Media Source Extensions (MSE), this is changing. MSE allows us to replace the usual single track
src value fed to media elements with a reference to a
MediaSource object, which is a container for information like the ready state of the media for being played, and references to multiple
SourceBuffer objects that represent the different chunks of media that make up the entire stream. MSE gives us finer grained control over how much and how often content is fetched, and some control over memory usage details, such as when buffers are evicted. It lays the groundwork for adaptive bitrate streaming clients (such as those using DASH or HLS) to be built upon its extensible API.
Creating assets that work with MSE in modern browsers is a laborious process, taking both significant time, computing power, and energy. The usage of external utilities to massage the content into a suitable format is required. While browser support for using the various media containers with MSE is spotty, usage of the H.264 video codec, AAC audio codec, and MP4 container format is a common baseline. MSE also provides an API for runtime detection of container and codec support.
If you do not require explicit control over video quality over time, the rate at which content is fetched, or rate at which memory is evicted, then the
<source> tags may well be a simpler, good enough solution.
Dynamic Adaptive Streaming over HTTP (DASH) is a protocol for specifying how adaptive content should be fetched. It is effectively a layer built on top of MSE for building adaptive bitrate streaming clients. While there are other such protocols available (such as HTTP Live Streaming (HLS), DASH is the most cross platform compatible), which is why we cover it in this documention.
DASH moves lots of logic out of the network protocol and into the client side application logic, using the simpler HTTP protocol to fetch files. Indeed, one can support DASH with a simple static file server, which is also great for CDNs. This is in direct contrast with previous streaming solutions that required expensive licenses for propriety non-standard client/server protocol implementations.
The two most common use cases for DASH involve watching content “on demand” or “live.” On demand allows a developer to take their time transcoding the assets into multiple resolutions of various quality.
Live profile content can introduce latency due to its transcoding and broadcasting, so DASH is not suitable for real time communication like WebRTC is — it can however support significantly more client connections than WebRTC.
Media Source Extensions Interfaces
- Represents a media source to be played via an
- Represents a chunk of media to be passed into an
- A simple container list for multiple
- Contains information about the quality of video being played by a
<video>element, such as number of dropped or corrupted frames. Returned by the
- Provides a
SourceBufferwith kind, label, and language information for tracks that do not contain this information in the initialization segments of a media chunk.
- A simple container list for multiple
Extensions to other interfaces
- Creates an object URL pointing to a
MediaSourceobject that can then be specified as the
srcvalue of an HTML media element to play a media stream.
- When a
MediaSourceobject is played by an HTML media element, this property will return a
TimeRangesobject that contains the time ranges that the user is able to seek to.
- Returns a
VideoPlaybackQualityobject for the currently played video.
- Returns the
SourceBufferthat created the track in question.
|Media Source Extensions||Candidate Recommendation||Initial definition.|
|Feature||Chrome||Firefox (Gecko)||Internet Explorer||Opera||Safari (WebKit)|
|Basic support||23||25.0 (25.0)
|Feature||Android||Firefox Mobile (Gecko)||Firefox OS (Gecko)||IE Phone||Opera Mobile||Safari Mobile|
|Not supported||11||30||Not supported|
 Available after switching the
true. In addition, support was limited to a whitelist of sites, for example YouTube, Netflix, and other popular streaming sites. The whitelist was removed and Media Source Extensions was enabled by default in 42+ for all sites.
 Only works on Windows 8+.
- Transcoding assets for Media Source Extensions
- Using MSE to create a basic streaming service (TBD)
- Using MPEG DASH to create a streaming application (TBD)
© 2016 Mozilla Contributors
Licensed under the Creative Commons Attribution-ShareAlike License v2.5 or later.