Basic concepts behind Web Audio API

Audio graphs

The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked via their inputs and outputs, forming a chain that starts with one or more sources, goes through one or more nodes, then ends up at a destination (although you don't have to provide a destination if you, say, just want to visualize some audio data.) A simple, typical workflow for web audio would look something like this:

1. Create audio context
2. Inside the context, create sources — such as <audio>, oscillator, stream
3. Create effects nodes, such as reverb, biquad filter, panner, compressor
4. Choose final destination of audio, for example your system speakers
5. Connect the sources up to the effects, and the effects to the destination.

Each input or output is composed of several channels, which represent a specific audio layout. Any discrete channel structure is supported, including mono, stereo, quad, 5.1, and so on.

Audio sources can come from a variety of places:

• Generated directly by JavaScript by an audio node (such as an oscillator.)
• Created from raw PCM data (the audio context has methods to decode supported audio formats.)
• Taken from HTML media elements (such as <video> or <audio>.)
• Taken directly from a WebRTC MediaStream (such as a webcam or microphone.)

Audio data: what's in a sample

When an audio signal is processed, sampling means the conversion of a continuous signal to a discrete signal; put another way, a continuous sound wave such as a band playing live is converted to a sequence of samples (a discrete-time signal) that allow a computer to handle the audio in distinct blocks.

A lot more information can be found at the Wikipedia page on Sampling (signal processing).

Audio buffers: frames, samples and channels

An AudioBuffer takes as its parameters a number of channels (1 for mono, 2 for stereo, etc), a length, meaning the number of sample frames inside the buffer, and a sample rate, which is the number of sample frames played per second.

A sample is a single float32 value that represents the value of the audio stream at each specific point in time, in a specific channel (left or right, if in the case of stereo). A frame, or sample frame is the set of all values for all channels that will play at a specific point in time: all the samples of all the channels that play at the same time (two for a stereo sound, six for 5.1, etc.)

The sample rate is the number of those samples (or frames, since all samples of a frame play at the same time) that will play in one second, measured in Hz. The higher the sample rate, the better the sound quality.

Let's look at a Mono and a Stereo audio buffer, each one second long, and playing at 44100Hz:

• The Mono buffer will have 44100 samples, and 44100 frames. The length property will be 44100.
• The Stereo buffer will have 88200 samples, but still 44100 frames. The length property will still be 44100 since it's equal to the number of frames.

When a buffer plays, you will hear the left most sample frame, and then the one right next to it, etc, etc. In the case of stereo, you will hear both channels at the same time. Sample frames are very useful, because they are independent of the number of channels, and represent time, in a useful way for doing precise audio manipulation.

Here's a couple of simple trivial examples:

var context = new AudioContext();
var buffer = context.createBuffer(2, 22050, 44100);

If you use this call, you will get a stereo buffer (two channels), that, when played back on an AudioContext running at 44100Hz (very common, most normal sound cards run at this rate), will last for 0.5 seconds: 22050 frames / 44100Hz = 0.5 seconds.

var context = new AudioContext();
var buffer = context.createBuffer(1, 22050, 22050);

If you use this call, you will get a mono buffer (one channel), that, when played back on an AudioContext running at 44100Hz, will be automatically *resampled* to 44100Hz (and therefore yield 44100 frames), and last for 1.0 second: 44100 frames / 44100Hz = 1 second.

Planar versus interleaved buffers

The Web Audio API uses a planar buffer format: the left and right channels are stored like this:

LLLLLLLLLLLLLLLLRRRRRRRRRRRRRRRR (for a buffer of 16 frames)

This is very common in audio processing: it makes it easy to process each channel independently.

The alternative is to use an interleaved buffer format:

LRLRLRLRLRLRLRLRLRLRLRLRLRLRLRLR (for a buffer of 16 frames)

This format is very common for storing and playing back audio without much processing, for example a decoded MP3 stream.

The Web Audio API exposes *only* planar buffers, because it's made for processing. It works with planar, but converts the audio to interleaved when it is sent to the
sound card, for playback. Conversely, when an MP3 is decoded, it starts off in interleaved format, but is converted to planar for processing.

Audio channels

Different audio buffers contain different numbers of channels, from the more basic mono (only one channel) and stereo (left and right channels) to more complex sets like quad and 5.1, which have different sound samples contained in each channel, leading to a richer sound experience. The channels are usually represented by standard abbreviations detaild in the table below:

0: L: left<br>
    1: R: right

0: L: left<br>
    1: R: right<br>
    2: SL: surround left<br>
    3: SR: surround right

0: L: left<br>
    1: R: right<br>
    2: C: center<br>
    3: LFE: subwoofer<br>
    4: SL: surround left<br>
    5: SR: surround right

Up-mixing and down-mixing

When the amount of channels doesn't match between an input and an output, up- or down-mixing happens according the following rules. This can be somewhat controlled by setting the AudioNode.channelInterpretation property to speakers or discrete.

output.L = input.M<br>
    output.R = input.M

output.L = input.M<br>
    output.R = input.M<br>
    output.SL = 0<br>
    output.SR = 0

output.L = 0<br>
    output.R = 0

output.C = input.M<br>
    output.LFE = 0<br>
    output.SL = 0<br>
    output.SR = 0

output.L = input.L<br>
    output.R = input.R<br>
    output.SL = 0<br>
    output.SR = 0

output.L = input.L<br>
    output.R = input.R<br>
    output.C = 0<br>
    output.LFE = 0<br>
    output.SL = 0<br>
    output.SR = 0

output.L = input.L<br>
    output.R = input.R<br>
    output.C = 0<br>
    output.LFE = 0<br>
    output.SL = input.SL<br>
    output.SR = input.SR

output.L = input.L + 0.7071 * (input.C + input.SL)<br>
    output.R = input.R 

output.L = input.L + 0.7071 * input.C<br>
    output.R = input.R + 0.7071 * input.C<br>
    <code>output.SL = input.SL<br>
    output.SR = input.SR

Visualizations

In general, audio visualizations are achieved by accessing an ouput of audio data over time (usually gain or frequency data), and then using a graphical technology to turn that into a visual output (such as a graph.) The Web Audio API has an AnalyserNode available that doesn't alter the audio signal passing through it, but instead outputs audio data that can be passed to a visualization technology such as <canvas>.

You can grab data using the following methods:

AnalyserNode.getFloatFrequencyData()

Copies the current frequency data into a Float32Array array passed into it.

AnalyserNode.getByteFrequencyData()

Copies the current frequency data into a Uint8Array (unsigned byte array) passed into it.

AnalyserNode.getFloatTimeDomainData()

Copies the current waveform, or time-domain, data into a Float32Array array passed into it.

AnalyserNode.getByteTimeDomainData()

Copies the current waveform, or time-domain, data into a Uint8Array (unsigned byte array) passed into it.

Spatialisations

Fan-in and Fan-out

In audio terms, fan-in describes the process by which a ChannelMergerNode takes a series of mono input sources and outputs a single multi-channel signal:

Fan-out describes the opposite process, whereby a ChannelSplitterNode takes a multi-channel input source and outputs multiple mono output signals: