WebCodecs, AudioWorklet & Low-Latency Media Processing Studio

Prior to WebCodecs, web browsers handled media through high-level, opaque abstractions like the HTML5

While WebRTC is the traditional standard for real-time media, it incurs heavy architectural overhead: mandatory ICE/STUN/TURN connection handshakes, complex SDP offer/answer negotiations, rigid RTP/SRTP packetization, and black-box bandwidth estimation algorithms that often downgrade resolution unnecessarily. In contrast, WebCodecs + WebTransport decouples the codec pipeline from the transport layer. Developers encode raw frames into EncodedVideoChunk instances, transmit them over lightweight QUIC datagrams or unidirectional streams without RTP packet headers, and decode them immediately in a remote VideoDecoder. This architecture completely eliminates SDP negotiation, reduces client connection setup from seconds to a single TLS 1.3 round-trip, allows custom jitter buffer logic, and permits server-side frame routing without full SFU/MCU transcoding overhead.

H.264 and H.265 video bitstreams consist of Network Abstraction Layer (NAL) units containing video slice data, Sequence Parameter Sets (SPS), or Picture Parameter Sets (PPS). In raw transport streams (Annex B, RFC 6184), NAL units are delimited by 3-byte or 4-byte start codes (0x000001 or 0x00000001) and use start-code emulation prevention bytes (0x000003). In container formats like MP4 and fMP4 (AVCC / length-prefixed format), start codes are stripped, and each NALU is prefixed by a 4-byte big-endian integer denoting its exact byte length. WebCodecs VideoDecoder expects AVCC length-prefixed NAL units when configured with avc1 codec strings, and requires the SPS/PPS out-of-band in the decoder description Uint8Array (AVCDecoderConfigurationRecord). Passing raw Annex B streams directly to VideoDecoder without stripping start codes and creating the AVCC description causes silent decoding failures or immediate DOMException errors.

The browser audio engine processes audio on a high-priority, real-time operating system thread running in fixed render quanta (128 samples per block, representing 2.66ms of audio at 48kHz). If the audio thread misses its 2.66ms execution deadline, the audio buffer starves, producing audible glitches, clicks, or dropouts (buffer underrun/xrun). Passing audio chunks from the main thread or Web Worker via postMessage() involves asynchronous message queue dispatching and garbage collection (GC) allocations that introduce unpredictable 5-50ms jitter spikes. To achieve glitch-free audio, high-performance engines allocate a SharedArrayBuffer shared between the worker/network thread and the AudioWorkletGlobalScope, using a lock-free Single-Producer Single-Consumer (SPSC) circular ring buffer synchronized via Atomics.load() and Atomics.store(). This guarantees zero heap allocation and zero lock contention on the real-time audio thread.

In modern video codecs (H.264, H.265, AV1), video compression utilizes three frame types: I-frames (Intra, self-contained keyframes), P-frames (Predicted, referencing previous frames), and B-frames (Bi-directional, referencing both earlier and later frames in temporal order). Because a B-frame references a future frame, that future reference frame must be decoded before the B-frame can be reconstructed. Consequently, the Decoding Timestamp (DTS) indicates when the hardware decoder must unpack the frame, whereas the Presentation Timestamp (PTS) indicates when the rendered frame must be displayed on screen. In low-latency interactive streaming (cloud gaming, video conferencing), B-frames are disabled (set to 0) to ensure DTS equals PTS, avoiding the mandatory 1-3 frame display buffering delay inherent in bidirectional prediction.

A VideoFrame object represents an allocated video surface in CPU system memory or directly in GPU VRAM (e.g. DirectX, Metal, or Vulkan texture surfaces). Because video frames at 4K or 1080p60 consume tens to hundreds of megabytes of raw uncompressed memory per second, JavaScript garbage collection is too slow to reclaim them before GPU memory is exhausted. Developers must explicitly call frame.close() as soon as the frame has been passed to VideoEncoder.encode(), drawn to a Canvas, or consumed. Failing to call frame.close() keeps the underlying GPU resource pinned, rapidly leading to browser tab crashes with out-of-memory errors. Similarly, VideoDecoder output callbacks must immediately call frame.close() after rendering.