Modern Codec Landscape

A Brief History of Video Coding Standards

Video compression standards didn't spring from nowhere. Each generation built on the last, and the choices made decades ago still constrain what's possible today.

• H.261 (1988) — The first practical video coding standard. Designed for ISDN videoconferencing at p×64 kbit/s. Introduced the hybrid block-based coding paradigm (motion-compensated prediction + DCT) that all modern codecs still use.
• MPEG-1 (1993) — Brought video to CD-ROM. Gave us the ubiquitous MP3 audio layer and grainy 352×240 video. First to standardise B-frames.
• MPEG-2 / H.262 (1995) — The DVD and digital TV standard. Supported interlaced video, higher resolutions, and scalable profiles. Still in active broadcast use today — nearly 30 years later.
• H.263 (1996) — Improved low-bitrate coding for videoconferencing. Introduced half-pixel motion compensation that became foundational.
• MPEG-4 Part 2 (1999) — The first "internet video" standard. DivX and XviD showed the world what was possible, but the standard itself was a compromise.

H.264 / AVC — The Undisputed King

H.264 (2003), also known as MPEG-4 Part 10 or AVC (Advanced Video Coding), is the most successful video codec in history. It is the default video format for Blu-ray, YouTube, Netflix, Zoom, FaceTime, and virtually every surveillance camera manufactured in the last 15 years.

What made H.264 revolutionary:

• Variable block sizes — 16×16 down to 4×4, enabling adaptive partitioning
• Multiple reference frames — Up to 16 frames for prediction (vs. just one in MPEG-2)
• Quarter-pixel motion — Much finer motion estimation than half-pixel predecessors
• CABAC entropy coding — Context-adaptive binary arithmetic coding for ~10% better compression over CAVLC
• In-loop deblocking filter — Reduced blocking artifacts within the coding loop

H.264 delivered roughly 50% bitrate reduction over MPEG-2 at the same quality. Combined with widespread hardware support in GPUs, SoCs, and FPGAs, it became the universal bridging format — the codec you can safely assume any device can decode.

H.265 / HEVC — Great Codec, Terrible Politics

H.265 (2013), or HEVC (High Efficiency Video Coding), is what happens when engineers design an almost perfect successor and patent holders proceed to ruin it. Technically, HEVC delivers roughly 50% bitrate reduction over H.264 at the same quality. But the story of its adoption is a case study in how not to standardise technology.

Key technical improvements:

• Coding Tree Units (CTUs) — Replaced fixed 16×16 macroblocks with a tree structure supporting blocks up to 64×64
• 35 intra-prediction modes — Up from 9 in H.264, providing much more accurate spatial prediction
• Sample Adaptive Offset (SAO) — Improved in-loop filtering beyond the deblocking filter
• Parallel processing tools — Tiles, wavefront parallel processing, and slice segmentation for multi-core decoding
• Improved CABAC — Higher throughput entropy coding

The result? HEVC has strong adoption in 4K Blu-ray (mandated by the spec), Apple devices (Apple pays the toll), and some streaming services for 4K content. But it never achieved the universal dominance of H.264 — and likely never will.

VP9 — Google's Royalty-Free Answer

While HEVC was floundering in patent politics, VP9 (2013) emerged from Google as a practical, royalty-free alternative. Developed from the earlier VP8 (acquired via On2 Technologies), VP9 targeted the same compression efficiency as HEVC but with a much simpler licensing story.

VP9's key advantages:

• Royalty-free — Google grants a perpetual, irrevocable patent license to anyone implementing the standard
• ~35% bitrate savings over H.264 — competitive with HEVC on most content
• Ultra HD support — 8K and beyond, with 10- and 12-bit color depth
• Hardware decode in virtually all modern GPUs and SoCs (NVIDIA, AMD, Intel, Apple, Qualcomm, Samsung)
• YouTube's primary codec — Google's massive infrastructure runs on VP9 for all non-AV1 streams

AV1 — The Alliance Strikes Back

AV1 (2018) is the codec equivalent of the Avengers assembling. Faced with the HEVC licensing catastrophe, a consortium of tech giants formed the Alliance for Open Media (AOMedia) to build a next-generation, royalty-free codec from the ground up.

The founding members read like a who's-who of the internet: Google, Apple, Netflix, Meta, Microsoft, Amazon, Mozilla, NVIDIA, Intel, AMD, Samsung, TENCENT, and ARM. When these players agree on royalty-free video, the industry listens.

AV1's technical innovations:

• Block partitioning — Up to 128×128 blocks with recursive quad-tree and binary-tree subdivision (much more flexible than HEVC)
• 56 intra-prediction modes — Including directional, smooth, and Paeth (derived from VP9)
• Warped motion — Affine motion compensation for rotations, zooms, and perspective changes
• Compound prediction — Weighted averaging from two reference frames simultaneously
• Global motion — Camera pan/zoom modelled as a single warp parameter shared across the frame
• CDEF and Loop Restoration — Advanced in-loop filters including constrained directional enhancement and wiener/SGR restoration
• Film grain synthesis — Parameterized grain model sent as metadata instead of encoding noisy grain at high bitrate

AV1 typically delivers ~30% bitrate reduction over HEVC and ~50% over H.264 at the same quality. The trade-off: encoding complexity is 3–5× higher than HEVC, making software encoding slow and hardware encoders still maturing.

AV2 — Next-Gen, Under Development

AV2 is the next generation from the Alliance for Open Media, currently under active development. Early drafts target 30% bitrate reduction over AV1 — a goal that would make AV2 roughly 60–65% more efficient than H.264.

Key experimental tools under consideration:

• Neural network-based coding tools — In-loop neural filters, learned entropy models
• Extended block geometry — Improved rectangular and non-rectangular partitioning
• Advanced motion tools — Optical-flow-based prediction, extended warped motion
• Increased transform sizes — 128×128 transforms for flat regions
• Improved palette mode — Better screen content coding (screen sharing, text overlays)

H.266 / VVC — The Technical Pinnacle

H.266 (2020), or VVC (Versatile Video Coding), represents the latest joint standard from MPEG and ITU. It achieves the highest compression efficiency of any standardized codec — roughly 30–40% better than HEVC and 60–70% better than H.264.

VVC's technical advances:

• CTU size up to 128×128 — With more flexible quadtree with nested multi-type tree (QTMT)
• 67 intra-prediction modes — Including wide-angle modes for rectangular blocks
• Multiple Transform Selection (MTS) — Choosing between DCT-II, DST-VII, DCT-VIII per block
• Low-Frequency Non-Separable Transform (LFNST) — Secondary transform for better energy compaction
• Adaptive Loop Filter (ALF) — Wiener-based filter that adapts per frame
• Decoder-side motion refinement — Subtle motion vector refinement at the decoder to reduce bitrate
• Subpixel accuracy up to 1/16 — Finest fractional motion yet standardised

EVC — Essential Video Coding

EVC (2021) is MPEG's attempt to create a simpler, more palatable alternative to VVC. It comes in two profiles:

• Baseline profile — Contains only technologies whose patents expired or were donated. Royalty-free by construction.
• Main profile — Enhances baseline with patented technologies (subject to licensing). Achieves roughly HEVC-level efficiency.

EVC's baseline profile deliberately avoids the patent thicket by restricting itself to well-established techniques (DCT, simple motion compensation, basic entropy coding). The result is a codec that's less efficient than HEVC but legally clean — a compelling option for industries that fear patent litigation.

LCEVC — Layered Enhancement

LCEVC (2021) (Low Complexity Enhancement Video Coding) takes an entirely different approach. Instead of a monolithic codec, it's a layered enhancement that sits on top of any existing codec (H.264, HEVC, AV1, etc.) to improve quality.

How it works:

• The base layer uses any existing codec to encode at a lower resolution
• The enhancement layer carries residual information (high-frequency details, corrections) using a lightweight transform
• A two-stage decoder reconstructs full resolution: decode base → apply enhancement → output

LCEVC's advantages:

• Backward compatibility — Works with existing codecs and hardware decoders
• Low complexity — The enhancement layer processing is trivially parallelisable
• Gradual quality improvement — Even partial enhancement data improves quality
• Useful for upscaling — Ideal for native-resolution encoding with AI upscaling at the client

Rate-Distortion: Seeing the Differences

The chart below shows each codec's rate-distortion curve — a measure of how much quality (PSNR) you get at a given bitrate. Higher curves mean better compression efficiency.

Codec Comparison — At a Glance

This table summarises the key characteristics of every major codec discussed in this lesson. All bitrate savings are approximate and content-dependent.

Adoption Realities — The Messy Truth

The codec standards war isn't fought on technical merit alone. Here's how things actually shake out in 2026:

Browser Support

• H.264 — 100% browser support. The universal fallback.
• HEVC — Safari and Edge (partial). Chrome dropped support on most platforms.
• VP9 — All browsers except Safari (before 16.4 added support).
• AV1 — All modern browsers now support software decode. Hardware decode varies.
• VVC — No browser support yet. Expect adoption to mirror HEVC's slow trajectory.

Hardware Support

• H.264 — Every GPU, every SoC, every phone made in the last 15 years. Hardware encoding is equally universal.
• HEVC — Broad hardware decode in Apple (A8+), NVIDIA (GTX 950+), Intel (Kaby Lake+), AMD (Vega+), Qualcomm (Snapdragon 820+).
• VP9 — Hardware decode in most GPUs since ~2016. Intel Jasper Lake+, NVIDIA Pascal+, AMD Polaris+.
• AV1 — Hardware decode in NVIDIA RTX 3050+, Intel Arc, Apple M3, Snapdragon 8 Gen 1+, MediaTek Dimensity.
• VVC — No consumer hardware decode yet. Fraunhofer's software decoder exists but is much higher complexity than AV1 software decode.

Content Availability

• YouTube — VP9 primary, AV1 for premium and high-volume streams.
• Netflix — AV1 for newer content, HEVC for Apple devices, H.264 fallback.
• Apple TV+ — HEVC exclusively (Apple pays patent tolls).
• Disney+, Max, Amazon Prime — Mostly HEVC for 4K, H.264 for 1080p. AV1 expanding.
• Blu-ray — H.264 for 1080p, HEVC for 4K. Mandated by spec.
• Broadcast TV — Still MPEG-2 and H.264. ATSC 3.0 supports HEVC.

What This Means

The codec landscape is no longer a simple chain of improvements. It's a branching tree with competing branches from different standards bodies, each with different licensing philosophies and adoption strategies.

The key takeaways:

• H.264 is the baseline — universal, decent quality, but showing its age.
• HEVC proved the industry can't standardise without solving licensing first.
• VP9 showed royalty-free was viable but couldn't break Apple's walled garden.
• AV1 is the most promising future — royalty-free, backed by industry giants, technically excellent. Encoding speed is the last hurdle.
• VVC offers the best compression but repeats HEVC's licensing mistakes — expect slow adoption.
• Fragmentation is permanent — no single codec will dominate like H.264 did. Multi-codec pipelines are the new normal.

In Lesson 5, we'll dive deep into the licensing and economics that drive these adoption decisions — and why hardware companies, not standards bodies, often decide which codecs succeed.