Codec Parameter Tuning: H.264, H.265, and AV1 Deep Dive

Why Codec Parameters Matter

Most people export video using whatever default settings their editing software provides. The result is usually fine -- but "fine" leaves a lot of quality and file size on the table. Understanding codec parameters lets you make informed tradeoffs between visual quality, file size, encoding speed, and compatibility.

The three codecs covered here -- H.264, H.265 (HEVC), and AV1 -- represent three generations of video compression technology. Each one is roughly 30-50% more efficient than its predecessor, but they share many of the same fundamental tuning concepts.

This guide goes deep into the parameters that actually matter, with practical encoding examples you can use directly.

Core Concepts That Apply to All Three Codecs

Before diving into codec-specific parameters, there are several concepts shared across H.264, H.265, and AV1.

Rate Control Modes

Rate control determines how the encoder allocates bits across the video:

• CRF (Constant Rate Factor) -- The encoder targets a consistent perceptual quality level. File size varies depending on content complexity. This is the best mode for most offline encoding scenarios. -- CBR (Constant Bitrate) -- Every second of video gets approximately the same number of bits. Required by some streaming and broadcast systems. Wastes bits on simple scenes and starves complex ones. -- VBR (Variable Bitrate) -- The encoder varies the bitrate, spending more bits on complex scenes and fewer on simple ones. Can be constrained (capped VBR) or unconstrained. -- CQ (Constant Quality) -- Similar to CRF but used by hardware encoders. Targets a quality level rather than a bitrate. -- 2-pass encoding -- The encoder analyzes the entire video first, then encodes with optimal bit allocation. Better than single-pass VBR but takes twice as long.

Recommendation: Use CRF for local files and archived content. Use constrained VBR or 2-pass for streaming delivery where you need to hit a specific bitrate target.

I-Frames, P-Frames, and B-Frames

Video codecs compress by exploiting redundancy between frames:

• I-frames (Intra) -- Complete frames encoded independently. Large but serve as reference points. Essential for seeking. -- P-frames (Predicted) -- Encode only differences from previous reference frames. Smaller than I-frames. -- B-frames (Bidirectional) -- Encode differences from both past and future frames. Smallest but require more computation.

The ratio and arrangement of these frame types significantly affects both compression efficiency and encoding/decoding complexity.

Chroma Subsampling

Video stores color information in luma (brightness) and chroma (color) channels:

• 4:4:4 -- Full color resolution. Used in production and VFX. -- 4:2:2 -- Half horizontal chroma resolution. Common in professional intermediate codecs. -- 4:2:0 -- Quarter chroma resolution. Standard for delivery. What most people watch.

For web delivery, 4:2:0 is almost always the right choice. The quality difference is imperceptible for most content at delivery resolutions.

Bit Depth

• 8-bit -- 256 levels per channel. Standard for SDR content. -- 10-bit -- 1024 levels per channel. Required for HDR. Also improves SDR encoding by reducing banding artifacts. -- 12-bit -- 4096 levels per channel. Rare in delivery, used in high-end production.

Pro tip: Even for SDR delivery, encoding in 10-bit can produce better results than 8-bit at the same file size, because the encoder has more precision to work with internally. Both H.265 and AV1 handle 10-bit well; H.264 10-bit has limited decoder support.

H.264 (AVC) Parameter Tuning

H.264 remains the most widely supported codec. If compatibility is your top priority, this is where you focus.

CRF Values

The CRF scale for H.264 (using x264) ranges from 0 (lossless) to 51 (worst quality):

• CRF 18 -- Visually lossless for most content. Large files. Use for masters or archival. -- CRF 20-22 -- Excellent quality. The sweet spot for high-quality delivery. -- CRF 23 -- x264 default. Good quality, reasonable file size. -- CRF 24-26 -- Good quality with noticeable compression on close inspection. Fine for most web delivery. -- CRF 28-30 -- Visible compression artifacts. Acceptable for low-bandwidth scenarios. -- CRF 32+ -- Significant quality loss. Only for extreme file size constraints.

Practical example:

ffmpeg -i input.mp4 -c:v libx264 -crf 22 -preset medium -c:a aac -b:a 128k output.mp4

Presets

Presets control the encoding speed vs compression efficiency tradeoff. Slower presets try more encoding options to find better compression at the same quality:

• ultrafast -- Fastest encoding. Largest files. Poor compression efficiency. -- superfast, veryfast, faster, fast -- Progressively slower and more efficient. -- medium -- Default. Good balance for most uses. -- slow -- Noticeably better compression. Recommended when encoding time is not critical. -- slower -- Diminishing returns start here. About 5-10% smaller files than slow. -- veryslow -- Maximum practical compression. Can take 5-10x longer than medium. -- placebo -- Negligible improvement over veryslow. Not worth the time.

Real-world impact: The difference between ultrafast and veryslow at the same CRF can be 40-60% in file size. The difference between medium and slow is typically 5-15%.

Practical recommendation: Use slow for final delivery. Use fast or veryfast for previews and drafts.

Profiles and Levels

Profiles define which encoding features are available:

• Baseline -- No B-frames, no CABAC. Maximum compatibility (old devices, video calling). -- Main -- B-frames and CABAC enabled. Good compatibility with most modern devices. -- High -- Adds 8x8 transforms. Best compression. Supported by virtually all modern hardware.

Levels define the maximum resolution, frame rate, and bitrate:

• Level 3.1 -- Up to 720p60 or 1080p30 -- Level 4.0 -- Up to 1080p30 or 720p60 with higher bitrate -- Level 4.1 -- Up to 1080p60 -- Level 5.1 -- Up to 4K30 -- Level 5.2 -- Up to 4K60

Practical example for web delivery:

ffmpeg -i input.mp4 -c:v libx264 -crf 22 -preset slow -profile:v high -level 4.1 -pix_fmt yuv420p -c:a aac -b:a 128k output.mp4

Key H.264 Tuning Flags

• -tune -- Optimizes for specific content types:

  • film -- Real-world video content with natural motion
  • animation -- Animated content with flat areas and sharp edges
  • grain -- Preserves film grain instead of removing it
  • stillimage -- Slideshow-type content
  • zerolatency -- Disables features that add latency (live streaming)

• -keyint / -g -- Maximum interval between I-frames. Default is 250 frames. For streaming, set to 2x frame rate (e.g., 60 for 30fps) to enable 2-second seeking granularity.

• -bf -- Number of B-frames. Default is 3. Higher values improve compression slightly but increase decoder complexity.

• -rc-lookahead -- How many frames the encoder looks ahead for rate control decisions. Higher values improve quality consistency but increase memory usage and latency.

H.265 (HEVC) Parameter Tuning

H.265 achieves roughly 50% better compression than H.264 at equivalent quality, but with higher encoding complexity.

CRF Values

The CRF scale for H.265 (using x265) also ranges from 0-51, but the scale is different from x264:

• CRF 18-20 -- Visually lossless. High-quality masters. -- CRF 22-24 -- Excellent quality. Good for high-quality delivery. -- CRF 26-28 -- x265 default range. Good quality, efficient file sizes. The sweet spot for most web delivery. -- CRF 30-32 -- Noticeable compression. Acceptable for bandwidth-constrained scenarios. -- CRF 34+ -- Significant quality loss.

Important: A CRF 28 in x265 is NOT the same visual quality as CRF 28 in x264. At the same visual quality, x265 will produce smaller files. Roughly, x265 CRF 28 is comparable to x264 CRF 23 in terms of visual quality.

Presets

x265 presets follow the same naming as x264 but the speed differences are more dramatic:

• ultrafast -- 3-5x faster than medium, but much worse compression. -- medium -- Default. Reasonable balance. -- slow -- Recommended for production encoding. Significant compression improvement. -- slower -- Good improvement over slow, worth the time for archival or streaming masters. -- veryslow -- Diminishing returns. 10-20x slower than medium.

Critical note: H.265 encoding is inherently 2-5x slower than H.264 at equivalent presets. If x264 medium takes 1 minute, x265 medium might take 3-5 minutes for the same source.

H.265-Specific Parameters

-x265-params opens up fine-grained control:

• aq-mode -- Adaptive quantization mode. aq-mode=3 (auto-variance with bias) often produces better perceptual quality than the default. -- psy-rd -- Psychovisual rate-distortion optimization. Controls how much the encoder favors perceptual quality over mathematical accuracy. Default is 2.0. Values of 1.5-2.5 work well for most content. -- deblock -- Deblocking filter strength. Default works well for most content. Reduce for content where you want to preserve sharp edges. -- sao -- Sample Adaptive Offset filter. Reduces banding and ringing artifacts. Enable for clean content, consider disabling for grain preservation. -- rect and amp -- Enable rectangular and asymmetric motion partitions. Improve compression at the cost of encoding speed.

Practical H.265 encoding example:

ffmpeg -i input.mp4 -c:v libx265 -crf 24 -preset slow -pix_fmt yuv420p10le -x265-params "aq-mode=3:psy-rd=2.0" -c:a aac -b:a 128k output.mp4

HDR Encoding with H.265

H.265 is currently the most practical codec for HDR delivery:

ffmpeg -i input.mp4 -c:v libx265 -crf 22 -preset slow -pix_fmt yuv420p10le \
  -x265-params "hdr-opt=1:repeat-headers=1:colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1):max-cll=1000,400" \
  -c:a aac -b:a 128k output_hdr.mp4

This example sets up HDR10 metadata with PQ transfer function, BT.2020 color space, and specific mastering display characteristics.

AV1 Parameter Tuning

AV1 is the newest of the three and offers the best compression, but encoding speed is a significant consideration.

Encoder Options

Unlike H.264 and H.265, which each have one dominant software encoder, AV1 has several:

• libaom-av1 -- The reference encoder. Slowest but most feature-complete. Used for quality benchmarking. -- SVT-AV1 -- Developed by Intel and Netflix. Much faster than libaom. The recommended choice for most practical encoding. -- rav1e -- Written in Rust. Fast development, good quality, but not as widely used.

For practical work, SVT-AV1 is the clear winner in the speed/quality tradeoff.

CRF Values (SVT-AV1)

SVT-AV1 uses a CRF scale from 0-63:

• CRF 18-22 -- Visually lossless. Large files. -- CRF 24-28 -- Excellent quality. High-quality delivery. -- CRF 30-35 -- Good quality. The sweet spot for web delivery. Comparable to H.265 CRF 24-26 in visual quality at significantly smaller file sizes. -- CRF 38-42 -- Acceptable quality for bandwidth-constrained scenarios. -- CRF 45+ -- Significant quality loss.

Speed Presets (SVT-AV1)

SVT-AV1 uses numeric presets from 0 (slowest, best compression) to 13 (fastest, worst compression):

• Preset 0-2 -- Research quality. Extremely slow. Not practical for production. -- Preset 3-4 -- High quality. Very slow but practical for archival or streaming masters. -- Preset 5-6 -- Good quality. Reasonable encoding speed. Recommended for most production work. -- Preset 7-8 -- Fast encoding. Good enough quality for most web delivery. -- Preset 9-10 -- Very fast. Some quality sacrifices visible on close inspection. -- Preset 11-13 -- Fastest options. Useful for previews and real-time applications.

Practical AV1 encoding example:

ffmpeg -i input.mp4 -c:v libsvtav1 -crf 30 -preset 6 -pix_fmt yuv420p10le -svtav1-params "tune=0" -c:a libopus -b:a 128k output.webm

Key SVT-AV1 Parameters

• tune -- 0 for visual quality (PSNR + perceptual), 1 for pure PSNR. Use 0 for content delivery. -- film-grain -- Synthesized film grain level (0-50). Instead of wasting bits encoding grain, the encoder removes it and instructs the decoder to re-add it. Massive bitrate savings for grainy content. Try values of 8-15 for light grain, 20-30 for heavy grain. -- film-grain-denoise -- Set to 0 to apply film grain synthesis without denoising the source. Set to 1 (default) to denoise and then re-synthesize. -- keyint -- Keyframe interval in frames. Similar to H.264/H.265 -g parameter. -- enable-overlays -- Enables overlay frames for better quality. Set to 1.

Film Grain Synthesis -- AV1's Secret Weapon

Film grain synthesis is one of AV1's most powerful features. Natural film grain and sensor noise are extremely expensive to encode because they are random and high-frequency. AV1 can:

1. Analyze and remove grain from the source
2. Encode the clean video at much lower bitrate
3. Embed grain parameters in the bitstream
4. The decoder re-synthesizes matching grain during playback

The bitrate savings can be 20-40% for grainy content with no perceived quality loss. This is unique to AV1 and not available in H.264 or H.265.

ffmpeg -i grainy_input.mp4 -c:v libsvtav1 -crf 30 -preset 6 -svtav1-params "film-grain=15:film-grain-denoise=1" -c:a libopus -b:a 128k output.webm

Practical Encoding Recipes

Recipe 1: High-Quality Archive (H.264)

Maximum compatibility, excellent quality, reasonable file size:

ffmpeg -i input.mp4 \
  -c:v libx264 -crf 18 -preset slow -profile:v high \
  -pix_fmt yuv420p \
  -c:a aac -b:a 192k \
  archive.mp4

Recipe 2: Web Delivery (H.264)

Broad compatibility, good quality, optimized file size:

ffmpeg -i input.mp4 \
  -c:v libx264 -crf 23 -preset slow -profile:v high -level 4.1 \
  -pix_fmt yuv420p -movflags +faststart \
  -c:a aac -b:a 128k \
  web_h264.mp4

The -movflags +faststart flag moves the MP4 metadata to the beginning of the file, enabling progressive playback in browsers without downloading the entire file first.

Recipe 3: Efficient Web Delivery (H.265)

Better compression for modern Apple devices and supporting browsers:

ffmpeg -i input.mp4 \
  -c:v libx265 -crf 26 -preset slow \
  -pix_fmt yuv420p10le \
  -x265-params "aq-mode=3" \
  -tag:v hvc1 \
  -c:a aac -b:a 128k \
  web_h265.mp4

The -tag:v hvc1 flag ensures Safari and Apple devices can play the file. Without it, some Apple software will reject the file even though the codec is supported.

Recipe 4: Maximum Efficiency Web Delivery (AV1)

Best compression, modern browser support:

ffmpeg -i input.mp4 \
  -c:v libsvtav1 -crf 30 -preset 6 \
  -pix_fmt yuv420p10le \
  -svtav1-params "tune=0:enable-overlays=1" \
  -c:a libopus -b:a 128k \
  web_av1.webm

Recipe 5: Streaming-Ready with Bitrate Cap (H.264)

For platforms requiring predictable bitrate:

ffmpeg -i input.mp4 \
  -c:v libx264 -crf 23 -preset slow -profile:v high \
  -maxrate 5M -bufsize 10M \
  -pix_fmt yuv420p -movflags +faststart \
  -g 60 -keyint_min 60 \
  -c:a aac -b:a 128k \
  streaming.mp4

This uses CRF with a maximum bitrate cap of 5 Mbps and a VBV buffer of 10 Mbps, ensuring the bitrate never spikes above the cap while still using CRF-based quality targeting.

Recipe 6: Grainy/Noisy Content (AV1)

Leveraging film grain synthesis for cinematic content:

ffmpeg -i grainy_film.mp4 \
  -c:v libsvtav1 -crf 28 -preset 5 \
  -pix_fmt yuv420p10le \
  -svtav1-params "tune=0:film-grain=20:film-grain-denoise=1:enable-overlays=1" \
  -c:a libopus -b:a 128k \
  film_av1.webm

Comparing the Three Codecs

At equivalent visual quality (using each codec's sweet spot settings):

File size comparison (relative to H.264 = 100%):

• H.264 (x264, CRF 22, slow) -- 100% (baseline) -- H.265 (x265, CRF 26, slow) -- ~55-65% of H.264 -- AV1 (SVT-AV1, CRF 30, preset 6) -- ~40-55% of H.264

Encoding time comparison (relative to H.264 = 1x):

• H.264 (x264, slow) -- 1x (baseline) -- H.265 (x265, slow) -- 3-5x slower -- AV1 (SVT-AV1, preset 6) -- 5-10x slower

Decoding compatibility:

• H.264 -- Universal. Every device made in the last 15 years. -- H.265 -- Wide but not universal. Safari, iOS, most Android, some browsers. -- AV1 -- Growing. Chrome, Firefox, Edge, Safari 17+, newer phones and GPUs.

Tips for Efficient Encoding Workflows

Use Vibbit for Streamlined Encoding

When you are managing multiple videos or need consistent quality across a library of content, tools like Vibbit can automate the codec selection and parameter tuning process, saving significant time compared to hand-tuning every export.

Test Before Committing

Before encoding an entire project, encode a representative 30-60 second sample at your chosen settings. Compare quality visually and check the file size. Adjust CRF or preset as needed before running the full encode.

Parallelize When Possible

Both x265 and SVT-AV1 support multi-threaded encoding, but they may not saturate all CPU cores at higher presets. For batch encoding, running multiple encodes in parallel can be more efficient than running one encode at a time.

Hardware Encoding as a Complement

Hardware encoders (NVENC, QSV, AMF) are dramatically faster but produce larger files at equivalent quality. They are excellent for:

• Live streaming where speed is critical -- Quick previews during editing -- First-pass encoding in a two-step workflow

For final delivery where quality and file size matter most, software encoders (x264, x265, SVT-AV1) remain superior.

Monitor Your Results

Use tools like FFprobe to verify your output:

ffprobe -v quiet -print_format json -show_format -show_streams output.mp4

Check the actual bitrate, resolution, pixel format, and codec profile to ensure they match your intentions.

Understanding codec parameters is not about memorizing every flag -- it is about knowing which parameters have the biggest impact and how to adjust them for your specific content and delivery requirements. The recipes in this guide cover the most common scenarios, and the parameter explanations give you the knowledge to adapt them when needed.