With this release Foxglove adds support for H.265, VP9, and AV1, extending codec support beyond H.264. We’re particularly excited about supporting all four leading codecs, both natively and directly in the browser. By embracing the strengths of each codec, Foxglove is delivering unparalleled flexibility, enabling you to choose the optimal solution for your specific needs.

H.265 doubles H.264’s compression efficiency with advanced techniques like coding tree units (CTUs), reducing redundancy and enabling efficient storage of ultra-high-definition (UHD) streams. Autonomous vehicles benefit significantly, using H.265 to analyze high-resolution camera data for navigation and decision-making.

VP9’s adaptive motion compensation efficiently encodes moving objects by analyzing motion across frames, reducing bitrate and minimizing video artifacts like blurring. Its wide browser support ensures seamless playback without plugins, making it ideal for robotics applications where clear, real-time video streaming is essential.

“We rely on high quality video. It is vital to record with high speed compression and encoding, then transcode and downsample. This enables widespread compatibility and playback across the company.” John Wyrwas, Software Director of Evaluation & Validation at Wayve

AV1, a royalty-free codec, offers exceptional compression for high-resolution and long-duration streams. With features like Constrained Directional Enhancement Filters (CDEF) and Warp Motion, AV1 is ideal for edge-based robotics in bandwidth-constrained environments, such as remote agriculture or logistics.

“It’s crucial for our operations to have quality video and low latency.” Vasily Goldobon, Senior Software Engineer at COCO

The typical video pipeline.

The typical video pipeline starts with an input source that generates raw image frames. These frames are then passed to an encoder—H.264, H.265, VP9, or AV1 in this case—which compresses the data for efficient handling. The encoder outputs Network Abstraction Layer (NAL) units for H.264 and H.265, Open Bitstream Units (OBU) for AV1, or Bitstream Units (or just Frames/Superframes) for VP9. These units form the foundation for optimized video storage and transmission, striking a balance between maintaining image quality and minimizing bandwidth usage.

What’s new with this release (and a little reminder)?

You can now choose between the codec being used—“h264”, “h265”, “vp9”, or “av1”—and define it in the format field of the foxglove.CompressedVideo message schema. Populate the data field with the appropriate units: NAL units in Annex B format for H.264 and H.265, OBUs for AV1, or Bitstream Units for VP9. Each message should contain enough packets to decode exactly one video frame.

As always, once your CompressedVideo messages are recorded into an MCAP file, the process becomes effortless. Simply drag and drop your .mcap file into the Foxglove UI, add an Image panel to your layout, and configure the panel settings to select your video topic (e.g., /videoTopic).

Foxglove’s support for CompressedVideo messages continues to extend beyond playback too. These messages can be viewed in both the Image and 3D panels. You can overlay 2D annotations, render 3D scene projections, and even download full frames via a simple right-click. Foxglove’s video support is compatible with various data sources—including local files, Foxglove WebSocket connections, and Foxglove data streaming—all of which benefit from the codec’s reduced bandwidth requirements.

Keep in mind:

• H.264 and H.265: Always use Annex B NAL units since AVCC format is unsupported. Most encoders default to Annex B or can be configured to use it.
• Message content: Each input frame processed by the encoder should result in a CompressedVideo message containing all relevant NAL, OBU or Bitstream units. For H.264 each message containing a key frame (IDR) must also include a SPS NAL unit. H.265 messages with key frames (IRAP) must include the relevant VPS/SPS/PPS NAL units. For AV1, include a Sequence Header OBU with key frames. VP9 uses the Uncompressed Frame Header within key frames to provide essential stream information.‍
• Keyframe management: Frequent keyframes (e.g., every second) simplify decoding and reduce playback latency, while infrequent keyframes (e.g., every five seconds) conserve storage but increase decoding overhead. Optimize keyframe intervals based on your application needs.‍
• Avoid B frames: To minimize latency, configure your encoder to disable frame reordering and avoid B frames. Using the “BASELINE” profile or equivalent settings ensures smoother performance in Foxglove.

By following these best practices, you can fully leverage Foxglove’s video support, achieving optimal performance and usability across your robotics development workflows.

A unified platform for robotics development.

Foxglove’s commitment to building the robotics development platform ensures that you can focus on what matters most: building systems that redefine the boundaries of what’s possible.

Pushing the boundaries of robotics innovation demands the efficient processing of visual data, a fundamental aspect of robotics development. Robotics increasingly relies on video for imitation and reinforcement learning, navigation, and teleoperation. However, handling video streams in robotics introduces distinct challenges, including encoding, compression, and visualization. These challenges are precisely where video codecs excel, enabling seamless management of high-resolution visual data while preserving quality and performance.

Whatever your use case is, join the Foxglove community to share the boundaries you’re pushing and provide feedback on Foxglove’s video capabilities. We’d love to hear your thoughts after you give it a try!

Check out the docs for full technical details and limitations →