Datasets play a pivotal role in advancing robotics and embodied AI, providing essential material for developing, evaluating, and refining algorithms used in perception, decision-making, and control. Yet, the diversity in dataset formats poses significant challenges, as many are designed with unique structures suited to specialized applications, making seamless integration difficult.

MCAP offers a universal, efficient format for storing and sharing multimodal data. Its standardized structure simplifies data organization, streamlines workflows, and ensures compatibility across tools, making it easier to manage. MCAP’s seamless integration with Foxglove further enhances visualization and analysis capabilities.

In this post, we’ll demonstrate how to convert the Waymo Open dataset to the MCAP format and visualize it using Foxglove. 

The conversion pipeline.

Datasets frequently consist of various data types and formats, such as images and point clouds. To streamline the process of converting a dataset to the MCAP format, it’s advisable to handle each data type separately, creating distinct files before combining them. This method helps maintain organized code focused on specific data types and makes troubleshooting more efficient, as issues can be resolved within individual files without needing to reprocess the entire dataset.

For each type of data follow these steps:

1. Select a data type: choose the type of data you want to process (e.g.: images)
2. Create schema messages: convert the data into messages using an existing Foxglove schema.
3. Generate the MCAP file: compile all messages of the chosen data type into dedicated MCAP files. After generating individual MCAP files for each data type merge them into a single MCAP file.

💡 A schema defines the structure, data types, and format of a message.

Things to remember:

• Replace DATASET_PATH with your actual path.
• Ensure mcap is installed.
• Use a meaningful topic name for topic="waymo/image_data".

Choose a type and iterate through the data.

Start by selecting the type of data to be converted. For this example, we will focus on converting the camera images from the Waymo Open dataset. Let’s iterate through each image and convert them step by step.

Begin by loading the data in Python:

# Waymo Open Dataset

import pandas as pd

DATASET_PATH = "path/to/waymo-open-dataset"
FOLDER = "camera_image"
SEGMENT = "8331804655557290264_4351_740_4371_740"
# This segment is an example from the validation dataset

segment = pd.read_parquet(
							f"{DATASET_PATH}/{FOLDER}/{SEGMENT_NAME}.parquet",
							engine='pyarrow', # pyarrow needs to be installed via pip
					)
			
for _, row in segment.iterrows():

		img_msg = getConvertedMsg(row) # Create message

To implement the getConvertedMsg() function , we need to choose an appropriate Schema and serialization format for storing our data in an MCAP file.

Install the `mcap-protobuf-support` package to handle logic for making an MCAP file with Protobuf serialized messages:

pip install mcap-protobuf-support

We’ll use the CompressedImage schema which is able to represent JPEG and PNG images.

Creating the schema messages.

You can initialize and populate the schema using the dataset’s fields.

For Protobuf `CompressedImage` messages, the code is:

# Waymo Open Dataset

import pandas as pd

# General protobuf import syntax:
# from foxglove_schemas_protobuf.SchemaName_pb2 import SchemaName

from foxglove_schemas_protobuf.CompressedImage_pb2 import CompressedImage
from google.protobuf.timestamp_pb2 import Timestamp

def getConvertedMsg(row: pd.Series) -> CompressedImage:

		camera_name = row['key.camera_name']
		timestamp_micros = row['key.frame_timestamp_micros']
		seconds = timestamp_micros // int(1e6)
		nanos = (timestamp_micros - int(1e6) * seconds) * int(1e3)
		
		img_msg = CompressedImage()
		img_msg.timestamp.CopyFrom(Timestamp(seconds=seconds, nanos=nanos))
		
		# This part depends on the schema type
		img_msg.frame_id = f"camera_{camera_name}"
		img_msg.data = row['[CameraImageComponent].image']
		img_msg.format = "jpg"
		
		return img_msg

Writing the MCAP file.

Writing an MCAP file from a Python file follows a very similar structure regardless of the schema and serialization, though imports may vary. Below is an example using Protobuf. 

Write to an MCAP file:

from mcap_protobuf.writer import Writer

with open(filename, "wb") as f, Writer(f) as mcap_writer:
		
		# Generate the messages here
		
		mcap_writer.write_message(
						topic = waymo/image_data,              # string
						message = protobuf_message,      # protobuf object
						log_time = timestamp_ns,         # integer timestamp (in ns)
						publish_time = timestamp_ns,     # integer timestamp (in ns)
				)

Repeat the above for other kinds of data, writing them to individual MCAP files.

Remember to use a meaningful topic name for topic="waymo/image_data". Once you get to the visualization step (below), you'll see your topic name within the Image Panel settings.

Putting it all together.

With the necessary code implemented, we can transform the Waymo Open dataset into separate MCAP files. Each file is organized by a topic that matches the chosen data type, such as images. After processing, the images will be displayed using their assigned topic name, such as “waymo/image_data,” ensuring smooth integration and efficient visualization through Foxglove.

The complete conversion pipeline code looks like this:

# Waymo Open Dataset to MCAP Conversion

import pandas as pd
from foxglove_schemas_protobuf.CompressedImage_pb2 import CompressedImage
from google.protobuf.timestamp_pb2 import Timestamp
from mcap_protobuf.writer import Writer

# Define paths and dataset information
DATASET_PATH = "path/to/waymo-open-dataset"
FOLDER = "camera_image"
SEGMENT_NAME = "8331804655557290264_4351_740_4371_740"  # Example from validation dataset
OUTPUT_MCAP_FILE = "waymo_dataset.mcap"

# Function to create a CompressedImage message
def getConvertedMsg(row: pd.Series) -> CompressedImage:
    camera_name = row['key.camera_name']
    timestamp_micros = row['key.frame_timestamp_micros']
    seconds = timestamp_micros // int(1e6)
    nanos = (timestamp_micros - int(1e6) * seconds) * int(1e3)
    timestamp_ns = (seconds * int(1e9)) + (nanos // int(1e3))

    # Create and populate the CompressedImage message
    img_msg = CompressedImage()
    img_msg.timestamp.CopyFrom(Timestamp(seconds=seconds, nanos=nanos))
    img_msg.frame_id = f"camera_{camera_name}"
    img_msg.data = row['[CameraImageComponent].image']
    img_msg.format = "jpg"

    return img_msg, timestamp_ns

# Load the dataset segment
segment = pd.read_parquet(
    f"{DATASET_PATH}/{FOLDER}/{SEGMENT_NAME}.parquet",
    engine='pyarrow',  # Ensure pyarrow is installed via pip
)

# Write data to an MCAP file
with open(OUTPUT_MCAP_FILE, "wb") as f, Writer(f) as mcap_writer:
    for _, row in segment.iterrows():
        # Generate the protobuf message
        img_msg, timestamp_ns = getConvertedMsg(row)

        # Write message to the MCAP file
        mcap_writer.write_message(
            topic="waymo/image_data",  # Use a meaningful topic name
            message=img_msg,
            log_time=timestamp_ns,
            publish_time=timestamp_ns,
        )
        print(f"Processed and wrote image for camera: {row['key.camera_name']}")

print(f"Conversion completed. Data written to {OUTPUT_MCAP_FILE}")

Merging the MCAP files.

At this stage, you’ve likely created a single MCAP file. However, if you’ve iterated through your data and created multiple MCAP files, the final step is to merge these individual files into a single unified MCAP file. This consolidation simplifies data management, making it easier to analyze and share your data in a cohesive format. After generating MCAP files for different data types, open them directly in Foxglove for visualization or merge them using the MCAP CLI.

If you don’t already have MCAP installing simply run the following command:

brew install mcap

And run the following:

mcap merge file1 file2 -o dataset_name.mcap

The best part: visualizing the dataset.

With your dataset now converted to MCAP, you can easily visualize it using Foxglove to get a better understanding of your data. In the Foxglove App, click on `Open Local File` and select your newly created MCAP file. Next, select the image Panel and lastly the topic to display the data. You can then customize your layout as you wish.

Get started with Foxglove and MCAP today! If you have any questions or need support, join our community—we’d love to hear from you and help you succeed.