Datasets are central to robotics and embodied AI development, serving as critical resources for training, testing, and validating algorithms that drive perception, decision-making, and control systems. However, the vast range of dataset formats creates integration challenges due to unique data structures tailored to specific applications.

MCAP offers a universal, efficient format for storing and sharing multimodal data. Its standardized structure simplifies data organization, streamlines workflows, and ensures compatibility across tools, making it easier to manage. MCAP’s seamless integration with Foxglove further enhances visualization and analysis capabilities.

In this post, we’ll demonstrate how to convert Wayve’s WayveScenes101 dataset to the MCAP format and visualize it using Foxglove.

The conversion pipeline.

Datasets are often bundles of many different formats and types of data  (e.g. images, point clouds, etc.). When converting a dataset to the MCAP format, we recommend processing each type of data into separate files before merging them. This approach keeps your code organized around types of data and simplifies debugging, as you can address issues in individual files without reprocessing the entire dataset.

For each type of data follow these steps:

1. Choose the data type and select a schema type: we’ll convert the data into messages using an existing Foxglove schema.
2. Iterate through the data: select the chosen data you want to process (e.g. images).
3. Generate an MCAP file: compile all messages of the chosen data type into a dedicated MCAP file.
4. Merge MCAP files: after generating individual MCAP files for each data type merge them into a single MCAP file.

💡 A schema defines the structure, data types, and format of a message.

Things to remember:

• Replace DATASET_PATH, SCENE, and MCAP_FILE with actual paths.
• Ensure mcap, pycolmap, and Pillow are installed.
• Use a meaningful topic name for topic="wayve/image_data" or adjust it to fit your application.

Choose the data type and match a schema type.

For this example, we'll choose the camera images data type to process and convert. Foxglove supports a variety of popular serialization formats (Protobuf, Flatbuffer, ROS, JSON) and has defined a set of schemas for well-known kinds of messages (e.g. images), all of which are available here. We’ll use Protobuf for this dataset conversion and the `CompressedImage` schema which is able to represent JPEG and PNG images.

Install Foxglove Protobuf schemas: 

pip install foxglove-schemas-protobuf

Install the `mcap-protobuf-support` package to handle logic for making an MCAP file with Protobuf serialized messages:

pip install mcap-protobuf-support

Creating the schema messages.

To implement the `getCompressedImgMsg()` function, we need to choose an appropriate schema and serialization format for storing the data in an MCAP file. To create the message object, first import the schema. Then initialize and populate it using the dataset’s fields.

Install Pillow:

pip install pillow

For Protobuf `CompressedImage` messages, the code is:

    import io
    from PIL import Image
    
    # General protobuf import syntax:
    # from foxglove_schemas_protobuf.SchemaName_pb2 import SchemaName
    
    from foxglove_schemas_protobuf.CompressedImage_pb2 import CompressedImage
    from google.protobuf.timestamp_pb2 import Timestamp
    
    def getImage(img_path: str) -> bytes:
    
        with Image.open(f"{DATASET_PATH}/{SCENE}/{IMAGES_FOLDER}/{img_path}") as img:
            img_byte_array = io.BytesIO()
            img.save(img_byte_array, format="JPEG")
            img_bytes = img_byte_array.getvalue()
        
        return img_bytes
    
    def getCompressedImgMsg(img: pycolmap.Image) -> CompressedImage:
    
    		img_path = img.name
    		timestamp_micros = int(img_path.split('/')[-1].split('.')[0])
    		seconds = timestamp_micros // int(1e6)
    		nanos = (timestamp_micros - int(1e6) * seconds) * int(1e3)
    		
    		img_msg = CompressedImage()
    	  img_msg.timestamp.CopyFrom(Timestamp(seconds=seconds, nanos=nanos))
    	  
    	  # This part depends on the schema type
    	  img_msg.frame_id = f"camera-{img.camera_id}"
    	  img_msg.data = getImage(img_path) # Image is in a different file
    	  img_msg.format = "jpg"
    	
    		return img_msg

Iterating through the data.

Select the chosen data type to be converted. Remember we're converting the camera images from WayveScenes101.

Install pycolmap:

pip install pycolmap

Load the data:

import pycolmap

# Replace with your actual dataset path
DATASET_PATH = "/path/to/wayve_scenes_101"
SCENE = "scene_001"
RECONSTRUCTION = "colmap_sparse/rig/"
IMAGES_FOLDER = "images"

# Load the reconstruction
scene = pycolmap.Reconstruction(f"{DATASET_PATH}/{SCENE}/{RECONSTRUCTION}")
# Process each image
for _, img in scene.images.items():
    img_msg = getCompressedImgMsg(img) # Create message
    print(f"Processed image {img.name}")

Writing the MCAP file.

Writing an MCAP file from a Python file follows a very similar structure regardless of the schema and serialization, though imports may vary. Below is an example using Protobuf.

Write to an MCAP file:

# Write to .mcap file
with open(MCAP_FILE, "wb") as f, Writer(f) as mcap_writer:
    for _, img in scene.images.items():
        # Generate the protobuf message
        img_msg, timestamp_ns = getCompressedImgMsg(img)

        # Write message to .mcap
        mcap_writer.write_message(
            topic="wayve/image_data",  # Use a meaningful topic name
            message=img_msg,
            log_time=timestamp_ns,
            publish_time=timestamp_ns,
        )
        print(f"Processed and wrote image: {img.name}")

print(f"Data successfully written to {MCAP_FILE}")

Repeat the above for other kinds of data, writing them to individual MCAP files.

Remember to use a meaningful topic name for topic="wayve/image_data" or adjust it to fit your application. Once you get to the visualization step below, you'll see your topic name within the Image Panel settings, just like the one in this image.

You’re almost there.

With the code in place, we can now convert the WayveScenes101 dataset into individual MCAP files. Each file contains a topic corresponding to the specific data type selected, such as images. Once processed, all images will be visualized using the designated topic name, like “wayve/image_data,” enabling seamless integration and streamlined visualization within Foxglove.

The completed data conversion pipeline code looks like this:

# WayveScenes101

# Import required libraries
import io
from PIL import Image
import pycolmap
from foxglove_schemas_protobuf.CompressedImage_pb2 import CompressedImage
from google.protobuf.timestamp_pb2 import Timestamp
from mcap_protobuf.writer import Writer

# Define paths
DATASET_PATH = "/path/to/wayve_scenes_101"
SCENE = "scene_001"
RECONSTRUCTION = "colmap_sparse/rig/"
IMAGES_FOLDER = "images"
MCAP_FILE = "wayve_scenes_101.mcap"

# Function to get image bytes
def getImage(img_path: str) -> bytes:
    with Image.open(f"{DATASET_PATH}/{SCENE}/{IMAGES_FOLDER}/{img_path}") as img:
        img_byte_array = io.BytesIO()
        img.save(img_byte_array, format="JPEG")
        img_bytes = img_byte_array.getvalue()
    return img_bytes

# Function to create a CompressedImage message
def getCompressedImgMsg(img: pycolmap.Image) -> CompressedImage:
    img_path = img.name

    # Extract timestamp from filename
    timestamp_micros = int(img_path.split('/')[-1].split('.')[0])
    seconds = timestamp_micros // int(1e6)
    nanos = (timestamp_micros - int(1e6) * seconds) * int(1e3)
    timestamp_ns = (seconds * int(1e9)) + (nanos // int(1e3))

    # Create and populate the CompressedImage message
    img_msg = CompressedImage()
    img_msg.timestamp.CopyFrom(Timestamp(seconds=seconds, nanos=nanos))
    img_msg.frame_id = f"camera-{img.camera_id}"
    img_msg.data = getImage(img_path)
    img_msg.format = "jpg"

    return img_msg, timestamp_ns

# Load the reconstruction
scene = pycolmap.Reconstruction(f"{DATASET_PATH}/{SCENE}/{RECONSTRUCTION}")

# Write to .mcap file
with open(MCAP_FILE, "wb") as f, Writer(f) as mcap_writer:
    for _, img in scene.images.items():
        # Generate the protobuf message
        img_msg, timestamp_ns = getCompressedImgMsg(img)

        # Write message to .mcap
        mcap_writer.write_message(
            topic="wayve/image_data",  # Use a meaningful topic name
            message=img_msg,
            log_time=timestamp_ns,
            publish_time=timestamp_ns,
        )
        print(f"Processed and wrote image: {img.name}")

print(f"Data successfully written to {MCAP_FILE}")

Merging the MCAP files.

Note: If you decided to process and convert multiple data types into individual MCAP files then the final step involves merging the individual MCAP files into a single unified file. If you followed this tutorial and created only one MCAP file, then you can simple skip to the visualization step next.

After generating MCAP files for different data types, open them directly in Foxglove for visualization or merge them using the MCAP CLI.

brew install mcap

We recommend merging into a single MCAP file for easier sharing and comprehensive data viewing, ensuring a more streamlined and efficient workflow.

mcap merge file1 file2 -o dataset_name.mcap

The best part: visualizing the dataset.

With your dataset now converted to MCAP, you can easily visualize it using Foxglove to get a better understanding of your data. In the Foxglove App, click on `Open Local File` and select your newly created MCAP file. Next, select the image Panel, and then lastly, the topic to display the data. You can then continue to customize your layout as you wish.

Get started with Foxglove and MCAP today! If you have any questions or need support, join our community—we’d love to hear from you and help you succeed.