As a scalable data management service, Foxglove Data Platform is a convenient one-stop shop for storing and analyzing your robotics data. You can streamline your data management process – usually an extremely time-consuming and expensive part of development – and reserve your team’s focus for less routine parts of your workflows.

Foxglove Data Platform currently supports importing and exporting ROS 1 (.bag) files and MCAP (.mcap) files. Due to upstream ROS 2 limitations, Data Platform cannot meaningfully store ROS 2 (.db3) files, as they do not contain the message definitions necessary for proper introspection and analysis.

This isn’t to say Foxglove Data Platform is off-limits for your ROS 2 team – there are several ways you can convert your .db3 files into self-contained MCAP files.

The easiest way to achieve this is to install our rosbag2\_storage\_mcap plugin, and have your robots record MCAP files directly. But if you want to stick with your existing recording plugin, or have already-recorded ROS 2 data waiting to be analyzed, continue along with this tutorial to convert your ROS 2 data into MCAP files for proper storage and analysis.

Install the mcap CLI

For a user-friendly way to convert your ROS 2 files, download the mcap CLI tool for your platform – we currently support Linux, Windows, and macOS.

Check out the releases page to download the latest release binary, or install directly via your command line using go:



Point to your custom message definitions

Next, you’ll need to specify to mcap where you store your ROS 2 message definitions.

Let’s say your msg files live in the /opt/ros/galactic directory. You’ll want to open up your setup bash file (i.e. /opt/ros/galactic/setup.bash) to set an environment variable called AMENT\_PREFIX\_PATH to your top-level ROS directory:



mcap will now know that it needs to search in /opt/ros/galactic to locate the ROS 2 message definitions on your hard drive, and to include those definitions when outputting your converted MCAP files.

Converting to MCAP

You’re now ready to convert your ROS 2 files into MCAP files!

Run the following command in your terminal to generate a MCAP file (e.g. my\_output.mcap) from a local ROS 2 file on your computer (e.g. your\_input.db3):



If you didn’t properly the AMENT\_PREFIX\_PATH environment variable in the previous step, you can even pass in your message definition directory paths as a command line flag:



If your message definitions live across multiple directories, use a colon-separated list of paths:



You should now be able to see a MCAP file output to your computer!

Run $ mcap info my\_output.mcap to see what your shiny new file actually contains – you should see a print out of your data's message count, time range, channels list, and other relevant information:



Congratulations on your first successful ROS 2 to MCAP conversion!

Uploading to Foxglove Data Platform

Now that you have your data in a format that Foxglove Data Platform understands, let’s get it imported so you can start tagging and analyzing your data with the rest of your team.

There are two ways to upload your data – through the Foxglove web console or with the foxglove-cli tool.

Web console

Create an account or sign in to your Foxglove account at console.foxglove.dev to view your dashboard.

Before uploading your MCAP to Foxglove Data Platform, you must add the device that your data should be associated with – i.e. the robot that your data was recorded on. If you haven’t yet created the necessary device, click “Add device” at the top of the Devices page to create a uniquely named robot for your team to track.

Foxglove Data Platform Devices view

Next, click "Import data" on the Files page to start uploading your my\_output.mcap file – be sure to associate it with your created device:

Foxglove Data Platform Files view

You should now be able to see your import on both the Files and Timeline pages! To see your data in the Timeline view, make sure you use the date picker to navigate to the time range in which your data was recorded:

Foxglove Data Platform Timeline view

foxglove-cli

Import your data with foxglove-cli if you want to write a script to automate or batch your imports. Install the latest release for your operating system and architecture, or install directly from source.

As with the web console, you must first authenticate yourself and add a device corresponding to your data:



To upload your output MCAP file, run the following command:



Alternatively, you can use the following shorthand to achieve the same results:



To verify that your import succeeded, print out a list of all your imports:



You should be able to locate an import with a device ID and filename that correspond to yours (i.e. dev\_drpLqjBZYUzus3gv and my\_output.mcap, respectively).

Continue to explore your data

Now that you’ve imported your ROS 2 data, you can leverage Foxglove Data Platform to better organize and analyze your team's data.

Attach rich metadata to your imported data, tag events of interest for your teammates, and submit hyper-specific queries to your data lake. Retrieve the information you need at lightning speed, with Data Platform's strategic data partitioning and indexing. You can even click a single button to stream a specific snippet of data into Foxglove Studio, for deeper visualization and debugging.

In short, this tutorial is just the start. Join us in our Slack channel if you have questions about getting started, or book a demo with our team to learn more about our active (and planned!) features.


In my spare time for the past several months, I’ve been developing an autonomous RC car for DIYRobocars – a community of makers who race their DIY autonomous cars of all sizes. At the Robot Block Party hosted by Silicon Valley Robotics this past April, my car came in second in the Robocar race and also won the demolition derby.

My robocar taking out the competition at the demolition derby.

As you might expect, a good data logging and visualization infrastructure was crucial to helping me develop my successful autonomous driver. It took several iterations and a few false starts before I landed on my final solution – using Foxglove’s MCAP C++ reader-writer with Protobuf messages. I found the MCAP writer to be a solid foundation for my logging architecture, affording me a lightweight solution for both out-of-the-box support and open-ended flexibility.

Initial experiments with Python and ROS

I decided against using ROS in my robocar stack for a variety of reasons. For one, I wanted to write most of the core software from scratch – it’s part of the fun! I also wanted to get started quickly, and I decided that the time I’d invest in learning the ecosystem probably would not pay off for me in my future projects.

Furthermore, it wasn’t feasible to simply import the rosbag writer into my existing C++ build. Getting ROS working with my C++ build would likely have been a nightmare, given my desire to both cross-compile and run Raspbian for the Raspberry Pi (instead of Ubuntu).

In my first foray into logging and visualization, I used a custom binary format to log video frames and CSV lines to log sensor data, PID control outputs, and all other data. I then wrote a Python script, leveraging Python’s dependency management and standalone rosbag writer, to export ROS 2 files that could then be read by Foxglove Studio.

This scheme quickly fell apart. Schema evolution for logged messages and topics was manual, tedious, and error-prone. Older log files quickly became unreadable. The additional Python conversion step also made things more complicated and annoying.

With this first failed experiment, I tried introducing minimal dependencies on sqlite and FastCDR, which let me cobble together my own rosbag2 writer. This let me natively log rosbag2 files which could then be viewed directly in Foxglove Studio.

However, there still remained a few pain points. With this direction, I was maintaining a significant amount of code – the rosbag2 writer and especially reader – that didn’t add much core value to my autonomous car. Since FastCDR’s IDL is not widely adopted, I had to write my own definitions for the standard ROS 2 messages I wanted to publish. Foxglove Studio also could not support custom ROS 2 messages, due to an inherent ROS 2 design decision. Finally, there was no easy way to do compression, which inevitably led to unwieldy data logs and longer data offloading times.

Using MCAP with Protobuf messages

Around this time, Foxglove released an early version of their MCAP reader and writer, with C++ and Python support (they currently also support TypeScript, Go, and Swift). Given the hurdles I’d encountered with my previous iterations, I decided to give the library a try, using custom Protobuf messages.

I immediately noticed a few advantages. For one, Protobuf is well documented, with robust C++ libraries and backwards- and forwards-compatible semantics for message schema evolution. I also found myself enjoying just how easy it was to integrate MCAP into my stack. It is a small library with relatively few dependencies, yet surprisingly flexible. I was able to leverage Foxglove’s already-defined protobuf message schemas for standard ROS messages, but also define my own custom messages. I also found the ability to tune compression settings incredibly useful for reducing the size of my data logs and for accelerating data offloading.

The only con I found was that with MCAP being fairly new, there hasn’t been much widespread adoption yet. With that said, I think anyone else who gives the library a try will have as positive of an experience as I had.

Creating an MCAP writer in C++

An MCAP file has messages, channels, and schemas.

In the same way that ROS publishes messages to a topic, MCAP writes messages to a channel. Messages for a given channel always follow the same schema, but a particular schema can be shared by multiple channels.

Creating an MCAP writer is relatively straightforward:



Configure your writer to your desired specifications using McapWriterOptions. For example, opts.compressionLevel = mcap::CompressionLevel::Fast customizes your writer to use a faster compression level.

Before we can write messages, we need to register a schema and a channel to write our messages to.

To register a schema in Protobuf, you must use the fully-qualified name of the message type (e.g. ros.nav\_msgs.Path) and provide a serialized google::protobuf::FileDescriptorSet for the schema itself. Generated Protobuf messages will contain enough information to reconstruct this FileDescriptorSet schema at runtime.



Once we have the schema registered, we can register our channel:



We can now finally write messages to the channel via its id:



Don’t forget to close the writer when you’re done:



Now, we can inspect our output MCAP file's messages. I used the Data source dialog in Foxglove Studio to “Open local file”:

Data source dialog

I then added a few relevant panels (Plot, Image, Raw Messages, 3D) to visualize my robocar's performance on the "road".

Foxglove Studio replaying the exact moment my robocar collided with a demolition derby competitor.

Conclusion

I found Foxglove’s MCAP writer to be a solid building block when building my robocar’s data logging system. I used it in tandem with Foxglove Studio to quickly publish, visualize, and iterate on my robot – all by getting a better understanding of the data it recorded.

I was even able to add several bespoke enhancements specific to my project. I wrapped the MCAP writer in a layer that serialized Protobuf messages, wrapped them in a mcap::Message, and registered schemas for unknown Protobuf types in one centralized place. I also moved mcap.write to a separate thread fed by a queue, which avoided the time-sensitive driver control loop being disturbed by the long-tail filesystem i/o latency observed on the Raspberry Pi 3.

If you find yourself unhappy with your current data logging solution, I encourage you to give MCAP a try – it certainly has turned out well for me!


While Foxglove Studio was originally designed for visualizing ROS data, we are on a mission to bring world-class visualization to all robots, including those with custom software frameworks.

Today, we're excited to announce Foxglove schemas: an entirely Foxglove-native way to represent and visualize data. Foxglove schemas are available for Protobuf, JSON, ROS 1, and ROS 2, and we plan to add support for more formats over time.

Becoming more framework-agnostic

We announced the MCAP log file format earlier this year, as an important milestone towards our goal of developing framework-agnostic robotics tools. However, even with MCAP, Foxglove Studio still required the use of existing ROS schemas such as sensor\_msgs/Image or visualization\_msgs/Marker to visualize certain data.

Our new Foxglove schemas have no dependency on ROS, and can be used from any framework. We also plan to continue fully supporting the existing ROS schemas as we do today.

Foxglove Studio panels

Studio’s Image panel now supports foxglove.CompressedImage in addition to ROS’s sensor\_msgs/Image. The 3D panel now supports foxglove.PointCloud in addition to ROS’s sensor\_msgs/PointCloud2.

To see a full list of our supported schemas, as well as the Foxglove Studio panels that use them, check out our schemas documentation.

Start writing your messages

To start publishing messages that adhere to the new Foxglove schemas, import the desired schema files for your framework.

Protobuf

Copy the .proto files directly from the foxglove/schemas repo into your project. You can use these imported schemas to publish messages via a live Foxglove WebSocket connection or to log data to an MCAP file.

We provide WebSocket libraries for live data (Python, JavaScript, C++), as well as MCAP writers for pre-recorded data files (Python, JavaScript, C++).

Check out our blog post on Recording Robocar Data with MCAP for an example on using an MCAP C++ writer to record your Protobuf data.

JSON Schema

Copy the .json files directly from the foxglove/schemas repo into your project, or import them from the @foxglove/schemas npm package:



Use these imported schemas to publish messages via a live Foxglove WebSocket connection or to log data to an MCAP file. We provide WebSocket libraries for live data (Python, JavaScript, C++), as well as MCAP writers for pre-recorded data files (Python, JavaScript, C++).

ROS 1 & 2

We’re in the process of publishing ROS packages for Foxglove schemas. After the next ROS sync, it will be as simple as running sudo apt install ros-humble-foxglove-msgs (replacing humble with your preferred ROS distribution).

In the meantime, you can copy .msg files directly from the ros1/ or ros2/ folder in foxglove/schemas into your ROS project, then importing these schemas wherever you want to start publishing messages:



TypeScript (bonus)

We also provide type definitions for Foxglove schemas for use in Foxglove Studio’s Node Playground panel, or your own TypeScript project.

For Node Playground, you can specify the schema you want to use with Message\<"foxglove.\[SchemaName]">:



For your own TypeScript project, you can import the type from the @foxglove/schemas npm package:



Leverage the full potential of Foxglove Studio

We developed Foxglove Studio to advance the state of the art in open source robotics. MCAP was our first step towards visualizing general-purpose robotics data, and Foxglove schemas are a continuation of that vision. We want to make it easy for all robotics teams, regardless of framework, to get data into Studio and take full advantage of our tools.

Explore all Foxglove schemas via our docs or the foxglove/schemas GitHub repo. As always, join our Slack community to ask questions, or reach out to us directly with any feedback.


Introducing Foxglove Schemas to Visualize Non-ROS Data

Read more:

Ready to get started?Download today on Linux, Windows, or macOS.