As a scalable data management service, Foxglove Data Platform is a convenient one-stop shop for storing and analyzing your robotics data. You can streamline your data management process – usually an extremely time-consuming and expensive part of development – and reserve your team’s focus for less routine parts of your workflows.

Foxglove Data Platform currently supports importing and exporting ROS 1 (.bag) files and MCAP (.mcap) files. Due to upstream ROS 2 limitations, Data Platform cannot meaningfully store ROS 2 (.db3) files, as they do not contain the message definitions necessary for proper introspection and analysis.

This isn’t to say Foxglove Data Platform is off-limits for your ROS 2 team – there are several ways you can convert your .db3 files into self-contained MCAP files.

The easiest way to achieve this is to install our rosbag2\_storage\_mcap plugin, and have your robots record MCAP files directly. But if you want to stick with your existing recording plugin, or have already-recorded ROS 2 data waiting to be analyzed, continue along with this tutorial to convert your ROS 2 data into MCAP files for proper storage and analysis.

Install the mcap CLI

For a user-friendly way to convert your ROS 2 files, download the mcap CLI tool for your platform – we currently support Linux, Windows, and macOS.

Check out the releases page to download the latest release binary, or install directly via your command line using go:



Point to your custom message definitions

Next, you’ll need to specify to mcap where you store your ROS 2 message definitions.

Let’s say your msg files live in the /opt/ros/galactic directory. You’ll want to open up your setup bash file (i.e. /opt/ros/galactic/setup.bash) to set an environment variable called AMENT\_PREFIX\_PATH to your top-level ROS directory:



mcap will now know that it needs to search in /opt/ros/galactic to locate the ROS 2 message definitions on your hard drive, and to include those definitions when outputting your converted MCAP files.

Converting to MCAP

You’re now ready to convert your ROS 2 files into MCAP files!

Run the following command in your terminal to generate a MCAP file (e.g. my\_output.mcap) from a local ROS 2 file on your computer (e.g. your\_input.db3):



If you didn’t properly the AMENT\_PREFIX\_PATH environment variable in the previous step, you can even pass in your message definition directory paths as a command line flag:



If your message definitions live across multiple directories, use a colon-separated list of paths:



You should now be able to see a MCAP file output to your computer!

Run $ mcap info my\_output.mcap to see what your shiny new file actually contains – you should see a print out of your data's message count, time range, channels list, and other relevant information:



Congratulations on your first successful ROS 2 to MCAP conversion!

Uploading to Foxglove Data Platform

Now that you have your data in a format that Foxglove Data Platform understands, let’s get it imported so you can start tagging and analyzing your data with the rest of your team.

There are two ways to upload your data – through the Foxglove web console or with the foxglove-cli tool.

Web console

Create an account or sign in to your Foxglove account at console.foxglove.dev to view your dashboard.

Before uploading your MCAP to Foxglove Data Platform, you must add the device that your data should be associated with – i.e. the robot that your data was recorded on. If you haven’t yet created the necessary device, click “Add device” at the top of the Devices page to create a uniquely named robot for your team to track.

Foxglove Data Platform Devices view

Next, click "Import data" on the Files page to start uploading your my\_output.mcap file – be sure to associate it with your created device:

Foxglove Data Platform Files view

You should now be able to see your import on both the Files and Timeline pages! To see your data in the Timeline view, make sure you use the date picker to navigate to the time range in which your data was recorded:

Foxglove Data Platform Timeline view

foxglove-cli

Import your data with foxglove-cli if you want to write a script to automate or batch your imports. Install the latest release for your operating system and architecture, or install directly from source.

As with the web console, you must first authenticate yourself and add a device corresponding to your data:



To upload your output MCAP file, run the following command:



Alternatively, you can use the following shorthand to achieve the same results:



To verify that your import succeeded, print out a list of all your imports:



You should be able to locate an import with a device ID and filename that correspond to yours (i.e. dev\_drpLqjBZYUzus3gv and my\_output.mcap, respectively).

Continue to explore your data

Now that you’ve imported your ROS 2 data, you can leverage Foxglove Data Platform to better organize and analyze your team's data.

Attach rich metadata to your imported data, tag events of interest for your teammates, and submit hyper-specific queries to your data lake. Retrieve the information you need at lightning speed, with Data Platform's strategic data partitioning and indexing. You can even click a single button to stream a specific snippet of data into Foxglove Studio, for deeper visualization and debugging.

In short, this tutorial is just the start. Join us in our Slack channel if you have questions about getting started, or book a demo with our team to learn more about our active (and planned!) features.


In my spare time for the past several months, I’ve been developing an autonomous RC car for DIYRobocars – a community of makers who race their DIY autonomous cars of all sizes. At the Robot Block Party hosted by Silicon Valley Robotics this past April, my car came in second in the Robocar race and also won the demolition derby.

My robocar taking out the competition at the demolition derby.

As you might expect, a good data logging and visualization infrastructure was crucial to helping me develop my successful autonomous driver. It took several iterations and a few false starts before I landed on my final solution – using Foxglove’s MCAP C++ reader-writer with Protobuf messages. I found the MCAP writer to be a solid foundation for my logging architecture, affording me a lightweight solution for both out-of-the-box support and open-ended flexibility.

Initial experiments with Python and ROS

I decided against using ROS in my robocar stack for a variety of reasons. For one, I wanted to write most of the core software from scratch – it’s part of the fun! I also wanted to get started quickly, and I decided that the time I’d invest in learning the ecosystem probably would not pay off for me in my future projects.

Furthermore, it wasn’t feasible to simply import the rosbag writer into my existing C++ build. Getting ROS working with my C++ build would likely have been a nightmare, given my desire to both cross-compile and run Raspbian for the Raspberry Pi (instead of Ubuntu).

In my first foray into logging and visualization, I used a custom binary format to log video frames and CSV lines to log sensor data, PID control outputs, and all other data. I then wrote a Python script, leveraging Python’s dependency management and standalone rosbag writer, to export ROS 2 files that could then be read by Foxglove Studio.

This scheme quickly fell apart. Schema evolution for logged messages and topics was manual, tedious, and error-prone. Older log files quickly became unreadable. The additional Python conversion step also made things more complicated and annoying.

With this first failed experiment, I tried introducing minimal dependencies on sqlite and FastCDR, which let me cobble together my own rosbag2 writer. This let me natively log rosbag2 files which could then be viewed directly in Foxglove Studio.

However, there still remained a few pain points. With this direction, I was maintaining a significant amount of code – the rosbag2 writer and especially reader – that didn’t add much core value to my autonomous car. Since FastCDR’s IDL is not widely adopted, I had to write my own definitions for the standard ROS 2 messages I wanted to publish. Foxglove Studio also could not support custom ROS 2 messages, due to an inherent ROS 2 design decision. Finally, there was no easy way to do compression, which inevitably led to unwieldy data logs and longer data offloading times.

Using MCAP with Protobuf messages

Around this time, Foxglove released an early version of their MCAP reader and writer, with C++ and Python support (they currently also support TypeScript, Go, and Swift). Given the hurdles I’d encountered with my previous iterations, I decided to give the library a try, using custom Protobuf messages.

I immediately noticed a few advantages. For one, Protobuf is well documented, with robust C++ libraries and backwards- and forwards-compatible semantics for message schema evolution. I also found myself enjoying just how easy it was to integrate MCAP into my stack. It is a small library with relatively few dependencies, yet surprisingly flexible. I was able to leverage Foxglove’s already-defined protobuf message schemas for standard ROS messages, but also define my own custom messages. I also found the ability to tune compression settings incredibly useful for reducing the size of my data logs and for accelerating data offloading.

The only con I found was that with MCAP being fairly new, there hasn’t been much widespread adoption yet. With that said, I think anyone else who gives the library a try will have as positive of an experience as I had.

Creating an MCAP writer in C++

An MCAP file has messages, channels, and schemas.

In the same way that ROS publishes messages to a topic, MCAP writes messages to a channel. Messages for a given channel always follow the same schema, but a particular schema can be shared by multiple channels.

Creating an MCAP writer is relatively straightforward:



Configure your writer to your desired specifications using McapWriterOptions. For example, opts.compressionLevel = mcap::CompressionLevel::Fast customizes your writer to use a faster compression level.

Before we can write messages, we need to register a schema and a channel to write our messages to.

To register a schema in Protobuf, you must use the fully-qualified name of the message type (e.g. ros.nav\_msgs.Path) and provide a serialized google::protobuf::FileDescriptorSet for the schema itself. Generated Protobuf messages will contain enough information to reconstruct this FileDescriptorSet schema at runtime.



Once we have the schema registered, we can register our channel:



We can now finally write messages to the channel via its id:



Don’t forget to close the writer when you’re done:



Now, we can inspect our output MCAP file's messages. I used the Data source dialog in Foxglove Studio to “Open local file”:

Data source dialog

I then added a few relevant panels (Plot, Image, Raw Messages, 3D) to visualize my robocar's performance on the "road".

Foxglove Studio replaying the exact moment my robocar collided with a demolition derby competitor.

Conclusion

I found Foxglove’s MCAP writer to be a solid building block when building my robocar’s data logging system. I used it in tandem with Foxglove Studio to quickly publish, visualize, and iterate on my robot – all by getting a better understanding of the data it recorded.

I was even able to add several bespoke enhancements specific to my project. I wrapped the MCAP writer in a layer that serialized Protobuf messages, wrapped them in a mcap::Message, and registered schemas for unknown Protobuf types in one centralized place. I also moved mcap.write to a separate thread fed by a queue, which avoided the time-sensitive driver control loop being disturbed by the long-tail filesystem i/o latency observed on the Raspberry Pi 3.

If you find yourself unhappy with your current data logging solution, I encourage you to give MCAP a try – it certainly has turned out well for me!


The robotics industry is at an inflection point in its history. Whereas robots up until now have been deployed mostly in controlled environments like factories and assembly lines, we’re now reaching a stage where advancements in artificial intelligence and computer vision are helping robots evolve past obedient drones.

Robots today can navigate cities and explore the ocean, mow lawns, and even perform surgery. They’re able to extract meaningful patterns out of previously unfathomable pixels; they’re able to improve themselves. They make dozens of invisible complex decisions in the blink of an eye – detecting targets, predicting futures, and planning next steps.

So how will our day-to-day lives look past this inflection point, when robotics permeates more areas of our lives? What are the biggest remaining challenges we have yet to tackle, and what are the possible upsides we have yet to enjoy?

A brief history of industrial revolutions

To understand how robotics will revolutionize our world, let’s first explore how previous industrial revolutions have redefined our lives and rewritten our political and economic destinies.

Often characterized as the farm-to-factory revolution, the 18th century’s First Industrial Revolution took the benefits of the Agricultural Revolution to pave the way towards large-scale industry. The population had boomed due to improved farming techniques, and the increased privatization of land motivated many to migrate en masse to cities in search of work. Factories and mines drew from this growing workforce to mine new power sources like coal, and invented new machines like the steam engine to mechanize huge swaths of production in industries like textile manufacturing.

​
First Industrial Revolution

A century later, the Second Industrial Revolution focused on producing iron and steel and harnessing electric power to birth entirely new industries, like automobile production and communication technologies. It is responsible for pivotal inventions like the light bulb, internal combustion engine, and telephone. This revolution also set itself apart by its unprecedented rate of production – it didn’t simply mechanically produce goods; it mass-produced them.

Second Industrial Revolution

The advancements catalyzed by these first two revolutions allowed people to enjoy cheaper goods, more wealth, and a higher standard of living. This trend continued with our most recent industrial revolution. The Digital Revolution of the late 20th century effectively migrated information online. It leveraged inventions like electronics, computers, and the Internet to automate entire phases of production. It empowered businesses to globalize, outsource, and expand at unprecedented speeds.

Third Industrial Revolution

Each of these revolutions was catalyzed by an emerging technology – steam power, electric power, the all-powerful Internet – that took work out of humans’ hands and accelerated it beyond incremental linear growth. They had an outsized effect on society, seemingly overnight – they redefined entire systems of working, rendered certain jobs obsolete, and birthed completely new industries. These revolutions weren’t just “bigger and better” – they were paradigm shifts that redefined labor and production for a generation.

The revolution of our generation

Despite massive leaps in technology from the Digital Revolution, much work and industry today is still manual and labor-intensive. That is set to change in the current Industrial Revolution. Instead of simply automating things, this Fourth Industrial Revolution is building technologies that combine knowledge across previously unrelated disciplines, to transform entire systems of production at an exponential rate. It steps into spaces previously dominated by human experts and builds technologies to outperform those humans – often by orders of magnitude.

This cross-pollination of our most advanced findings opens up a world of possibilities. We see digital technology interfacing with the physical world, with the Internet of Things and 3D printing. We also see digital tools making sense of biological problems, as with medical software used to detect diseases and develop vaccines. And of course, we have physical robots run by digital minds – from warehouse robots to self-driving cars.

Internet of Things

If there is any cross-disciplinary field that will play a huge role in defining this Fourth Industrial Revolution, it’s robotics. This field is building “collaborators” who can think, reason, and learn alongside us. With advancements across artificial intelligence, computer vision, and hardware, we’re now able to tackle new problems by leaning on the knowledge of our newly “sentient” colleagues.

How Foxglove is contributing to the revolution

It’s awe-inspiring to recognize that roboticists are solving problems that we believed intractable just a few years ago. With that said, there are still several categories of challenges that we have yet to address – from generalization and hardware challenges, to deploying humanoid robots for especially complex tasks.

Our team at Foxglove is tackling a less frequently discussed challenge in robotics – building robust developer tools. We think building robots is much more difficult than it needs to be. With the painful lack of off-the-shelf options, roboticists have to navigate difficult-to-integrate applications, fragmented or overlapping resources, and a steep learning curve for the tools that do exist. In short, robotics currently feels like building websites in the ‘90s – instead of delivering value to our users, we’re fighting against baseline problems like how to buy and rack our servers (or in our case, how to get our tools working).

Foxglove's guiding mission is to close the gap between the industry's tooling and its needs. As we develop exciting new technologies, we need equally novel ways to troubleshoot and iterate on them. By reimagining the next generation of robotics tooling, we hope we can help roboticists – across sectors, use cases, and audiences – accelerate their development and bring their products to market.

Foxglove Studio screenshot

Working with robotics data is an especially important part of robotics development. By gleaning insights from their data, engineers can get a sense of how their robots are performing, and where in the stack they may need to adjust an algorithm or squash a bug. Rich interactive visualizations – like the ones in Foxglove Studio – give roboticists a barometer by which they can assess and track their progress. Intuitive interfaces for organizing, tagging, and searching data – like Foxglove Data Platform – help them spend less time wandering aimlessly through petabytes of data and more time separating the meaningful signals from the noise.

Foxglove has been partnering with robotics teams – across industries like space exploration, autonomous vehicles, and ocean research – to build solutions that can be generalized across the wider community. Our goal is to chip away at the friction around managing and exploring our data, regardless of your particular team's use cases and audience.

What the future holds

Since robotics can be applied to almost every single industry in the world, it will likely disrupt them all to varying degrees. It’s exciting to think about the explosive growth we'll experience in this Fourth Industrial Revolution – especially as robots can reliably free up more time for us to focus on our yet-unsolved problems.

Even though Foxglove is only one company in this sprint towards the future, we’re excited to contribute in our own way. We’ve started by focusing on helping people visualize and manage their robotics data, but the possibilities are endless. We don't know what else we’ll find, but we do know this revolution will redefine our lives – we hope you’ll join us for the ride.


Manage your robotics data with Foxglove Data Platform

Scalable data management for fast-moving robotics teams

Manage your team settings and data through a convenient web UI.

Analyze meaningful segments of your data.

Use our public API endpoints or CLI to quickly retrieve the data that you're interested in.

Ready to get started?Upgrade to a paid plan today.