Scaling Your Robotics Data Strategy

Find smarter ways to organize and collaborate over data

Esther Weon ·

6 min read

Published August 1, 2022

As a robotics developer, you are likely handling a wide variety of multimodal data on a daily basis – whether it’s live or recorded, real or simulated. You’re also probably figuring out how to engage with this data at the correct “level” for a given task. Do you want a bird’s-eye view of how your perception has improved across releases? Or do you want to drill down into an issue your robot encountered this morning? Something in between?

Managing intimidating amounts and classes of data, and switching back and forth between the macro and micro views necessary to understand them, is critical to developing successful robots. By centralizing your team’s data practices with Foxglove Data Platform, you can streamline collaboration and dramatically accelerate your team’s progress.

Before Foxglove Data Platform

So what does life before a scalable data management solution look like for a typical robotics team?

When building robots, you are wrangling multiple categories of data. You need to view live data to debug your robot and offload recorded data to analyze it. You need to visualize logs to see how your robot navigated real-world scenarios, and run simulations to predict how it would perform in hypothetical ones. In all these scenarios, you also need to be able to search for especially relevant segments of data, and share them with your teammates.

Without a centralized data strategy, you and your teammates may be using a variety of manual workarounds to organize this data. You may be passing hard drives around the office, to get data to the people who need it. You may be naming Google Drive folders by dates, devices, software release versions, or any number of other namespaces. You may even be duplicating data across hard drives and cloud folders, because different organizational hierarchies make sense for different teams.

As data grows, it becomes increasingly difficult to find what you need. It can exist in multiple formats and multiple places, be accessed by different groups of people, and be used for different goals. Searching data becomes such an opaque process that multiple sources of truth start proliferating across the team. Developers spend more time locating data and performing data housekeeping than they do on actually tackling the core robotics challenges integral to their team’s success.

Disorganized data often leads to equally haphazard ways of collaborating on it. It is crucial for members of a robotics team to be able to share data and coordinate work across levels of the robotics stack – otherwise, it becomes significantly harder to iterate quickly. You may want to share notes on a specific incident, debrief on a simulation test’s results, or simply debug an issue together. You may have to record your screen to demonstrate how you debugged an issue, physically pair with a teammate at a single machine, or create even more Google Drive folders and documents to explain your thought processes and insights. These extra assets (and the unspoken systems for managing them) proliferate alongside your data, adding to the original problem.

Finally, switching back and forth between macro and micro views to properly analyze data can be time-consuming and expensive. When data is stored without regard for efficient reading or retrieval, it becomes tedious to zoom in and out of it. You may want to scan months of data for examples of a particular issue, and then drill down into short segments to debug what happened in each case. How efficient is it to download enormous files from multiple hard drives or cloud buckets to your local machine to get the breadth of coverage you need? Or to wade through terabytes of irrelevant data to get to the relevant megabytes you need? There is no way to extract just the data you are interested in at either level, wasting time and adding unnecessary friction to the development process.

After Foxglove Data Platform

Without intervention, data issues can compound and breed new ones. All these obstacles prevent your team from iterating more intelligently on your robot and getting a working product to market faster.

Foxglove Data Platform tackles these common data management pitfalls by providing teams with a centralized repository for collaborating on their data. Once you create a Foxglove account at console.foxglove.dev and invite your team members, everyone on your team can start uploading data and collaborating on it together. Any contributions you make to this repository are instantly shared with every member of your organization, so there is no danger of working on outdated or inaccurate data.

After uploading files, you can annotate them with rich metadata, making it easier to navigate and search your data. In addition to tagging every import with a recording device and time, you can add other descriptive factors of your choice – like release version, location, weather conditions, and more. You can even tag events within your data to identify points of interest. In short, there’s no limit to how you can choose to describe and organize data. You have the freedom to choose the organizing principles that will help your team explore your data most meaningfully – and since data is shared, your teammates can build on top of your work instead of recreating it. You no longer have to duplicate data across different hierarchies of folders or drives – you can simply filter by your desired metadata values.

Foxglove Data Platform

Data Platform also indexes data to more efficiently retrieve the information you need – and nothing more. This makes it possible to specify as granular a query as you need, whether it’s GPS data recorded by a specific robot within a given time range, or all LIDAR data recorded in a particular city under rainy conditions. Instead of spending time and storage space on downloading, analyzing, then deleting enormous files from your computer, you can now zoom in precisely on the data you care about.

This indexing is what allows you to go from wide-scale analysis across months to hyper-specific incident debugging in a matter of seconds. Do you want to see data recorded on a particular robot, or all the files you’ve imported? Do you want to see all your robots’ data on a timeline? Or do you want to drill down on specific events for more focused analysis? Whatever your workflow, Data Platform lets you specify the lens through which you want to engage with your data. Rapidly switch back and forth between the macro and micro views that can help you learn about your robots and drive future iterations on them.

Finally, Data Platform also integrates natively with Foxglove Studio, making it effortless to explore data from yet another standpoint – visually. Once you’ve located the event or time range you want to inspect, you can click a single button to stream it in Studio within seconds, for instant visualization and debugging. Studio provides a rich suite of configurable panels to help you analyze different aspects of your data – whether it’s diagnostics, camera feeds, or 3D scenes. Studio also allows you to maintain team layouts, so you can build curated visualization and debugging dashboards to use with your teammates.

Foxglove Studio

Managing your data (while also trying to build a functioning robot) is a huge undertaking, no matter the scope of your project. Whether you’re working on a single robot or an entire fleet, leveraging your data well can determine whether your team is able to ship a successful product on time.

By transforming how you relate to your data, Foxglove Data Platform helps you completely reimagine your team's best practices from the ground up, streamline every aspect of your development, and scale with your team’s growth.

To get started, sign your team up for an account and start importing data today. If you’re interested in a private cloud or on-premises solution, please don’t hesitate to reach out to us directly.