We've shipped many visualization features to Foxglove Studio in its first year of development.

To better showcase these features, we've added a sample dataset and layout for exploring the different stages of a self-driving car robotics pipeline.

Foxglove Studio's data source dialog
Foxglove Studio is an open source data visualization platform for robotics and autonomy developers. Download Studio here, and click "Explore sample data" on app load to see the sample dataset and layout.

We hope this demo will inspire you to find new ways of leveraging Foxglove Studio for your own team's development workflows!

A brief history of self-driving datasets

Since our own team's history is steeped in self-driving technology, we knew we wanted our first demo to spotlight autonomous vehicles. But with the wide availability of open source self-driving datasets, we had to ask ourselves – which dataset should we choose?

The KITTI Vision benchmark, which was published in 2012 as a research paper, is considered the original publicly available dataset for autonomous vehicles. It consists of six hours of driving around a rural German town, and includes GPS coordinates, camera images, lidar returns, bounding boxes, and basic lane markers.

KITTI remained the default self-driving dataset until 2019, when Motional released nuScenes. This similarly sized dataset consisted of 1000 scenes from Boston and Singapore, and included more cameras, radar sensors, and object classification labels (32, up from KITTI's seven). It was even accompanied by a Python library to facilitate accessing the data. This prompted Lyft to release their own dataset that same year, using a fork of the nuScenes Python library and a similar data organization. With this revived interest in public datasets, other automakers and AV companies – like Argo, Audi, Berkeley, Ford, and Waymo – followed suit.

Meanwhile, the nuScenes dataset has continued to evolve with expansion packs that added raw CAN bus data, rich semantic maps, class- and instance-level lidar segmentation annotations, planning trajectories, per-pixel image masks, and more. Taken together, this data starts to resemble the data pipelines powering today's production autonomous vehicles, making it a great fit for demonstrating the power and flexibility of Foxglove Studio.

nuScenes data is copyright © 2020 nuScenes and available under a non-commercial license.

A closer look at nuScenes

This section provides a technical breakdown of the nuScenes dataset. Feel free to skip to the next section if you're not interested in these details.

To visualize the nuScenes dataset, we first have to understand the data it contains and the format it's stored in.

Downloading the Full dataset (v1.0) contains the majority of the data: sensor captures (camera, radar, lidar), localization, and maps for the relevant regions. We then integrate the CAN bus expansion for high-frequency diagnostics, Map expansion for semantic map layers, and nuPlan maps for high-definition 2D lidar raster maps.

Once these datasets have been downloaded and unpacked into the same folder, you have all the source data used to generate our final dataset. We've made our own script for downloading and unpacking this data available on GitHub, so the following information is purely for reference.

Sensors

Camera images, lidar returns, and radar returns are all stored under the samples folder.

Camera images are organized into subfolders for each sensor, with individual frames stored as 1600x900 JPEG compressed images. The dataset contains "keyframes" at 2Hz (every 500ms), so that each sensor has a frame with corresponding bounding box annotations for a given timestamp. Camera images are also available outside of the keyframe captures, at roughly 12Hz.

Lidar returns are stored in the LIDAR\_TOP subfolder. Individual point cloud captures are stored as .pcd.bin files, which use a simple packing of six 32-bit floating-point values (little-endian, four bytes per float) storing x, y, z, intensity, and ring values per point. The x/y/z coordinates are in the LIDAR\_TOP coordinate frame.

Radar returns are also organized into subfolders for each sensor. Individual point clouds are stored in PCD v0.7 format, which uses an ASCII header followed by binary data that stores 18 different fields using different primitive types and byte widths.

Localization

Localization data lives in a single ego\_pose.json file.

Each entry in the JSON array contains a timestamp in microseconds, a rotation quaternion, and a translation vector3. Although the ego pose (i.e. the car's position) is available at a higher frequency than 2Hz, each keyframe links the sensor samples to the nearest ego pose in time.

Vehicles

The CAN bus expansion data lives as JSON files in a can\_bus subfolder.

These files contain raw CAN vehicle bus data – like position, velocity, steering, lights, battery, and more – per scene. For more information, check out the CAN bus expansion README or tutorial.

Map

The nuScenes map expansion and nuPlan maps include the following:

High-definition lidar maps

Raster PNG files for driveable area bitmaps and lidar intensity maps

JSON vector data for lane centerlines, lane boundaries, sidewalks, crosswalks, stop lights, and more

More information can be found in the nuScenes maps expansion tutorial.

Visualizing nuScenes with Foxglove Studio

To visualize this data in Foxglove Studio, we need to convert it to a format that Studio understands.

While Foxglove and the ROS Middleware Working Group are designing a generic recording format for robotics data (MCAP), that file specification is still a work in progress. In the meantime, the ROS bag format provides a convenient way of packaging heterogeneous data streams into a single file with type definitions, indexing, and compression.

We created a Python Jupyter Notebook to load the nuScenes data and convert each scene into a ROS bag file. Each category of data had to be converted into a ROS message type that Studio understands:

Camera images – sensor\_msgs/CompressedImage, sensor\_msgs/CameraInfo

Lidar and radar returns – sensor\_msgs/PointCloud2

CAN bus data – diagnostic\_msgs/DiagnosticArray

Raster maps – nav\_msgs/OccupancyGrid

Semantic maps and bounding box annotations – visualization\_msgs/Marker

Car's position (or the ego pose) – geometry\_msgs/PoseStamped, sensor\_msgs/NavSatFix, tf2\_msgs/TFMessage

Additional data superimposed on camera images – foxglove\_msgs/ImageMarkerArray

Next, we created a Foxglove Studio layout that visualizes all this data. Since there is a lot to see in this dataset, starting with a top-level Tab panel helped organize the data around four main use cases: Perception, Planning, Controls, and Diagnostics.

Perception

Our Perception tab focuses on sensor data, with the majority of the screen dedicated to displaying sensor\_msgs/CompressedImage topics across six Image panels. The top row shows the front camera feeds; the bottom row the rear cameras. Since the Motional car has 360° camera coverage, we manually aligned the camera feeds in this layout to create two 180° views.

Each camera image also has its own pair of foxglove\_msgs/ImageMarkerArray topics for the lidar points and bounding box annotations, both of which have been projected into the individual 2D camera frames.

The bottom-most panel in this tab is a 3D panel, which shows the world from the perspective of the front camera – with bounding box annotations, lidar returns, the lidar raster map, and more.

Planning

The Planning tab focuses on getting a comprehensive understanding of the whole scene, by combining a map of the area with a richly annotated 3D scene. It is split into a Map panel showing the GPS track of the vehicle, and a 3D panel in orthographic mode showing a bird's-eye view. The 3D scene displays lane centerlines from the semantic map, our car's pose, bounding box annotations for nearby detections, and lidar and radar point clouds.

The smooth movement of the vehicle is achieved using transform interpolation. This recently added feature allows transforms to be smoothly updated at full framerate, as long as there is a transform before and after the current time. Even though our transform messages are only published every 500ms (2Hz), we can achieve smooth movement throughout playback by publishing the transform for time t+1 at time t. Publishing future data is a luxury of post-processing an existing recording, but this can be used on live robots as well by publishing a prediction of the transform at t+1 (and overwriting it with observed data once time advances to t+1). For a deeper technical understanding of transform interpolation in robotics, the whitepaper tf: The Transform Library is a good starting point.

While the vehicle moves smoothly, everything else in the scene is moving at the publish rate of the different data streams. Lidar and radar scans come in at about 12Hz while bounding boxes are only updated at 2Hz, which appears as visibly snapping from position to position.

This tab uses the following datatypes to display the various components:

sensor\_msgs/NavSatFix – GPS

tf2\_msgs/TFMessage – Coordinate frames, relative positions of the car and its sensors

visualization\_msgs/MarkerArray – Bounding box annotations, lane centers

geometry\_msgs/PoseStamped – Car pose

sensor\_msgs/PointCloud2 – Radar and lidar returns

nav\_msgs/OccupancyGrid – Driveable area around car, grayscale map

Controls

The Controls tab compares the throttle input against the vehicle's acceleration, as measured using the Inertial Measurement Unit (IMU) and speed of all four wheels.

Each data series in the two Plot panels is specified using a message path, Studio's own syntax for referencing message data.

Diagnostics

The final Diagnostics tab gives you a peek under the hood at the raw data and vehicle diagnostics.

The State Transitions panel at the top also uses message path syntax to show the brake\_switch value change over time – between 1, 2, and 4. According to the nuScenes CAN documentation, possible states for this brake\_switch value are 1 (pedal not pressed), 2 (pedal pressed) or 3 (pedal confirmed pressed). The unexpected 4 we see is worth investigating, if we're interested in using this signal.

The Diagnostic Summary and Diagnostic Details panels below show the CAN bus messages formatted as a diagnostic\_msgs/DiagnosticArray.

The Data Source Info panel in the top right lists all topics, datatypes, and publish frequencies for our bag file.

The last view is a Raw Messages panel, used here in a rather unique way. By setting the message path to filter for a marker with a specific ID, we can monitor a single marker over time. For another creative use of this panel, try changing the message path to point to a primitive value (e.g. /markers/annotations.markers\[:]\{id==28114}.pose.position.x) for a single value monitor.

Further exploration

This sample dataset and layout was built to showcase the breadth of Foxglove Studio's newest features and to inspire you to visualize your own robotics datasets. This nuScenes example will be the first of many upcoming demos in Studio, all tailored to different verticals in robotics.

In the spirit of our open source philosophy, the code we used to convert the nuScenes data into ROS bag format is available on GitHub. While this was our approach, there are many ways this conversion could be extended – visualizing nuPlan trajectories, upgrading the car marker to a 3D mesh marker, or segmenting lidar points to different topics for each object classification. There is also additional nuScenes data that we could have pulled in to provide even richer visualizations.

If you have questions or ideas on how to visualize your data with Foxglove Studio, join us in our Slack community's #lounge channel.


Since v0.21, Foxglove Studio has supported loading mesh resource markers in the 3D panel. This means that you can now visualize custom models in the context of your 3D scene, alongside the other players and perceived objects present in your robot's world.

So how can you start loading 3D models into your scene?

Write a Python publisher

First, you’ll need to publish a marker topic, following the message definition outlined in ROS 1’s visualization\_msgs/Marker or ROS 2’s visualization\_msgs/msg/Marker. In both schemas, you'll notice a mesh\_resource field, where you can specify the file containing your 3D model. Studio currently supports reading an HTTP, HTTPS, or ROS package URL from this field to load glTF (.glb), COLLADA (.dae), and STL (.stl) files.

On a computer with a working ROS setup, create a Python file (e.g. mesh\_marker.py) containing a simple publisher that outputs marker messages with a currently empty mesh\_resource field:



Now that we've written our publisher, let's try loading a few different types of mesh resources in Foxglove Studio's 3D panel!

Load a glTF (.glb) file with an HTTPS URL

Let's load a glTF file in our published marker's mesh\_resource field. In your mesh\_marker.py file, replace the empty field value with the following URL:



Run the following command in your terminal to start publishing marker messages on the /avocado topic:



To verify that your newly published messages include a link to the .glb file, inspect the /avocado topic in a Raw Messages panel in Foxglove Studio. If everything is working as expected, you’ll see that the mesh\_resource field points to an HTTPS URL for a glTF asset.

Topic in Foxglove Studio's Raw Messages panel

Next, add a 3D panel to your layout to see the mesh marker in 3D space. Open the topic picker to toggle on your mesh marker topic (/avocado). You’ll see a 3D avocado appear floating in the scene.

Avocado mesh marker in Foxglove Studio's 3D panel

To see this working immediately, you can also drag and drop this meshmarkers.bag into Foxglove Studio to see the same results.

Load a COLLADA (.dae) file with an HTTPS URL

Now, let’s try loading a different mesh resource. Replace the mesh\_resource field value to point to an HTTPS URL for a COLLADA (.dae) asset:



In your 3D panel, you should now see a Gundam robot part where your avocado used to be:

Gundam robot part in Foxglove Studio's 3D panel

Load an STL (.stl) file with a ROS package URL

Finally, let’s try a link to a file included in a ROS package.

First, install the turtlebot3\_description ROS package on your machine. If you are using ROS Noetic, run the following command in your terminal:



Source your ROS setup file, then grab the correct package path:



Next, open the Preferences sidebar in Foxglove Studio to set your ROS\_PACKAGE\_PATH environment variable to your package path (e.g. /opt/ros/noetic/share). This makes it so that all ROS package:// URLs referenced in the app will now use your ROS\_PACKAGE\_PATH to find the relevant package assets (e.g. /opt/ros/noetic/share/turtlebot3\_description). Note that this will work only on the desktop app, as we are referencing files in our local file system.

Finally, replace the same line of code in your mesh\_marker.py file with the following:



If your ROS\_PACKAGE\_PATH was set to /opt/ros/noetic/share, the mesh\_resource above will look in /opt/ros/noetic/share/turtlebot3\_description/meshes/bases/burger\_base.stl for the file in question.

The STL file in question contains the following model:

Turtlebot base

You should now see it displayed in your 3D panel.

Show us what you’ve got!

We hope this new feature helps you build ever-richer representations of the world your robots navigate, to better analyze their behavior and improve their performance. If you would like more file formats (beyond .glb, .dae, .stl) to be supported in the future, please let us know! You can open a GitHub issue or message us directly in our Slack community.

This was a long-anticipated feature, so we’re excited to see what your Foxglove Studio dashboards look like now! Share images of your models in our Slack community’s #lounge channel or on Twitter (don't forget to tag us @foxglovedev). We’re so excited to see the workflows that this feature unlocks for your team.


It’s hard to overstate the impact that robotics is poised to have on the world within our lifetimes. The industry has the potential to completely reimagine our economies, our politics, and our daily lives. From applications as wide sweeping as self-driving to as specific as burger-flipping, robots are already transforming how we navigate our lives.

Which makes it no surprise that so many people’s imaginations have been captivated by this field. The thought of “working in robotics” seems glamorous in a way that almost no other tech sector currently does. But despite the overwhelming interest, robotics maintains a long-standing reputation as an esoteric industry, accessible only to the select few who received formal training in artificial intelligence, machine learning, or hardware. Talented applicants exclude themselves from the field all the time, based purely on their preconceived notions of what it takes to work on a robotics team.

The reality, however, is that you don’t need to have this hyper-specific background or to work directly on some component of the robot itself to succeed in the field. Robotics is an extremely cross-disciplinary endeavor, and it’s very likely that your professional experience in tech or any other industry translates to this exciting new frontier.

Below are several ways you can cultivate a long-term career in robotics – even if you don’t have a master’s in computer vision or a PhD in AI.

Front-end engineer

While writing algorithms to run on a robot may require specialized machine learning knowledge, that work makes up only half the battle. People often completely overlook the many tools that robotics engineers need to do their job at all.

That’s where developer tools come in. Robotics data visualizations, in particular, are crucial to helping teams analyze the data their robots gather. With the help of web-based tools, roboticists can now dive into the inner workings of their robot and more deeply understand what it's seeing, thinking, and deciding while out in the world. Understanding their robot’s current strengths and weaknesses, in turn, helps them iterate more intelligently towards a production-ready robot.

Foxglove Studio data visualization

As a front-end web developer working in the field, you’d be responsible for building these rich interactive interfaces for roboticists to debug their work. Foxglove Studio is a prime example of a web-based visualization app that does exactly that. Because it portrays data in a digestible visual format – all from the comfort of a browser or desktop app window – it empowers engineers to diagnose problems and formulate possible solutions efficiently.

Back-end engineer

Back-end development is one of the most relevant and in-demand skills for robotics teams today. Robots generate huge amounts of data – while communicating with each other, with their operators, and with the world around them. Operators need to manage fleets, assign tasks, and track issues. In turn, robots need to receive new tasks, share their plans, and provide results to operators. In order to deploy a successful fleet, teams not only perfect their robots’ capabilities, but also prepare an infrastructure for managing and housing this formidable amount of data.

Data management is an often overlooked, but clearly significant, aspect of robotics development. Ingesting and sifting through petabytes of data can be time-consuming, labor-intensive, and expensive. As time goes on, data accumulates, gets lost, or is unintentionally duplicated – making it increasingly difficult to know what information lives where. In addition to the actual stored data, roboticists also need an intuitive interface for storing, retrieving, and organizing this data.

Foxglove Data Platform timeline
Foxglove Data Platform is a back-end service that provides an API, web interface, and CLI tool for managing your data.

With robust back-end support, robotics teams can successfully wrangle the data that so centrally drives their development. As a team grows, so should the number of people working on back-end systems to coordinate between all these moving parts.

UX and UI designer

With the sheer breadth of robotics products, the industry serves a wide array of customers – both technical and non-technical, internal and external, consumer and enterprise. Roboticists need developer tools, sales associates need metrics dashboards, and QA engineers need debugging utilities. Once robots are deployed, their operators need fleet management software, troubleshooting support, and specialized training.

Since robotics use cases vary so wildly, the industry needs designers who are undaunted by the opportunity to solve problems that have never been solved before – for entirely novel workflows and niche customers. Because the product criteria and design standards are so unique to this field, designers will have no shortage of interesting challenges to unlock. By understanding their users’ pain points and finding creative solutions for them, designers will play a pivotal role in defining the best practices for this budding industry.

Foxglove Studio designs
We recently redesigned Foxglove Studio's Plot panel to help users compare each series' current values at a glance.

Marketing strategist

Educating the public about your product comes with the territory of working in a bleeding edge industry like robotics. Earning trust and building rapport with your potential customers is a huge part of this education process. As with any new technology, people can often be hesitant, confused, or even outright fearful of trying a solution for the first time – whether it’s as trivial as ordering a cappuccino prepared by a robot or as serious as introducing a robot companion to your child. In addition to teaching them what your product does, you need to demonstrate in concrete terms how they will benefit from (and enjoy!) using it.

Self-driving companies Cruise and Waymo

Many self-driving car companies, for example, plan years-long campaigns to build public trust in their technologies – by sharing demo videos, hosting community events, and inviting journalists to try beta versions of their technology. Their marketing can also come in more traditional forms, from advertisements that reveal under-the-hood details about the technology, to op-eds about the transportation industry. They expound on their stringent safety standards, on the benefits of going autonomous. All this marketing ensures that by the time they have a production-ready vehicle, they also have excited customers that trust their brand enough to pay for the product.

Marketing specialists with a technical background would be especially successful in this field, but anyone who is passionate about advocating for innovative products and their users will do well. Having both the empathy to understand users’ challenges and the creativity to present possible solutions for them is paramount to succeeding in this role.

Sales engineer

Because the business of building robots is such an inherently cross-disciplinary endeavor, many previously siloed roles must redefine themselves in the context of robotics. Nowhere is this more apparent in sales. Since robotics deployments are often so technically complex, robotics solutions providers need a sales team that is deeply familiar with how the underlying technology works. Equipped with this expertise, they can then educate and ultimately persuade their customers – both upper management and technical operators – to adopt their robotics solution and reap the benefits.

A sales engineer is not only responsible for educating potential customers on how a product will be useful, but also helping those customers reach their full potential with that product. For example, sales engineers may walk a team through the benefits of adopting Autonomous Mobile Robots (AMRs) in warehouses – how it will mean less work for more throughput, a safer working environment, and a future-proof technology that will eventually pay for itself. But after that initial sale, they will also have to teach the AMR operators an entirely new set of skills – like how to manage multiple robots from one command station, how to assign specific robots certain tasks, and how to troubleshoot common issues.

Seegrid's Palion Lift AMR
Seegrid’s Autonomous Mobile Robots (AMRs) do everything from transporting, spearing, and horizontally and vertically handling goods – effectively reducing downtime and increasing output.

Since they partner so intimately with customers during onboarding, sales engineers are at a crucial juncture between the public and the rest of the robotics team. They're privy to valuable feedback that the rest of the company is not – the problems users have, the solutions they wish they had. Robotics teams can then use this information to determine what to prioritize next on the roadmap, thus ensuring that the their focus is always closely aligned with their customers’ needs.

Technical writer

Though the humanities are usually far from one's mind when thinking about robotics careers, there's a real need for good technical writing in this industry. In addition to the educational writing that comes with marketing your product to potential customers, technical writers are responsible for describing how these products work for an internal and technical audience. They catalog the relationships between various systems and services, document API endpoints and other interfaces, and essentially make it easy for members of their growing engineering team to understand, contribute to, and maintain these projects.

Great technical writing empowers your entire team to get involved in aspects of the product that they may not have built themselves. It also reminds us that our user base is not just the customer who will be operating our robots. They are also our teammates who need to get quick context on our ongoing work; they are the customers who will have to read our documentation when troubleshooting issues. With thorough documentation, we bridge the knowledge gap between the people who built the product and the audience for whom it was built.

Product manager

In many sectors of the industry, roboticists are working on problems that have never been solved before. We’ve never had fully autonomous robots successfully transport commuters, harvest our produce, or assemble homes – at least at significant scale. These are all work-in-progress ideals, and so we need imaginative leaders to help us navigate this uncharted territory and get at the essence of what this new generation of products needs. Synthesizing information across different verticals to successfully deliver a solution to the public is an extremely difficult, and thus valuable, skillset to have.

alt\_text
Applying robotics technology to agriculture could make our means of production safer, more efficient, and more resilient to climate change.

If there's one cross-disciplinary role in robotics and beyond, it's that of the product manager – by necessity, they must be both technical and user-obsessed, interested in building new features and also listening to the market. Because this role is such a mixture of so many different skills, it's especially possible to transfer your existing skills across fields.

Product managers work with sales engineers and developers, designers and marketers – all to serve the customers' interests. Some may even take on higher-level program manager responsibilities – like coordinating between teams to stay on schedule, adjusting the scope of deliverables to adapt to new information, and constantly assessing roadmaps to ensure they're doing the most important work for any given moment. If you enjoy the challenge of incorporating learnings from different teams to deliver a solution to your users, a product manager role may be a fit for you.

Join our team

As we’ve seen, robotics is an extremely cross-disciplinary and creative field. There are a wide array of problems – that we are still uncovering! – that we still need to solve. Finding the best talent out there in the world, whether that talent has yet been applied to robotics problems or not, is crucial to our collective success. If you’ve had experience as a web developer, marketer, or writer, you likely have the transferable skills needed to contribute to the bleeding edge of technology.

If you’re interested in envisioning a world enhanced by robotics technology, the industry would love your experience and passion. Job boards, like the one run by Silicon Valley Robotics, are a great place to start. You can also peruse robotics publications like The Robot Report to find companies working on problems that excite you, and then check out their career pages.

To learn more about the data and internal tools that these companies may use, check out our sample self-driving data in the Foxglove Studio app. For ways to get involved with the Foxglove team, you can sign up for our newsletter, join our Slack channel, or check out our own careers page.


How to Work in Robotics When You're Not a Roboticist

Read more:

Ready to get started?Download today on Linux, Windows, or macOS.