Whether you just got started with ROS or have a robotic duck roaming around your house, giving your robot more sensor inputs can be a great next step to take in your robotics journey, and adding a camera to your robot can be a simple and powerful way to do just that.

Teaching a robot to understand what it's seeing through a camera might sound complicated, but it doesn't have to be! As machine learning has become popular, it's also become much more accessible to non-experts, and there are now plenty of off-the-shelf toolkits that make it simple to analyze data and detect patterns.

When analyzing your sensor data, it can be helpful to see your outputs visually. A common approach for camera images is to produce an image that has been annotated with relevant markers. However, there are several limitations to drawing directly on the image – markers can cover pixels in the original image, appear blurry on small images, and be costly to store.

You can tackle these issues by using image markers, a kind of vector graphics that Studio can draw on top of the image. Studio even enables you to easily turn them on or off in your layout's Image panel.

In this post, we'll set up a camera, run a face detection algorithm on the images, and use image markers in Studio to visualize the results. (If you're not yet familiar with ROS nodes and topics, try reading The Building Blocks of ROS first — we'll be using them in this tutorial.)

Setting up a camera node

This post will assume you have a working ROS 1 installation on your computer. I'm using ROS Noetic on Ubuntu 20.04, running in a virtual machine. If you need to set up ROS, follow these installation instructions.

The first thing we need is a camera connected to our robot. For the purposes of this demo, I'm using my laptop's webcam, but you might be using another camera that plugs into a USB port. Let's install a ROS package called usb\_cam to interface with the camera and produce ROS sensor\_msgs/Image messages:



usb\_cam comes with a launch file that will start up the camera driver and an image viewer, so we can see that our camera is working:



image\_view window displaying images from the camera

(If this doesn't work for you, you might need to modify the usb\_cam parameters to select the correct camera device, pixel format, and other settings. You can create your own copy of the usb\_cam-test.launch file and adjust the parameters there.)

Looking deeper with Foxglove Studio

Behind the scenes, this has launched a ROS node that publishes sensor\_msgs/Image messages on a new topic. To get a better idea of how this works, let's hook up Foxglove Studio and peek at the ROS graph. First we'll connect to a ROS data source.

First, choose Open connection from the data source selection dialog:
connecting to ROS in Foxglove Studio

Select ROS 1 as the connection type. Since I'm using Ubuntu inside a virtual machine, I need to type an IP address where I can reach the Ubuntu guest before opening the connection.

connecting to ROS in Foxglove Studio

Next we'll create a new layout, add an Image panel, and display the /usb\_cam/image\_raw topic:

viewing camera images in an Image panel

Let's also open up a Topic Graph panel to visualize the flow of data in the system. Here, blue rectangles represent nodes, and pink diamonds represent topics. You can see how raw images, camera parameters, and log messages flow between the /usb\_cam driver node, the /image\_view window, and of course Studio itself.

visualizing the flow of data with a Topic Graph panel

Creating an image processor node in Python

Now that we've set up a camera and understood how data flows between nodes in the graph, we can begin processing images. We'll use rospy to write a simple ROS node in Python.

First, let's set up a node to receive camera images from the driver in a new my\_node.py file:



Make sure roslaunch usb\_cam usb\_cam-test.launch is still running, before running your script in a new terminal window:



By adding a Log panel to our current layout, we can view the output from rospy.loginfo directly in Studio. We can also confirm that /my\_node has appeared in the graph, and is receiving data from /usb\_cam/image\_raw.

viewing log messages in a log panel

The next step is detecting faces! We'll use a library called Dlib, which provides a face detector out of the box:



We can call dlib.get\_frontal\_face\_detector() to create a face detector. The face detector only accepts grayscale images as input, so when we receive an image in our image\_callback, we'll first use cv\_bridge to convert it to an OpenCV image, then convert that to grayscale using cvtColor.



Finally, we'll run the face detector, which returns a list of rectangles where faces were found. The callback logs a message indicating how many faces were detected — moving your face around on camera should affect the node's output in the Log panel.



log messages showing face detections

Now that we've determined whether a camera image contains faces, let's use image markers to see where these faces were detected.

Adding image markers

ROS provides a message called ImageMarker, which describes a visual annotation on an image. Since we might have a variable number of faces in the image, we'll need an array of markers. This isn't provided by default with ROS, but Foxglove has created our own ImageMarkerArray data type. You can install it by running:



Now let's update our code to publish an image marker for each face in the image! We're going to publish a new topic called /usb\_cam/face\_markers. (We use the same /usb\_cam namespace prefix as the camera driver to indicate that the face markers use the same coordinate space as the camera driver's images.)

In order to do this, we need to reorganize our code a little bit, moving the subscriber and publisher into a class so they can share state via self.



Then in our image\_callback, we'll publish an ImageMarkerArray every time we receive an image from the camera:



Finally, we'll add Node() to our def main().

Back in Studio, the Image panel now shows our new topic in the dropdown menu under "markers".

dropdown menu to enable image markers

When I turn it on, a box appears around my face — the detector is working! The /usb\_cam/face\_markers topic also appears in the graph.

camera image overlaid with a box around the detected face

Predicting facial features

Let's take it one step further. Not only does Dlib provide a face detector — it also provides a facial landmark predictor, which understands the location of several common points on a face. To use it, we'll need a trained model file, which you can download from this link here (99MB decompressed). Information about the model is provided in the imutils package, so let's install that too.



We can initialize the predictor with dlib.shape\_predictor(), passing it the path to the data file we just downloaded and extracted.

Once we've detected a face is present in a rectangular region, we'll run the predictor on that region to get a series of 68 points. Then we'll interpret those points using imutils.face\_utils.FACIAL\_LANDMARKS\_68\_IDXS, which groups them into 8 facial features (mouth, inner\_mouth, right\_eyebrow, left\_eyebrow, right\_eye, left\_eye, nose, and jaw).



We'll upgrade our image markers by drawing a line of a unique color for each facial feature.



Then, in our image\_callback:



And that's all for this tutorial — thanks for following along! Now that your robot can analyze images, and you can visualize the results with image markers, I hope it will be easier for you to add amazing new functionality to your robot.

face with lines drawn around eyebrows, eyes, nose, mouth, and jaw

Full sample code for this post is available on GitHub.

For more information, see the following reference materials:

usb\_cam documentation

Dlib documentation

Detect eyes, nose, lips, and jaw with dlib, OpenCV, and Python by Adrian Rosebrock

If you have questions or just want to chat about robotics visualization, join us in our Slack community.


The typical cycle for robotics development usually starts with a deployed robot collecting data on a real-world excursion – whether that world is a warehouse, a public road, or the ocean's "twilight zone".

Typically, this data is offloaded into storage at the end of the excursion, and then analyzed to determine the improvements the robot's software requires. The necessary changes are made, the new and improved software is deployed onto the robot, and that robot is sent back out into the world to collect more data.

And so the cycle continues.

In the robotics community, there is no shortage of tools that can help you inspect, analyze, and visualize your data. From native Robot Operating System (ROS) tools like rqt and rviz, to newer generations of tools like PlotJuggler and Foxglove Studio, there are a multitude of ways you can inspect the inner workings of your robot's brain.

But how do you get to a point where you have robotics data to visualize in the first place? What technical decisions do you need to make, and what steps do you need to take, to collect your data? How do you weigh all the possible strategies for this vital step?

Though there isn't an industry-wide standard or one right answer, there are basic trade-offs to consider when deciding how to collect your robotics data. The decisions you make will depend on your project's specific goals and constraints. By having a list of options to weigh, you can make more informed choices about the best data strategy for your project.

Data collection approaches

Before you can store or analyze any data, you need to collect it. There are several ways you can do this:

Local disk recording

Live telemetry

Hybrid approach

All these methods have their pros and cons – the frequency with which your robot needs to send real-time data, the volume of data it records on a typical excursion, and its offload bandwidth will all inform which method you should use.

Your guiding principle should be finding the strategy that will most consistently return uncompromised data from the dynamic real-world environments that your robot navigates. Your data will guide future iterations of your robot, so the method that allows you to collect and learn from your data most reliably is often a good place to start.

Local disk recording

Robots are often deployed to areas that are inconvenient or dangerous for humans to go, and consequently cannot rely on a consistent internet connection to safely offload data. These robots with inconsistent or limited in-the-field connectivity can write their data to local storage to ensure that it can be collected and retrieved properly.

robot in hazardous environment

Your robot's disk space will also affect how you collect data. With limited space, your robot must be more discerning about what data to record and what data to throw away. Running more business logic on the robot itself will conserve disk space – in this scenario, only the lighter-weight derived insights would be recorded to disk. While this “stretches out” disk space by writing a "lower resolution" version of your data, processing raw data live on your device will impact your robot's real-time performance.

Finally, the importance of real-time feedback will determine how you record your data. If your data does not need to be referenced until the robot is back from the field, purely local disk recording is ideal. However, if you need to monitor your robots in real-time, local disk recording would not be able to address this requirement.

Live telemetry

If it is crucial for you to know at any given time where your robot is, your robot must be able to offload data to a storage solution – like the cloud or your laptop – as it actively collecting it.

autonomous cars

Unlike local disk recording, which is blissfully independent of an internet connection, real-time offload requires a consistent and reliable connection to your robot – either wirelessly or via a physical cable. Since network connections can be unreliable, your robot can use in-memory buffering to avoid losing vital data whenever connectivity drops, transmitting these stored messages once a stable connection returns.

Due to bandwidth constraints, live telemetry is usually reserved for a small subset of your data. However, real-time offload of full-resolution data can be particularly useful in controlled scenarios, when your internet connection is reliable or when the scope of your task is within certain limits – for example, when you are conducting a sanity check on your robot or calibrating its sensors at homebase before sending it off into the field.

Hybrid approach

Both local disk recording and live telemetry can be present at once on a given robot – some data can be analyzed asynchronously after the robot has returned, and other data can be immediately transmitted for real-time monitoring.

Given that these approaches are useful in different scenarios, you can mix and match local disk recording and live telemetry for subsets of your robotics data. Ultimately, the environment that your robot navigates, the type of data it wants to collect, and its goals while out in the field will all determine how you record your data.

A hybrid approach is especially useful when you want both real-time insight into your fleet and high resolution data to diagnose potential failures. With live telemetry, you can monitor your fleet's location and current status. In the event of a failure, you can upload the full fidelity data recorded on the robot's local hard disk to review it for diagnosis. In short, the hybrid approach navigates the trade-off between how much local disk space you have (local disk recording), and how much data you can reliably offload via your available bandwidth (live telemetry).

Recording formats

Knowing how to record your robotics data is only half the battle. Another equally important question to address is the format in which to record this data.

There are several main criteria to consider when weighing data format options. Namely, you will want to choose a format that makes your data:

Conducive to analysis

Trivial to recover in case of hardware or software failure

Possible to record with low overhead

Easy to offload

Self-describing

Different data formats facilitate different types of analysis. Choose a format while keeping in mind the type of analysis you may want to perform – whether it's visually playing back a recorded session, or using SQL to query petabytes of recordings.

Your data should be stored in a format that makes it easy to recover. For example, in the event of an unexpected shutdown or accident, the data your robot has already recorded should be preserved and remain uncompromised.

Your robot's recording process should be IO and CPU efficient. The volume of data you write and its compressibility will all affect performance and offloading.

Finally, your data should contain self-describing schemas for easier deserialization and compression. This enables you to decode your data without having to reference separate interface definition files, which may change frequently.

In short, choose the data format that allows your robot to record and offload its data for future analysis – all within its performance and bandwidth constraints. Let's cover your options, based on your robotics framework.

ROS 1 robots

ROS is currently the industry standard for robotics data recording, mainly because it allows you to get up and running quickly, and easily integrate features that would be very time and labor-intensive to build yourself – i.e. modules for computer vision, SLAM, etc. This allows you to leverage the expertise of your peers, and makes collaborative robotics development extremely easy.

ROS allows you to write the messages published by your nodes throughout the course of a robot's excursion to a bag file. Bag files store ROS messages in a format that can then be played back, visualized, and analyzed – along with message definitions for all these messages. Many tools like rqt\_bag and rosbag have been written to help you record, play back, and otherwise interact with these bag files. Writing your data to bag files optimizes recording and playback performance, because you avoid deserializing and reserializing your messages.

ROS 2 robots

Whereas ROS 1 is a bundle of data formats, transport protocols, and reference implementations (C++ and Python), ROS 2 is a more generic C API with pluggable implementations. Even though ROS 2 has a default network transport protocol (DDS) and a default bag storage implementation (sqlite3), ROS 2 functions less as specific implementations and more as an abstract middleware interface for serialization, transport, and discovery. As such, rosbag2 allows recording and replaying of different data formats via a flexible interface, improving communication over different networks.

While ROS 2 seems like a promising new direction for ROS, there are a few in-progress issues that could pose a problem to recording and playback. Most notably, ROS 2 bags currently do not store message definitions alongside their recorded data. This omission makes it more difficult for external tools like Foxglove Studio to inspect and analyze this data. With that said, storing message definitions in ROS 2 bags is an open issue that the rosbag2 team is actively tracking.

Non-ROS robots

Outside of the ROS ecosystem, there are no clear-cut standards for data serialization formats.

Some teams use an open source data serialization format like Apache Avro, which optimizes for space and performance by storing data in compact binary format. Avro files are easy to process and read, because they store their data type schemas in JSON format. While JSON alone would get unwieldy for repeated fields across records, AVRO files use a direct mapping that makes it efficient for higher volume usage.

Other teams record a raw stream of protobuf messages to a file. Protobuf, Google's mechanism for serializing structured data, transmits data as a compressed binary to conserve space, bandwidth, and CPU usage. Because there isn't a storage protocol, you will have to do some upfront work to get started. You can use length-delimited streams to distinguish between subsequent messages, and design your own container format to provide message definitions for your data stream.

It is not uncommon for non-ROS teams to convert their data in custom recording formats into bag files, so they can load their data with ROS analysis tools.

Summary

There are many tradeoffs to consider before your robot even starts collecting data. What we covered is not a one-size-fits-all approach, nor is it an exhaustive list of all the ways you can optimize data collection. Knowing your robot's working environment, your constraints, and your data's usage is paramount to assessing how you should tackle these questions.


Today, we're excited to announce that Foxglove Studio now supports loading ROS 2 bags for playback and visualization. With v0.13.0, a new data source option (ROS 2 Bag Folder (local)) will be available in the sidebar’s Connection tab. With this option, you can load rosbag2 files for playback.

Current limitations and future plans

While we’re excited for ROS 2 to reach true feature parity with ROS 1 support in Studio, it’s important to note several important differences in the features as they stand today.

For one, since ROS 2 bags currently do not contain the message definitions that were used to generate the bag files, Studio is currently unable to load custom message definitions. However, it will still support standard ROS message definitions (e.g. std\_msgs/ColorRGBA, geometry\_msgs/Pose, etc.) as it does today. With that said, the Foxglove team is working closely with ROS developers to ensure that future ROS 2 bags will include the message definitions used to generate them. This will allow third-party tools like Studio to decode and inspect ROS 2 data.

Studio also does not yet support preloading ROS 2 data. After initializing a data source connection, Studio typically preloads data to pre-populate panels like the Plot panel. Adding this same support for ROS 2 sources is high on our list of priorities – join our Slack community or sign up for our newsletter to be the first to know when this functionality lands.

preloading data in plot panel
Preloading ROS 1 data into the Plot panel before starting playback.

We're also working on allowing users to connect to live ROS 2 data sources. In the same way that users can currently connect to a live ROS 1 stack in Studio via a native connection or a Websocket bridge, we want our users to be able to listen live to their ROS 2 nodes.

Share your feedback

Foxglove Studio support for ROS 2 is still a work in progress, so let us know what improvements or new features you'd like to see!

We also know not all roboticists use ROS as their robotics platform of choice, so we’d love to hear your thoughts on how we should expand our supported data sources in the future.

To share your feedback, join our Slack channel, get involved in our GitHub community, or message us on Twitter.

Check out our docs for details on all our currently supported data sources.


Announcing ROS 2 Bag Support

Read more:

Ready to get started?Download today on Linux, Windows, or macOS.