
Whether you just got [started with ROS](/blog/running-your-first-ros-node) or have a [robotic duck](/blog/building-and-visualizing-your-first-robot) roaming around your house, **giving your robot more sensor inputs can be a great next step to take in your robotics journey**, and adding a camera to your robot can be a simple and powerful way to do just that.

**Teaching a robot to understand what it's seeing through a camera might sound complicated, but it doesn't have to be!** As machine learning has become popular, it's also become much more accessible to non-experts, and there are now plenty of off-the-shelf toolkits that make it simple to analyze data and detect patterns.

When analyzing your sensor data, it can be helpful to see your outputs visually. A common approach for camera images is to produce an image that has been annotated with relevant markers. However, there are several limitations to drawing directly on the image – markers can cover pixels in the original image, appear blurry on small images, and be costly to store.

You can tackle these issues by using **image markers**, a kind of vector graphics that Studio can draw on top of the image. Studio even enables you to easily turn them on or off in your layout's [_Image_](/docs/panels/image) panel.

In this post, we'll set up a camera, run a face detection algorithm on the images, and use image markers in Studio to visualize the results. (If you're not yet familiar with ROS _nodes_ and _topics_, try reading [The Building Blocks of ROS](/blog/the-building-blocks-of-ros) first — we'll be using them in this tutorial.)

# Setting up a camera node

_This post will assume you have a working ROS 1 installation on your computer. I'm using ROS Noetic on Ubuntu 20.04, running in a virtual machine. If you need to set up ROS, follow these [installation instructions](https://wiki.ros.org/Installation/Ubuntu)._

The first thing we need is a camera connected to our robot. For the purposes of this demo, I'm using my laptop's webcam, but you might be using another camera that plugs into a USB port. Let's install a ROS package called [usb_cam](https://wiki.ros.org/usb_cam) to interface with the camera and produce ROS [`sensor_msgs/Image`](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) messages:

```bash
$ sudo apt install ros-noetic-usb-cam
```

`usb_cam` comes with a [launch file](https://wiki.ros.org/roslaunch) that will start up the camera driver and an image viewer, so we can see that our camera is working:

```bash
$ roslaunch usb_cam usb_cam-test.launch
```

![image_view window displaying images from the camera](/images/blog/annotate-your-robots-camera-images-with-image-markers/camera-test.png)

(If this doesn't work for you, you might need to modify the `usb_cam` parameters to select the correct camera device, pixel format, and other settings. You can create your own copy of the [`usb_cam-test.launch`](https://github.com/ros-drivers/usb_cam/blob/develop/launch/usb_cam-test.launch) file and adjust the parameters there.)

# Looking deeper with Foxglove Studio

Behind the scenes, this has launched a ROS node that publishes `sensor_msgs/Image` messages on a new topic. To get a better idea of how this works, let's hook up [Foxglove Studio](/download) and peek at the ROS graph. First we'll [connect to a ROS data source](/docs/connection/data-sources). Since I'm using Ubuntu inside a virtual machine, I need to type an IP address where I can reach the Ubuntu guest:

![connecting to ROS in Foxglove Studio](/images/blog/annotate-your-robots-camera-images-with-image-markers/ros-connection.png)

Next we'll create a new [layout](/docs/layouts), add an [_Image_](/docs/panels/image) panel, and display the `/usb_cam/image_raw` topic:

![viewing camera images in an Image panel](/images/blog/annotate-your-robots-camera-images-with-image-markers/image-panel.png)

Let's also open up a [_Topic Graph_](/docs/panels/topic-graph) panel to visualize the flow of data in the system. Here, blue rectangles represent nodes, and pink diamonds represent topics. You can see how raw images, camera parameters, and log messages flow between the `/usb_cam` driver node, the `/image_view` window, and of course Studio itself.

![visualizing the flow of data with a Topic Graph panel](/images/blog/annotate-your-robots-camera-images-with-image-markers/topic-graph-panel.png)

# Creating an image processor node in Python

Now that we've set up a camera and understood how data flows between nodes in the graph, we can begin processing images. We'll use [rospy](https://wiki.ros.org/rospy) to write a simple ROS node in Python.

First, let's set up a node to receive camera images from the driver in a new `my_node.py` file:

```py
import rospy
from sensor_msgs.msg import Image

def image_callback(msg: Image):
    rospy.loginfo("Got an image!")

def main():
    rospy.init_node("my_node")
    rospy.Subscriber("/usb_cam/image_raw", Image, image_callback, queue_size=1)
    rospy.spin()

if __name__ == "__main__":
    main()
```

Make sure `roslaunch usb_cam usb_cam-test.launch` is still running, before running your script in a new terminal window:

```bash
$ python3 my_node.py
```

By adding a [_Log_](/docs/panels/log) panel to our current layout, we can view the output from `rospy.loginfo` directly in Studio. We can also confirm that `/my_node` has appeared in the graph, and is receiving data from `/usb_cam/image_raw`.

![viewing log messages in a log panel](/images/blog/annotate-your-robots-camera-images-with-image-markers/node-step1.png)

The next step is detecting faces! We'll use a library called [Dlib](http://dlib.net), which provides a face detector out of the box:

```bash
$ pip3 install dlib
```

We can call `dlib.get_frontal_face_detector()` to create a face detector. The face detector only accepts grayscale images as input, so when we receive an image in our `image_callback`, we'll first use [cv_bridge](https://wiki.ros.org/cv_bridge) to convert it to an OpenCV image, then convert that to grayscale using `cvtColor`.

```py
import cv2
import dlib
from cv_bridge import CvBridge

cv_bridge = CvBridge()
face_detector = dlib.get_frontal_face_detector()
```

Finally, we'll run the face detector, which returns a list of rectangles where faces were found. The callback logs a message indicating how many faces were detected — moving your face around on camera should affect the node's output in the _Log_ panel.

```py
def image_callback(msg: Image):
    # Convert the ROS Image to a grayscale OpenCV image
    cv_img = cv_bridge.imgmsg_to_cv2(msg)
    grayscale_img = cv2.cvtColor(cv_img, cv2.COLOR_BGR2GRAY)

    # Run the face detector on the grayscale image
    rects = face_detector(grayscale_img, 0)
    rospy.loginfo(f"Detected {len(rects)} faces")
```

![log messages showing face detections](/images/blog/annotate-your-robots-camera-images-with-image-markers/faces-detected.png)

Now that we've determined whether a camera image contains faces, let's use image markers to see where these faces were detected.

# Adding image markers

ROS provides a message called [ImageMarker](https://docs.ros.org/en/melodic/api/visualization_msgs/html/msg/ImageMarker.html), which describes a **visual annotation on an image**. Since we might have a variable number of faces in the image, we'll need an array of markers. This isn't provided by default with ROS, but Foxglove has created our own [ImageMarkerArray](https://docs.ros.org/en/noetic/api/foxglove_msgs/html/msg/ImageMarkerArray.html) data type. You can install it by running:

```bash
$ sudo apt install ros-noetic-foxglove-msgs
```

Now let's update our code to publish an image marker for each face in the image! We're going to **publish a new topic called `/usb_cam/face_markers`**. (We use the same `/usb_cam` namespace prefix as the camera driver to indicate that the face markers use the same coordinate space as the camera driver's images.)

In order to do this, we need to reorganize our code a little bit, moving the subscriber and publisher into a [class](https://docs.python.org/3/tutorial/classes.html) so they can share state via `self`.

```py
from std_msgs.msg import ColorRGBA
from foxglove_msgs.msg import ImageMarkerArray
from visualization_msgs.msg import ImageMarker
from geometry_msgs.msg import Point

class Node:
    pub_markers: rospy.Publisher

    def __init__(self):
       rospy.init_node("my_node")
       rospy.Subscriber("/usb_cam/image_raw", Image, self.image_callback, queue_size=1)

       # On initialization, set up a Publisher for ImageMarkerArrays
       self.pub_markers = rospy.Publisher(
           "/usb_cam/face_markers", ImageMarkerArray, queue_size=1
       )
       rospy.spin()
```

Then in our `image_callback`, we'll publish an `ImageMarkerArray` every time we receive an image from the camera:

```py
markers = ImageMarkerArray()
for rect in rects:
    markers.markers.append(
        # Display each face as a cyan rectangle
        ImageMarker(
            header=msg.header,
            type=ImageMarker.POLYGON,
            outline_color=ColorRGBA(0, 255, 255, 1),
            points=[
                Point(rect.left(), rect.top(), 0),
                Point(rect.right(), rect.top(), 0),
                Point(rect.right(), rect.bottom(), 0),
                Point(rect.left(), rect.bottom(), 0),
            ],
        )
    )
self.pub_markers.publish(markers)
```

Finally, we'll add `Node()` to our `def main()`.

Back in Studio, the _Image_ panel now shows our new topic in the dropdown menu under "markers".

![dropdown menu to enable image markers](/images/blog/annotate-your-robots-camera-images-with-image-markers/select-image-markers.png)

When I turn it on, a box appears around my face — the detector is working! The `/usb_cam/face_markers` topic also appears in the graph.

![camera image overlaid with a box around the detected face](/images/blog/annotate-your-robots-camera-images-with-image-markers/face-box.png)

# Predicting facial features

Let's take it one step further. Not only does Dlib provide a face _detector_ — it also provides a _facial landmark predictor_, which understands the location of several common points on a face. To use it, we'll need a trained model file, which you can download from [this link here](http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2) (99MB decompressed). Information about the model is provided in the `imutils` package, so let's install that too.

```bash
$ wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
$ bunzip2 shape_predictor_68_face_landmarks.dat.bz2
$ pip3 install imutils
```

We can initialize the predictor with `dlib.shape_predictor()`, passing it the path to the data file we just downloaded and extracted.

Once we've detected a face is present in a rectangular region, we'll run the predictor on that region to get a series of 68 **points**. Then we'll interpret those points using `imutils.face_utils.FACIAL_LANDMARKS_68_IDXS`, which groups them into 8 facial **features** (`mouth`, `inner_mouth`, `right_eyebrow`, `left_eyebrow`, `right_eye`, `left_eye`, `nose`, and `jaw`).

```py
from imutils.face_utils import FACIAL_LANDMARKS_68_IDXS
```

We'll upgrade our image markers by drawing a line of a unique color for each facial feature.

```py
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
COLORS = [
   ColorRGBA(0, 0, 0, 1),
   ColorRGBA(0, 0, 255, 1),
   ColorRGBA(0, 255, 0, 1),
   ColorRGBA(0, 255, 255, 1),
   ColorRGBA(255, 0, 0, 1),
   ColorRGBA(255, 0, 255, 1),
   ColorRGBA(255, 255, 0, 1),
   ColorRGBA(255, 255, 255, 1),
]
```

Then, in our `image_callback`:

```py
for rect in rects:
    # Run the predictor, which returns a list of 68 facial landmarks as (x,y) points
    points = predictor(grayscale_img, rect).parts()

    # Draw a line around each face region
    for region_idx, (name, (start_idx, end_idx)) in enumerate(
        FACIAL_LANDMARKS_68_IDXS.items()
    ):
        # Connect the points for each region in a loop, except for the jaw
        type = ImageMarker.LINE_STRIP if name == "jaw" else ImageMarker.POLYGON
        markers.markers.append(
            ImageMarker(
                header=msg.header,
                type=type,
                points=[Point(p.x, p.y, 0) for p in points[start_idx:end_idx]],
                outline_color=COLORS[region_idx % len(COLORS)],
            )
        )
```

And that's all for this tutorial — thanks for following along! Now that your robot can analyze images, and you can visualize the results with image markers, I hope it will be easier for you to add amazing new functionality to your robot.

![face with lines drawn around eyebrows, eyes, nose, mouth, and jaw](/images/blog/annotate-your-robots-camera-images-with-image-markers/face-features.png)

Full sample code for this post is available [on GitHub](https://gist.github.com/jtbandes/938a6d39308f145ebc59b4b1c5c8a11d).

For more information, see the following reference materials:

- [`usb_cam` documentation](https://wiki.ros.org/usb_cam)
- [Dlib documentation](http://dlib.net)
- [Detect eyes, nose, lips, and jaw with dlib, OpenCV, and Python](https://www.pyimagesearch.com/2017/04/10/detect-eyes-nose-lips-jaw-dlib-opencv-python/) by Adrian Rosebrock

If you have questions or just want to chat about robotics visualization, join us in our [Slack community](/join-slack).



The typical cycle for robotics development usually starts with a deployed robot collecting data on a real-world excursion – whether that world is [a warehouse](https://www.youtube.com/watch?v=RFNrPXSGRIM), [a public road](https://www.youtube.com/watch?v=Pa7xU7PW-oM), or [the ocean's "twilight zone"](https://www.youtube.com/watch?v=5hjZtBvsmVY).

Typically, this data is offloaded into storage at the end of the excursion, and then analyzed to determine the improvements the robot's software requires. The necessary changes are made, the new and improved software is deployed onto the robot, and that robot is sent back out into the world to collect more data.

And so the cycle continues.

In the robotics community, there is no shortage of tools that can help you inspect, analyze, and visualize your data. From native Robot Operating System ([ROS](https://www.ros.org/)) tools like [rqt](http://wiki.ros.org/rqt) and [rviz](http://wiki.ros.org/rviz), to newer generations of tools like [PlotJuggler](https://github.com/facontidavide/PlotJuggler) and [Foxglove Studio](https://foxglove.dev/), there are a multitude of ways you can inspect the inner workings of your robot's brain.

**But how do you get to a point where you have robotics data to visualize in the first place?** What technical decisions do you need to make, and what steps do you need to take, to collect your data? How do you weigh all the possible strategies for this vital step?

Though there isn't an industry-wide standard or one right answer, **there are basic trade-offs to consider when deciding how to collect your robotics data**. The decisions you make will depend on your project's specific goals and constraints. By having a list of options to weigh, you can make more informed choices about the best data strategy for your project.

# Data collection approaches

Before you can store or analyze any data, you need to collect it. There are several ways you can do this:

- **Local disk recording**
- **Live telemetry**
- **Hybrid approach**

All these methods have their pros and cons – **the frequency with which your robot needs to send real-time data, the volume of data it records on a typical excursion, and its offload bandwidth will all inform which method you should use**.

Your guiding principle should be finding the strategy that will most consistently return uncompromised data from the dynamic real-world environments that your robot navigates. Your data will guide future iterations of your robot, so the method that allows you to collect and learn from your data most reliably is often a good place to start.

## Local disk recording

Robots are often deployed to areas that are inconvenient or dangerous for humans to go, and consequently cannot rely on a consistent internet connection to safely offload data. **These robots with inconsistent or limited in-the-field connectivity can write their data to local storage to ensure that it can be collected and retrieved properly**.

![robot in hazardous environment](/images/blog/recording-your-robotics-data/local-disk.jpeg)

**Your robot's disk space** will also affect how you collect data. With limited space, your robot must be more discerning about what data to record and what data to throw away. Running more business logic on the robot itself will conserve disk space – in this scenario, only the lighter-weight derived insights would be recorded to disk. While this “stretches out” disk space by writing a "lower resolution" version of your data, processing raw data live on your device will impact your robot's real-time performance.

Finally, **the importance of real-time feedback** will determine how you record your data. If your data does not need to be referenced until the robot is back from the field, purely local disk recording is ideal. However, if you need to monitor your robots in real-time, local disk recording would not be able to address this requirement.

## Live telemetry

**If it is crucial for you to know at any given time where your robot is, your robot must be able to offload data to a storage solution – like the cloud or your laptop – as it actively collecting it**.

![autonomous cars](/images/blog/recording-your-robotics-data/live-telemetry.jpeg)

Unlike local disk recording, which is blissfully independent of an internet connection, **real-time offload requires a consistent and reliable connection to your robot – either wirelessly or via a physical cable**. Since network connections can be unreliable, your robot can use in-memory buffering to avoid losing vital data whenever connectivity drops, transmitting these stored messages once a stable connection returns.

Due to bandwidth constraints, live telemetry is usually reserved for a small subset of your data. However, **real-time offload of full-resolution data can be particularly useful in controlled scenarios**, when your internet connection is reliable or when the scope of your task is within certain limits – for example, when you are conducting a sanity check on your robot or calibrating its sensors at homebase before sending it off into the field.

## Hybrid approach

Both local disk recording and live telemetry can be present at once on a given robot – some data can be analyzed asynchronously after the robot has returned, and other data can be immediately transmitted for real-time monitoring.

**Given that these approaches are useful in different scenarios, you can mix and match local disk recording and live telemetry for subsets of your robotics data**. Ultimately, the environment that your robot navigates, the type of data it wants to collect, and its goals while out in the field will all determine how you record your data.

**A hybrid approach is especially useful when you want both real-time insight into your fleet and high resolution data to diagnose potential failures**. With live telemetry, you can monitor your fleet's location and current status. In the event of a failure, you can upload the full fidelity data recorded on the robot's local hard disk to review it for diagnosis. In short, the hybrid approach navigates the trade-off between how much local disk space you have (local disk recording), and how much data you can reliably offload via your available bandwidth (live telemetry).

# Recording formats

Knowing how to record your robotics data is only half the battle. Another equally important question to address is the format in which to record this data.

There are several main criteria to consider when weighing data format options. Namely, you will want to choose a format that makes your data:

- **Conducive to analysis**
- **Trivial to recover in case of hardware or software failure**
- **Possible to record with low overhead**
- **Easy to offload**
- **Self-describing**

**Different data formats facilitate different types of analysis**. Choose a format while keeping in mind the type of analysis you may want to perform – whether it's visually playing back a recorded session, or using SQL to query petabytes of recordings.

**Your data should be stored in a format that makes it easy to recover**. For example, in the event of an unexpected shutdown or accident, the data your robot has already recorded should be preserved and remain uncompromised.

**Your robot's recording process should be IO and CPU efficient**. The volume of data you write and its compressibility will all affect performance and offloading.

Finally, **your data should contain self-describing schemas for easier deserialization and compression**. This enables you to decode your data without having to reference separate interface definition files, which may change frequently.

In short, choose the data format that allows your robot to record and offload its data for future analysis – all within its performance and bandwidth constraints. Let's cover your options, based on your robotics framework.

## ROS 1 robots

ROS is currently the industry standard for robotics data recording, mainly because it allows you to get up and running quickly, and easily integrate features that would be very time and labor-intensive to build yourself – i.e. modules for computer vision, SLAM, etc. This allows you to leverage the expertise of your peers, and makes collaborative robotics development extremely easy.

ROS allows you to write the [messages](http://wiki.ros.org/Messages) published by your [nodes](http://wiki.ros.org/Nodes) throughout the course of a robot's excursion to a bag file. [Bag files](http://wiki.ros.org/Bags) store ROS messages in a format that can then be played back, visualized, and analyzed – along with message definitions for all these messages. Many tools like [`rqt_bag`](http://wiki.ros.org/rqt_bag) and [`rosbag`](http://wiki.ros.org/rosbag) have been written to help you record, play back, and otherwise interact with these bag files. Writing your data to bag files optimizes recording and playback performance, because you avoid deserializing and reserializing your messages.

## ROS 2 robots

Whereas ROS 1 is a bundle of data formats, transport protocols, and reference implementations (C++ and Python), ROS 2 is a more generic C API with pluggable implementations. Even though ROS 2 has a default network transport protocol ([DDS](https://www.dds-foundation.org/what-is-dds-3/)) and a default bag storage implementation ([`sqlite3`](https://docs.python.org/3/library/sqlite3.html)), ROS 2 functions less as specific implementations and more as an abstract middleware interface for serialization, transport, and discovery. As such, [rosbag2](https://github.com/ros2/rosbag2) allows recording and replaying of different data formats via a flexible interface, improving communication over different networks.

While ROS 2 seems like a promising new direction for ROS, there are a few in-progress issues that could pose a problem to recording and playback. Most notably, [ROS 2 bags](https://github.com/ros2/rosbag2) currently do not store message definitions alongside their recorded data. This omission makes it more difficult for external tools like [Foxglove Studio](https://foxglove.dev) to inspect and analyze this data. With that said, storing message definitions in ROS 2 bags is an [open issue](https://github.com/ros2/rosbag2/issues/782) that the `rosbag2` team is actively tracking.

## Non-ROS robots

Outside of the ROS ecosystem, there are no clear-cut standards for data serialization formats.

Some teams use an open source data serialization format like [Apache Avro](https://avro.apache.org/), which optimizes for space and performance by storing data in compact binary format. Avro files are easy to process and read, because they store their data type schemas in JSON format. While JSON alone would get unwieldy for repeated fields across records, AVRO files use a direct mapping that makes it efficient for higher volume usage.

Other teams record a raw stream of [protobuf](https://github.com/protocolbuffers/protobuf) messages to a file. Protobuf, Google's mechanism for serializing structured data, transmits data as a compressed binary to conserve space, bandwidth, and CPU usage. Because there isn't a storage protocol, you will have to do some upfront work to get started. You can use [length-delimited streams](https://seb-nyberg.medium.com/length-delimited-protobuf-streams-a39ebc4a4565) to distinguish between subsequent messages, and design your own container format to provide message definitions for your data stream.

It is not uncommon for non-ROS teams to convert their data in custom recording formats into bag files, so they can load their data with ROS analysis tools.

# Summary

There are many tradeoffs to consider before your robot even starts collecting data. What we covered is not a one-size-fits-all approach, nor is it an exhaustive list of all the ways you can optimize data collection. Knowing your robot's working environment, your constraints, and your data's usage is paramount to assessing how you should tackle these questions.



Today, we're excited to announce that **Foxglove Studio now supports loading ROS 2 bags for playback and visualization**. With [v0.13.0](https://github.com/foxglove/studio/releases), a new data source option (`ROS 2 Bag Folder (local)`) will be available in the sidebar’s _Connection_ tab. With this option, you can load [rosbag2](https://github.com/ros2/rosbag2) files for playback.

# Current limitations and future plans

While we’re excited for ROS 2 to reach true feature parity with ROS 1 support in Studio, it’s important to note several important differences in the features as they stand today.

For one, since ROS 2 bags currently do not contain the [message definitions](http://docs.ros.org/en/rolling/Concepts/About-ROS-Interfaces.html?highlight=idl#message-description-specification) that were used to generate the bag files, **Studio is currently unable to load custom message definitions**. However, it will still support standard ROS message definitions (e.g. [std_msgs/ColorRGBA](http://docs.ros.org/en/api/std_msgs/html/msg/ColorRGBA.html), [geometry_msgs/Pose](http://docs.ros.org/en/api/geometry_msgs/html/msg/Pose.html), etc.) as it does today. With that said, **the Foxglove team is [working closely with ROS developers](https://github.com/ros2/rosbag2/issues/782) to ensure that future ROS 2 bags will include the message definitions used to generate them**. This will allow third-party tools like Studio to decode and inspect ROS 2 data.

**Studio also does not yet support preloading ROS 2 data**. After initializing a data source connection, Studio typically preloads data to pre-populate panels like the [Plot](/docs/panels/plot) panel. Adding this same support for ROS 2 sources is high on our list of priorities – [join our Slack community](/join-slack) or [sign up for our newsletter]() to be the first to know when this functionality lands.

![preloading data in plot panel](/images/blog/announcing-ros2-bag-support/preloading.png)
_Preloading ROS 1 data into the Plot panel before starting playback._

**We're also working on allowing users to connect to live ROS 2 data sources**. In the same way that users can currently connect to a live ROS 1 stack in Studio via a native connection or a Websocket bridge, we want our users to be able to listen live to their ROS 2 nodes.

# Share your feedback

Foxglove Studio support for ROS 2 is still a work in progress, so **let us know what improvements or new features you'd like to see**!

We also know not all roboticists use ROS as their robotics platform of choice, so **we’d love to hear your thoughts on how we should expand our supported data sources in the future**.

To share your feedback, join our [Slack channel](/join-slack), get involved in our [GitHub community](https://github.com/foxglove/studio), or message us on [Twitter](https://twitter.com/foxglovedev).

Check out [our docs](/docs/connection/data-sources) for details on all our currently supported data sources.


Announcing ROS 2 Bag Support

Current limitations and future plans

Share your feedback

Read more:

Ready to get started?Download today on Linux, Windows, or macOS.