Today, we're excited to announce that Foxglove Studio now supports loading ROS 2 bags for playback and visualization. With v0.13.0, a new data source option (ROS 2 Bag Folder (local)) will be available in the sidebar’s Connection tab. With this option, you can load rosbag2 files for playback.

Current limitations and future plans

While we’re excited for ROS 2 to reach true feature parity with ROS 1 support in Studio, it’s important to note several important differences in the features as they stand today.

For one, since ROS 2 bags currently do not contain the message definitions that were used to generate the bag files, Studio is currently unable to load custom message definitions. However, it will still support standard ROS message definitions (e.g. std\_msgs/ColorRGBA, geometry\_msgs/Pose, etc.) as it does today. With that said, the Foxglove team is working closely with ROS developers to ensure that future ROS 2 bags will include the message definitions used to generate them. This will allow third-party tools like Studio to decode and inspect ROS 2 data.

Studio also does not yet support preloading ROS 2 data. After initializing a data source connection, Studio typically preloads data to pre-populate panels like the Plot panel. Adding this same support for ROS 2 sources is high on our list of priorities – join our Slack community or sign up for our newsletter to be the first to know when this functionality lands.

preloading data in plot panel
Preloading ROS 1 data into the Plot panel before starting playback.

We're also working on allowing users to connect to live ROS 2 data sources. In the same way that users can currently connect to a live ROS 1 stack in Studio via a native connection or a Websocket bridge, we want our users to be able to listen live to their ROS 2 nodes.

Share your feedback

Foxglove Studio support for ROS 2 is still a work in progress, so let us know what improvements or new features you'd like to see!

We also know not all roboticists use ROS as their robotics platform of choice, so we’d love to hear your thoughts on how we should expand our supported data sources in the future.

To share your feedback, join our Slack channel, get involved in our GitHub community, or message us on Twitter.

Check out our docs for details on all our currently supported data sources.


Whether you just got started with ROS or have a robotic duck roaming around your house, giving your robot more sensor inputs can be a great next step to take in your robotics journey, and adding a camera to your robot can be a simple and powerful way to do just that.

Teaching a robot to understand what it's seeing through a camera might sound complicated, but it doesn't have to be! As machine learning has become popular, it's also become much more accessible to non-experts, and there are now plenty of off-the-shelf toolkits that make it simple to analyze data and detect patterns.

When analyzing your sensor data, it can be helpful to see your outputs visually. A common approach for camera images is to produce an image that has been annotated with relevant markers. However, there are several limitations to drawing directly on the image – markers can cover pixels in the original image, appear blurry on small images, and be costly to store.

You can tackle these issues by using image markers, a kind of vector graphics that Studio can draw on top of the image. Studio even enables you to easily turn them on or off in your layout's Image panel.

In this post, we'll set up a camera, run a face detection algorithm on the images, and use image markers in Studio to visualize the results. (If you're not yet familiar with ROS nodes and topics, try reading The Building Blocks of ROS first — we'll be using them in this tutorial.)

Setting up a camera node

This post will assume you have a working ROS 1 installation on your computer. I'm using ROS Noetic on Ubuntu 20.04, running in a virtual machine. If you need to set up ROS, follow these installation instructions.

The first thing we need is a camera connected to our robot. For the purposes of this demo, I'm using my laptop's webcam, but you might be using another camera that plugs into a USB port. Let's install a ROS package called usb\_cam to interface with the camera and produce ROS sensor\_msgs/Image messages:



usb\_cam comes with a launch file that will start up the camera driver and an image viewer, so we can see that our camera is working:



image\_view window displaying images from the camera

(If this doesn't work for you, you might need to modify the usb\_cam parameters to select the correct camera device, pixel format, and other settings. You can create your own copy of the usb\_cam-test.launch file and adjust the parameters there.)

Looking deeper with Foxglove Studio

Behind the scenes, this has launched a ROS node that publishes sensor\_msgs/Image messages on a new topic. To get a better idea of how this works, let's hook up Foxglove Studio and peek at the ROS graph. First we'll connect to a ROS data source.

First, choose Open connection from the data source selection dialog:
connecting to ROS in Foxglove Studio

Select ROS 1 as the connection type. Since I'm using Ubuntu inside a virtual machine, I need to type an IP address where I can reach the Ubuntu guest before opening the connection.

connecting to ROS in Foxglove Studio

Next we'll create a new layout, add an Image panel, and display the /usb\_cam/image\_raw topic:

viewing camera images in an Image panel

Let's also open up a Topic Graph panel to visualize the flow of data in the system. Here, blue rectangles represent nodes, and pink diamonds represent topics. You can see how raw images, camera parameters, and log messages flow between the /usb\_cam driver node, the /image\_view window, and of course Studio itself.

visualizing the flow of data with a Topic Graph panel

Creating an image processor node in Python

Now that we've set up a camera and understood how data flows between nodes in the graph, we can begin processing images. We'll use rospy to write a simple ROS node in Python.

First, let's set up a node to receive camera images from the driver in a new my\_node.py file:



Make sure roslaunch usb\_cam usb\_cam-test.launch is still running, before running your script in a new terminal window:



By adding a Log panel to our current layout, we can view the output from rospy.loginfo directly in Studio. We can also confirm that /my\_node has appeared in the graph, and is receiving data from /usb\_cam/image\_raw.

viewing log messages in a log panel

The next step is detecting faces! We'll use a library called Dlib, which provides a face detector out of the box:



We can call dlib.get\_frontal\_face\_detector() to create a face detector. The face detector only accepts grayscale images as input, so when we receive an image in our image\_callback, we'll first use cv\_bridge to convert it to an OpenCV image, then convert that to grayscale using cvtColor.



Finally, we'll run the face detector, which returns a list of rectangles where faces were found. The callback logs a message indicating how many faces were detected — moving your face around on camera should affect the node's output in the Log panel.



log messages showing face detections

Now that we've determined whether a camera image contains faces, let's use image markers to see where these faces were detected.

Adding image markers

ROS provides a message called ImageMarker, which describes a visual annotation on an image. Since we might have a variable number of faces in the image, we'll need an array of markers. This isn't provided by default with ROS, but Foxglove has created our own ImageMarkerArray data type. You can install it by running:



Now let's update our code to publish an image marker for each face in the image! We're going to publish a new topic called /usb\_cam/face\_markers. (We use the same /usb\_cam namespace prefix as the camera driver to indicate that the face markers use the same coordinate space as the camera driver's images.)

In order to do this, we need to reorganize our code a little bit, moving the subscriber and publisher into a class so they can share state via self.



Then in our image\_callback, we'll publish an ImageMarkerArray every time we receive an image from the camera:



Finally, we'll add Node() to our def main().

Back in Studio, the Image panel now shows our new topic in the dropdown menu under "markers".

dropdown menu to enable image markers

When I turn it on, a box appears around my face — the detector is working! The /usb\_cam/face\_markers topic also appears in the graph.

camera image overlaid with a box around the detected face

Predicting facial features

Let's take it one step further. Not only does Dlib provide a face detector — it also provides a facial landmark predictor, which understands the location of several common points on a face. To use it, we'll need a trained model file, which you can download from this link here (99MB decompressed). Information about the model is provided in the imutils package, so let's install that too.



We can initialize the predictor with dlib.shape\_predictor(), passing it the path to the data file we just downloaded and extracted.

Once we've detected a face is present in a rectangular region, we'll run the predictor on that region to get a series of 68 points. Then we'll interpret those points using imutils.face\_utils.FACIAL\_LANDMARKS\_68\_IDXS, which groups them into 8 facial features (mouth, inner\_mouth, right\_eyebrow, left\_eyebrow, right\_eye, left\_eye, nose, and jaw).



We'll upgrade our image markers by drawing a line of a unique color for each facial feature.



Then, in our image\_callback:



And that's all for this tutorial — thanks for following along! Now that your robot can analyze images, and you can visualize the results with image markers, I hope it will be easier for you to add amazing new functionality to your robot.

face with lines drawn around eyebrows, eyes, nose, mouth, and jaw

Full sample code for this post is available on GitHub.

For more information, see the following reference materials:

usb\_cam documentation

Dlib documentation

Detect eyes, nose, lips, and jaw with dlib, OpenCV, and Python by Adrian Rosebrock

If you have questions or just want to chat about robotics visualization, join us in our Slack community.


We sat down with Steve Crowe, the editorial director of The Robot Report, to pick his brain on the state of robotics. As someone who’s covered the industry for about a decade, Steve has seen a lot of ideas (and companies) come and go. We discuss the major industry trends he’s reported on over the last decade, and where he sees the field going in the next ten years.

As the editorial director at The Robot Report, you write about the business and applications of robotics. Can you tell us a little bit about what sparked your initial foray into the world of robotics?

I didn't go to school for robotics or have any interest in the industry before my career path steered me in that direction. I’m a trained journalist, and the publishing company I was working at happened to own a robotics brand, Robotics Business Review. After several years of building out another brand, I was asked to steer that ship to accelerate its online growth. Then in early 2018, my colleague Dan Kara reached out to me as he was growing the robotics editorial team at WTWH Media – that’s how I got started working on The Robot Report.

It’s now been about a decade since I started covering robotics – it’s a ton of fun. I'm located in Boston, which is certainly in the top three robotics clusters in the world, along with California and Austin. It's a great community, a pretty small community of folks.

What does a typical day for you look like?

If I have to summarize what I do, it's educating the robotics community on how to create better, more intelligent systems. Whatever the medium is, whatever the format of the content is, we're trying to build a relationship with stakeholders in the robotics community and gain their trust. We show them that they can come to us to learn about the innovations driving the industry forward. They can come to our events to network with each other, or to learn about the products that will help them create the next generation of commercial robotics systems.

As far as what my typical day looks like, it depends. This week we announced the winners for our annual RBR50 Robotics Innovation Awards. I also moderated a webinar with Rich Mahoney, the former director of SRI International and current VP of Research at Intuitive, the world’s leading robotic assisted surgical company.

This is on top of producing engaging content across our 4 websites – we’ve built a diverse portfolio and grown the group pretty significantly. Our flagship brand The Robot Report covers engineering news, emerging technologies, and investments and acquisitions. Collaborative Robotics Trends teaches companies about the benefits of collaborative robotic arms. Robotics Business Review, the original robotics brand I’d worked on, provides breaking news and informed analysis on the robotics and automation sectors. And finally, Mobile Robot Guide educates companies about the benefits of autonomous mobile robots (AMRs), automated guided vehicles (AGVs), and other mobile robots in logistics and supply chain environments.

How have you seen the industry evolve in the last several years? Are there any particularly interesting industry trends that you’ve seen emerge?

For so long, robotics referred to the industrial robot arms at manufacturing plants – particularly automotive plants. As enabling technologies came down in price and industries were educated about the benefits of adopting these systems, robotics has expanded out of those plants and research labs into a lot of different areas.

One industry that’s attracting more players is food services. Miso Robotics, for example, is automating kitchen operations within fast food restaurants. With COVID and other airborne and foodborne illnesses, you’re seeing more companies trying to make the food preparation process safer.

Miso Robotics' Flippy is an AI-powered kitchen assistant boosting performance and safety in restaurants.

Construction and agriculture are two markets that have often been talked about as being ripe for automation, and now it seems to be happening. Several companies have been developing picking robots to help farms with the harvesting process. They’ve built pretty sophisticated AI that uses vision algorithms to understand when different types of produce are ready to be picked, and what exact mechanism is needed to harvest them.

E-commerce is another area where robotic systems are currently gaining a lot of traction. Mobile robots that transport items throughout a facility are one of the hottest markets right now. It was already growing at a pretty good rate before 2020, but the pandemic introduced more people to online shopping. The only way for many e-commerce players to keep pace with the demand was to automate parts of their operation. Amazon’s own AMRs, for example, have been vital in helping them keep up with the demand.

What do you see as the industry’s biggest technical challenge?

I'm pretty excited to see how the legged robotics market develops – Boston Dynamics and ANYbotics are interesting ones to watch. There's been a lot of skepticism about the benefits of legged robots – they’ve been under research and development for decades, but nobody ever really thought that they were commercially viable or that there was a compelling use case for them. But within the last year, several quadruped robots have been commercialized, the most well known one being Boston Dynamic’s Spot. Humans have teleoperated this system to inspect sites around the Chernobyl nuclear power plant disaster for radiation. It can even be set up to perform autonomous missions to keep humans out of harm's way – a great use case for robotics.

Boston Dynamics' Spot allows users to automate routine inspection tasks and capture data safely and accurately.

As for bipedal robots, Agility Robotics has a fascinating one named Digit. They've studied animals – in particular, ostriches – to inform their locomotion design and planning algorithms. With Digit, they're targeting not last mile delivery, but something like last hundred feet delivery. They've done some crazy tests with partners like Ford where these robots get out of the car, deliver a package to a door, and then reload themselves into the car.

Agility Robotics has partnered with Ford to test doorstep deliveries with its bipedal robot Digit.

These robots have very sophisticated locomotion capabilities – they can climb stairs and navigate over all sorts of terrain – giving them a major advantage over wheeled robots. The construction, oil and gas, mining industries are all trying to figure out how to put these platforms to use. We’ll have to wait and see whether they prove themselves useful outside the lab.

What are the major business hurdles that robotics startups face today?

A lot of this stuff is hardware-centric – it just takes so much capital to develop these systems that don't have a tremendous amount of customers. For example, Locus Robotics is an AMR company that’s been around for about 7 years and recently raised $150 million in Series E funding. They’ve said that their next step is to IPO, but here’s the thing – they have roughly 40 actual customers. It takes an incredible amount of time and money to develop these things, to hire the incredibly smart and hardworking people to iterate on these systems. Building robots and gaining any traction is incredibly challenging.

Locus Robotics offer automated warehouse robots that increase productivity and order accuracy.

Interoperability is another hurdle that's talked about a lot now. Mobile robots developed by different companies often operate in roughly the same way, but they all have their own proprietary communication systems and operating systems. So there's an emerging problem with adopting heterogeneous fleets of robots from different manufacturers, since these robots must operate together in the same environment, but can’t communicate with each other on their own. If you have robots from 4 different vendors that you want to reprogram in any way, there’s a good chance you have to learn 4 different operating systems. It makes the whole operation much more challenging for the end user.

Thankfully, there's a lot of work being done around setting a standardized communication protocol. For example, READY Robotics is creating a universal operating system so that users only have to learn one approach, regardless of how many different robotics vendors they choose. Open Robotics, the group behind the Robot Operating System, has been collaborating with a hospital network in Singapore to develop a fleet management system that allows their healthcare robots communicate with each other. These robots were eventually able to recognize each other’s movements as they navigated hallway traffic and called elevators to travel between floors.

Open Robotics has been collaborating with Singapore’s Centre for Healthcare Assistive and Robotics Technology (CHART) to develop an open source infrastructure framework that simplifies cross-fleet collaboration.

Have there been any sectors of the industry that haven’t been able to live up to the hype?

There hasn’t been much progress in the consumer market. iRobot was very successful with its robot vacuums, but there really hasn't been much progress with other robots for your home. iRobot had been talking about a robot lawnmower for over a decade, and they finally teased it last year. But they eventually shelved it, citing fallout from the COVID-19 pandemic. Social robots have often been hyped as the next in-home consumer robot, but it's been a disaster of an industry with a few high-profile failures.

One consumer robot company that does seem to be having some success is Embodied. Their Moxie robot helps young kids build social-emotional skills, by recognizing what a particular child is struggling with and then serving up content that might be beneficial to that child. I think they’ve been successful, because they weren't trying to be an all consumer companion robot – they honed in on one specific application.

Moxie uses play-based activites created by child development experts to promote children's mental well-being.

What are some of your favorite robotics projects out there in the wild today?

I love self-driving cars. I love following Cruise, Waymo, and the other leading autonomous vehicle developers. It's just a fascinating concept and a super challenging problem from an engineering standpoint. There are so many people who are killed in careless car accidents, so if these systems could be scaled and deployed commercially all over the world, that would be tremendously valuable to society.

There are also so many applications for an autonomous vehicle outside of a robo taxi. I've seen use cases where autonomous luggage cart vehicles operate on fixed repeatable routes – at a certain terminal, multiple times a day. There’s opportunity there.

British Airways started trials of driverless luggage carts at Heathrow Airport in 2019 to speed up loading carts.

And I’ve said it before, but the realm of legged robotics is super exciting as well. I'm sure you’ve seen the viral videos of Boston Dynamics’ two-legged robot doing gymnastics routines and going for a jog. But we have to see how that progresses. That's what gets me fired up – seeing these challenges that are still left for us to solve.

What advice do you have for someone who's interested in entering the robotics field, whether as an entrepreneur, engineer, or investor?

If you’re starting a robotics startup – don't create a solution that's looking for a problem. Don't just go build a supply chain robot because you know that’s a really hot market right now. Find a problem that has yet to be solved or automated, and figure out how to do that. So many companies we've reported on have gone out of business, because there was never a great use case for what they built. They took a variety of enabling technologies that had improved and come down in price over the years, and fused them together. But the problem it was solving was never really there. They were so concerned about building the coolest product with the coolest tech that they forgot about their end user.

The startups that are still around and getting customers and funding have that one thing in common – they really understand the problem that their user is trying to solve. An example is 6 River Systems, which was acquired by Shopify for almost half a billion dollars. Its executives may have had some robotics knowledge, but their expertise really lay in the logistics business. They had great firsthand knowledge of the challenges that supply chain operators face on an everyday basis. They could think clearly about the tools and system that they could develop to help these operators overcome those challenges.

6 River System's Chuck uses machine learning and artificial intelligence to fulfill orders more efficiently.

Also, talk to as many people as you can. Find people who have the domain knowledge you want to learn, and pick their brains – that's really the key. It’s a pretty small community, and most people that I've met in my 10 years in the industry are super passionate about what they do. They’d love to talk to you about what they're working on, or about the business and technological challenges they face on a regular basis. They want nothing but to move this industry forward – to get more robots out in the world helping people overcome daily challenges.

A lot of the general public is afraid of these systems, as they start to appear out there in the real world. Sci-fi over the years has probably done a major disservice to the industry. But so many folks in the industry want to reverse that misconception and educate people about how these systems actually work and how they can help us tackle daily challenges.

To keep up with Steve’s latest work, you can follow him on Twitter or read his pieces on The Robot Report.

This interview has been edited and condensed for clarity.


Spotlight: Steve Crowe, Editorial Director of The Robot Report

Read more:

Ready to get started?Download today on Linux, Windows, or macOS.