
It’s hard to overstate the impact that robotics is poised to have on the world within our lifetimes. The industry has the potential to completely reimagine [our economies](https://mitsloan.mit.edu/ideas-made-to-matter/a-new-study-measures-actual-impact-robots-jobs-its-significant), [our politics](https://medium.com/@Brookings/will-robots-and-ai-take-your-job-the-economic-and-political-consequences-of-automation-80cfec27d757), and [our daily lives](https://www.zdnet.com/article/the-robots-are-coming-and-this-is-how-they-will-change-the-future-of-work/). From applications as wide sweeping as [self-driving](https://getcruise.com/) to as specific as [burger-flipping](https://misorobotics.com/cookright/), robots are already transforming how we navigate our lives.

Which makes it no surprise that so many people’s imaginations have been captivated by this field. The thought of “working in robotics” seems glamorous in a way that almost no other tech sector currently does. But despite the overwhelming interest, robotics maintains a long-standing reputation as an esoteric industry, accessible only to the select few who received formal training in artificial intelligence, machine learning, or hardware. **Talented applicants exclude themselves from the field all the time, based purely on their preconceived notions of what it takes to work on a robotics team.**

The reality, however, is that you don’t need to have this hyper-specific background or to work directly on some component of the robot itself to succeed in the field. **Robotics is an extremely cross-disciplinary endeavor, and it’s very likely that your professional experience in tech or any other industry translates to this exciting new frontier.**

Below are several ways you can cultivate a long-term career in robotics – even if you don’t have a master’s in computer vision or a PhD in AI.

## Front-end engineer

While writing algorithms to run on a robot may require specialized machine learning knowledge, that work makes up only half the battle. People often completely overlook the many tools that robotics engineers need to do their job at all.

That’s where developer tools come in. **[Robotics data visualizations](/studio), in particular, are crucial to helping teams analyze the data their robots gather.** With the help of web-based tools, roboticists can now dive into the inner workings of their robot and more deeply understand what it's seeing, thinking, and deciding while out in the world. Understanding their robot’s current strengths and weaknesses, in turn, helps them iterate more intelligently towards a production-ready robot.

![Foxglove Studio data visualization](/images/blog/how-to-work-in-robotics-when-youre-not-a-roboticist/front-end.png)

**As a front-end web developer working in the field, you’d be responsible for building these rich interactive interfaces for roboticists to debug their work.** [Foxglove Studio](/studio) is a prime example of a [web-based visualization app](https://studio.foxglove.dev/) that does exactly that. Because it portrays data in a digestible visual format – all from the comfort of a browser or desktop app window – it empowers engineers to diagnose problems and formulate possible solutions efficiently.

## Back-end engineer

Back-end development is one of the most relevant and in-demand skills for robotics teams today. **Robots generate huge amounts of data – while communicating with each other, with their operators, and with the world around them.** Operators need to manage fleets, assign tasks, and track issues. In turn, robots need to receive new tasks, share their plans, and provide results to operators. In order to deploy a successful fleet, teams not only perfect their robots’ capabilities, but also prepare an infrastructure for managing and housing this formidable amount of data.

**Data management is an often overlooked, but clearly significant, aspect of robotics development.** Ingesting and sifting through petabytes of data can be time-consuming, labor-intensive, and expensive. As time goes on, data accumulates, gets lost, or is unintentionally duplicated – making it increasingly difficult to know what information lives where. In addition to the actual stored data, roboticists also need an intuitive interface for storing, retrieving, and organizing this data.

![Foxglove Data Platform timeline](/images/blog/how-to-work-in-robotics-when-youre-not-a-roboticist/back-end.png)
_[Foxglove Data Platform](/data-platform) is a back-end service that provides an [API](/docs/data-platform/api/introduction), [web interface](https://console.foxglove.dev), and [CLI tool](https://github.com/foxglove/data-platform#cli-utility) for managing your data._

**With robust back-end support, robotics teams can successfully wrangle the data that so centrally drives their development.** As a team grows, so should the number of people working on back-end systems to coordinate between all these moving parts.

## UX and UI designer

With the sheer breadth of robotics products, the industry serves a wide array of customers – both technical and non-technical, internal and external, consumer and enterprise. Roboticists need developer tools, sales associates need metrics dashboards, and QA engineers need debugging utilities. Once robots are deployed, their operators need fleet management software, troubleshooting support, and specialized training.

**Since robotics use cases vary so wildly, the industry needs designers who are undaunted by the opportunity to solve problems that have never been solved before – for entirely novel workflows and niche customers.** Because the product criteria and design standards are so unique to this field, designers will have no shortage of interesting challenges to unlock. By understanding their users’ pain points and finding creative solutions for them, designers will play a pivotal role in defining the best practices for this budding industry.

![Foxglove Studio designs](/images/blog/how-to-work-in-robotics-when-youre-not-a-roboticist/ux-ui.png)
_We recently redesigned Foxglove Studio's panel selection flow to help users understand at a glance how each [panel](/docs/panels/introduction) can solve a specific problem for them._

## Marketing strategist

**Educating the public about your product comes with the territory of working in a bleeding edge industry like robotics.** Earning trust and building rapport with your potential customers is a huge part of this education process. As with any new technology, people can often be hesitant, confused, or even outright fearful of trying a solution for the first time – whether it’s as trivial as [ordering a cappuccino prepared by a robot](https://cafexapp.com/) or as serious as [introducing a robot companion to your child](https://embodied.com/). In addition to teaching them what your product does, you need to demonstrate in concrete terms how they will benefit from (and enjoy!) using it.

![Self-driving companies Cruise and Waymo](/images/blog/how-to-work-in-robotics-when-youre-not-a-roboticist/marketing.jpeg)

Many self-driving car companies, for example, plan years-long campaigns to build public trust in their technologies – by sharing demo videos, hosting community events, and inviting journalists to try beta versions of their technology. Their marketing can also come in more traditional forms, from advertisements that reveal under-the-hood details about the technology, to op-eds about the transportation industry. They expound on their stringent safety standards, on the benefits of going autonomous. All this marketing ensures that by the time they have a production-ready vehicle, they also have excited customers that trust their brand enough to pay for the product.

Marketing specialists with a technical background would be especially successful in this field, but anyone who is passionate about advocating for innovative products and their users will do well. **Having both the empathy to understand users’ challenges and the creativity to present possible solutions for them is paramount to succeeding in this role.**

## Sales engineer

Because the business of building robots is such an inherently cross-disciplinary endeavor, many previously siloed roles must redefine themselves in the context of robotics. Nowhere is this more apparent in sales. **Since robotics deployments are often so technically complex, robotics solutions providers need a sales team that is deeply familiar with how the underlying technology works.** Equipped with this expertise, they can then educate and ultimately persuade their customers – both upper management and technical operators – to adopt their robotics solution and reap the benefits.

**A sales engineer is not only responsible for educating potential customers on how a product will be useful, but also helping those customers reach their full potential with that product.** For example, sales engineers may walk a team through the benefits of adopting Autonomous Mobile Robots (AMRs) in warehouses – how it will mean less work for more throughput, a safer working environment, and a future-proof technology that will eventually pay for itself. But after that initial sale, they will also have to teach the AMR operators an entirely new set of skills – like how to manage multiple robots from one command station, how to assign specific robots certain tasks, and how to troubleshoot common issues.

![Seegrid's Palion Lift AMR](/images/blog/how-to-work-in-robotics-when-youre-not-a-roboticist/sales-engineer.jpeg)
_[Seegrid](https://seegrid.com/)’s Autonomous Mobile Robots (AMRs) do everything from transporting, spearing, and horizontally and vertically handling goods – effectively reducing downtime and increasing output._

**Since they partner so intimately with customers during onboarding, sales engineers are at a crucial juncture between the public and the rest of the robotics team.** They're privy to valuable feedback that the rest of the company is not – the problems users have, the solutions they wish they had. Robotics teams can then use this information to determine what to prioritize next on the roadmap, thus ensuring that the their focus is always closely aligned with their customers’ needs.

## Technical writer

Though the humanities are usually far from one's mind when thinking about robotics careers, there's a real need for good technical writing in this industry. **In addition to the educational writing that comes with marketing your product to potential customers, technical writers are responsible for describing how these products work for an internal and technical audience.** They catalog the relationships between various systems and services, document API endpoints and other interfaces, and essentially make it easy for members of their growing engineering team to understand, contribute to, and maintain these projects.

**Great technical writing empowers your entire team to get involved in aspects of the product that they may not have built themselves.** It also reminds us that our user base is not just the customer who will be operating our robots. They are also our teammates who need to get quick context on our ongoing work; they are the customers who will have to read our documentation when troubleshooting issues. **With thorough documentation, we bridge the knowledge gap between the people who built the product and the audience for whom it was built.**

## Product manager

In many sectors of the industry, roboticists are working on problems that have never been solved before. We’ve never had fully autonomous robots successfully [transport commuters](https://techcrunch.com/2022/01/14/2021-robotaxi-china/), [harvest our produce](https://grist.org/fix/future-farming-artificial-intelligence-robots-drones-precision-agriculture/), or [assemble homes](http://www.bbc.com/future/bespoke/the-disruptors/the-house-the-robots-built/) – at least at significant scale. **These are all work-in-progress ideals, and so we need imaginative leaders to help us navigate this uncharted territory and get at the essence of what this new generation of products needs.** Synthesizing information across different verticals to successfully deliver a solution to the public is an extremely difficult, and thus valuable, skillset to have.

![alt_text](/images/blog/how-to-work-in-robotics-when-youre-not-a-roboticist/product-manager.jpeg)
_Applying robotics technology to agriculture could make our means of production safer, more efficient, and [more resilient to climate change](https://grist.org/fix/future-farming-artificial-intelligence-robots-drones-precision-agriculture/)._

**If there's one cross-disciplinary role in robotics and beyond, it's that of the product manager – by necessity, they must be both technical and user-obsessed, interested in building new features and also listening to the market.** Because this role is such a mixture of so many different skills, it's especially possible to transfer your existing skills across fields.

Product managers work with sales engineers and developers, designers and marketers – all to serve the customers' interests. Some may even take on higher-level program manager responsibilities – like coordinating between teams to stay on schedule, adjusting the scope of deliverables to adapt to new information, and constantly assessing roadmaps to ensure they're doing the most important work for any given moment. **If you enjoy the challenge of incorporating learnings from different teams to deliver a solution to your users, a product manager role may be a fit for you.**

## Join our team

As we’ve seen, robotics is an extremely cross-disciplinary and creative field. There are a wide array of problems – that we are still uncovering! – that we still need to solve. **Finding the best talent out there in the world, whether that talent has yet been applied to robotics problems or not, is crucial to our collective success.** If you’ve had experience as a web developer, marketer, or writer, you likely have the transferable skills needed to contribute to the bleeding edge of technology.

If you’re interested in envisioning a world enhanced by robotics technology, the industry would love your experience and passion. Job boards, like the one run by [Silicon Valley Robotics](https://svrobo.org/robojobs/), are a great place to start. You can also peruse robotics publications like [The Robot Report](https://www.therobotreport.com/) to find companies working on problems that excite you, and then check out their career pages.

To learn more about the data and internal tools that these companies may use, check out our [sample self-driving data](/blog/visualizing-nuscenes-data-with-foxglove-studio) in the [Foxglove Studio app](/download). For ways to get involved with the Foxglove team, you can sign up for our [newsletter](https://www.getrevue.co/profile/foxglove), join our [Slack channel](/slack), or check out our own [careers page](/careers).



We've shipped many visualization features to Foxglove Studio in its first year of development.

To better showcase these features, we've added a sample dataset and layout for exploring the different stages of a self-driving car robotics pipeline.

![Foxglove Studio's Open Dialog](/images/blog/visualizing-nuscenes-data-with-foxglove-studio/dialog.png)
_Foxglove Studio is an open source data visualization platform for robotics and autonomy developers. [Download Studio here](/download), and click "Explore sample data" on app load to see the sample dataset and layout._

We hope this demo will inspire you to find new ways of leveraging Foxglove Studio for your own team's development workflows!

## A brief history of self-driving datasets

Since [our own team's history](/about) is steeped in self-driving technology, we knew we wanted our first demo to spotlight autonomous vehicles. But with the wide availability of open source self-driving datasets, we had to ask ourselves – which dataset should we choose?

The [KITTI Vision benchmark](http://www.cvlibs.net/datasets/kitti/), which was published in 2012 as a research paper, is considered the original publicly available dataset for autonomous vehicles. It consists of six hours of driving around a rural German town, and includes GPS coordinates, camera images, lidar returns, bounding boxes, and basic lane markers.

KITTI remained the default self-driving dataset until 2019, when Motional released [nuScenes](https://www.nuscenes.org/). This similarly sized dataset consisted of 1000 scenes from Boston and Singapore, and included more cameras, radar sensors, and object classification labels (32, up from KITTI's seven). It was even accompanied by a [Python library](https://github.com/nutonomy/nuscenes-devkit) to facilitate accessing the data. This prompted [Lyft](https://www.lyft.com/) to release [their own dataset](https://level-5.global/data/) that same year, using a fork of the nuScenes Python library and a similar data organization. With this revived interest in public datasets, other automakers and AV companies – like [Argo](https://www.argoverse.org/), [Audi](https://www.a2d2.audi/a2d2/en.html), [Berkeley](https://bdd-data.berkeley.edu/), [Ford](https://avdata.ford.com/), and [Waymo](https://github.com/waymo-research/waymo-open-dataset) – followed suit.

Meanwhile, the nuScenes dataset has continued to evolve with expansion packs that added raw CAN bus data, rich semantic maps, class- and instance-level lidar segmentation annotations, planning trajectories, per-pixel image masks, and more. Taken together, this data starts to resemble the data pipelines powering today's production autonomous vehicles, making it a great fit for demonstrating the power and flexibility of Foxglove Studio.

_nuScenes data is copyright © 2020 nuScenes and available under a [non-commercial license](https://www.nuscenes.org/terms-of-use)._

## A closer look at nuScenes

> _This section provides a technical breakdown of the nuScenes dataset. Feel free to skip to the next section if you're not interested in these details._

**To visualize the nuScenes dataset, we first have to understand the data it contains and the format it's stored in.**

Downloading the [Full dataset (v1.0)](https://www.nuscenes.org/nuscenes#download) contains the majority of the data: sensor captures (camera, radar, lidar), localization, and maps for the relevant regions. We then integrate the [CAN bus expansion](https://www.nuscenes.org/tutorials/can_bus_tutorial.html) for high-frequency diagnostics, [Map expansion](https://www.nuscenes.org/tutorials/map_expansion_tutorial.html) for semantic map layers, and [nuPlan maps](https://www.nuscenes.org/nuplan) for high-definition 2D lidar raster maps.

Once these datasets have been downloaded and unpacked into the same folder, you have all the source data used to generate our final dataset. We've made [our own script](https://github.com/foxglove/nuscenes2bag) for downloading and unpacking this data available on GitHub, so the following information is purely for reference.

### Sensors

Camera images, lidar returns, and radar returns are all stored under the `samples` folder.

**Camera images** are organized into subfolders for each sensor, with individual frames stored as 1600x900 JPEG compressed images. The dataset contains "keyframes" at 2Hz (every 500ms), so that each sensor has a frame with corresponding bounding box annotations for a given timestamp. Camera images are also available outside of the keyframe captures, at roughly 12Hz.

**Lidar returns** are stored in the `LIDAR_TOP` subfolder. Individual point cloud captures are stored as `.pcd.bin` files, which use a simple packing of six 32-bit floating-point values (little-endian, four bytes per float) storing `x`, `y`, `z`, `intensity`, and `ring` values per point. The x/y/z coordinates are in the `LIDAR_TOP` coordinate frame.

**Radar returns** are also organized into subfolders for each sensor. Individual point clouds are stored in PCD v0.7 format, which uses an ASCII header followed by binary data that stores 18 different fields using different primitive types and byte widths.

### Localization

Localization data lives in a single `ego_pose.json` file.

Each entry in the JSON array contains a `timestamp` in microseconds, a `rotation` quaternion, and a `translation` vector3. Although the ego pose (i.e. the car's position) is available at a higher frequency than 2Hz, each keyframe links the sensor samples to the nearest ego pose in time.

### Vehicles

The CAN bus expansion data lives as JSON files in a `can_bus` subfolder.

These files contain raw CAN vehicle bus data – like position, velocity, steering, lights, battery, and more – per scene. For more information, check out the [CAN bus expansion README](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/can_bus/README.md) or [tutorial](https://www.nuscenes.org/tutorials/can_bus_tutorial.html).

### Map

The nuScenes map expansion and nuPlan maps include the following:

- High-definition lidar maps
- Raster PNG files for driveable area bitmaps and lidar intensity maps
- JSON vector data for lane centerlines, lane boundaries, sidewalks, crosswalks, stop lights, and more

More information can be found in the nuScenes [maps expansion tutorial](https://www.nuscenes.org/tutorials/map_expansion_tutorial.html).

## Visualizing nuScenes with Foxglove Studio

**To visualize this data in Foxglove Studio, we need to convert it to a format that Studio understands.**

While Foxglove and the ROS Middleware Working Group are designing a generic recording format for robotics data ([MCAP](https://github.com/foxglove/mcap)), that file specification is still a work in progress. In the meantime, the [ROS bag format](http://wiki.ros.org/Bags/Format/2.0) provides a convenient way of packaging heterogeneous data streams into a single file with type definitions, indexing, and compression.

**We created a [Python Jupyter Notebook](https://github.com/foxglove/nuscenes2bag) to load the nuScenes data and convert each scene into a ROS bag file.** Each category of data had to be converted into a ROS message type that Studio understands:

- Camera images – [sensor_msgs/CompressedImage](https://docs.ros.org/en/api/sensor_msgs/html/msg/CompressedImage.html), [sensor_msgs/CameraInfo](https://docs.ros.org/en/api/sensor_msgs/html/msg/CameraInfo.html)
- Lidar and radar returns – [sensor_msgs/PointCloud2](https://docs.ros.org/en/api/sensor_msgs/html/msg/PointCloud2.html)
- CAN bus data – [diagnostic_msgs/DiagnosticArray](https://docs.ros.org/en/api/diagnostic_msgs/html/msg/DiagnosticArray.html)
- Raster maps – [nav_msgs/OccupancyGrid](https://docs.ros.org/en/api/nav_msgs/html/msg/OccupancyGrid.html)
- Semantic maps and bounding box annotations – [visualization_msgs/Marker](https://docs.ros.org/en/api/visualization_msgs/html/msg/Marker.html)
- Car's position (or the ego pose) – [geometry_msgs/PoseStamped](https://docs.ros.org/en/api/geometry_msgs/html/msg/PoseStamped.html), [sensor_msgs/NavSatFix](https://docs.ros.org/en/api/sensor_msgs/html/msg/NavSatFix.html), [tf2_msgs/TFMessage](http://docs.ros.org/en/api/tf2_msgs/html/msg/TFMessage.html)
- Additional data superimposed on camera images – [foxglove_msgs/ImageMarkerArray](https://docs.ros.org/en/api/foxglove_msgs/html/msg/ImageMarkerArray.html)

**Next, we created a Foxglove Studio [layout](/docs/layouts) that visualizes all this data.** Since there is a lot to see in this dataset, starting with a top-level [Tab panel](/docs/panels/tab) helped organize the data around four main use cases: **Perception**, **Planning**, **Controls**, and **Diagnostics**.

### Perception

<video autoplay loop muted playsinline src="/images/blog/visualizing-nuscenes-data-with-foxglove-studio/001.mp4" poster="001.jpg"></video>

**Our Perception tab focuses on sensor data**, with the majority of the screen dedicated to displaying `sensor_msgs/CompressedImage` topics across six [Image panels](/docs/panels/image). The top row shows the front camera feeds; the bottom row the rear cameras. Since the Motional car has 360° camera coverage, we manually aligned the camera feeds in this layout to create two 180° views.

Each camera image also has its own pair of `foxglove_msgs/ImageMarkerArray` topics for the lidar points and bounding box annotations, both of which have been projected into the individual 2D camera frames.

The bottom-most panel in this tab is a [3D panel](/docs/panels/3d), which shows the world from the perspective of the front camera – with bounding box annotations, lidar returns, the lidar raster map, and more.

### Planning

<video autoplay loop muted playsinline src="/images/blog/visualizing-nuscenes-data-with-foxglove-studio/002.mp4" poster="002.jpg"></video>

**The Planning tab focuses on getting a comprehensive understanding of the whole scene**, by combining a map of the area with a richly annotated 3D scene. It is split into a [Map panel](/docs/panels/map) showing the GPS track of the vehicle, and a [3D panel](/docs/panels/3d) in orthographic mode showing a bird's-eye view. The 3D scene displays lane centerlines from the semantic map, our car's pose, bounding box annotations for nearby detections, and lidar and radar point clouds.

The smooth movement of the vehicle is achieved using [transform interpolation](https://github.com/foxglove/studio/pull/2243). This recently added feature allows transforms to be smoothly updated at full framerate, as long as there is a transform before and after the current time. Even though our transform messages are only published every 500ms (2Hz), we can achieve smooth movement throughout playback by publishing the transform for time `t+1` at time `t`. Publishing future data is a luxury of post-processing an existing recording, but this can be used on live robots as well by publishing a prediction of the transform at `t+1` (and overwriting it with observed data once time advances to `t+1`). For a deeper technical understanding of transform interpolation in robotics, the whitepaper [_tf: The Transform Library_](https://gaoyichao.com/Xiaotu//ros/refs/tf_The_Transform_Library.pdf) is a good starting point.

While the vehicle moves smoothly, everything else in the scene is moving at the publish rate of the different data streams. Lidar and radar scans come in at about 12Hz while bounding boxes are only updated at 2Hz, which appears as visibly snapping from position to position.

This tab uses the following datatypes to display the various components:

- `sensor_msgs/NavSatFix` – GPS
- `tf2_msgs/TFMessage` – Coordinate frames, relative positions of the car and its sensors
- `visualization_msgs/MarkerArray` – Bounding box annotations, lane centers
- `geometry_msgs/PoseStamped` – Car pose
- `sensor_msgs/PointCloud2` – Radar and lidar returns
- `nav_msgs/OccupancyGrid` – Driveable area around car, grayscale map

### Controls

<video autoplay loop muted playsinline src="/images/blog/visualizing-nuscenes-data-with-foxglove-studio/003.mp4" poster="003.jpg"></video>

**The Controls tab compares the throttle input against the vehicle's acceleration**, as measured using the Inertial Measurement Unit (IMU) and speed of all four wheels.

Each data series in the two [Plot panels](/docs/panels/plot) is specified using a [message path](/docs/app-concepts/message-path-syntax), Studio's own syntax for referencing message data.

### Diagnostics

<video autoplay loop muted playsinline src="/images/blog/visualizing-nuscenes-data-with-foxglove-studio/004.mp4" poster="004.jpg"></video>

**The final Diagnostics tab gives you a peek under the hood at the raw data and vehicle diagnostics.**

The [State Transitions panel](/docs/panels/state-transitions) at the top also uses message path syntax to show the `brake_switch` value change over time – between 1, 2, and 4. According to the nuScenes CAN documentation, possible states for this `brake_switch` value are `1 (pedal not pressed), 2 (pedal pressed) or 3 (pedal confirmed pressed)`. The unexpected 4 we see is worth investigating, if we're interested in using this signal.

The [Diagnostic Summary and Diagnostic Details panels](/docs/panels/diagnostics) below show the CAN bus messages formatted as a `diagnostic_msgs/DiagnosticArray`.

The [Data Source Info panel](/docs/panels/data-source-info) in the top right lists all topics, datatypes, and publish frequencies for our bag file.

The last view is a [Raw Messages panel](/docs/panels/raw-messages), used here in a rather unique way. By setting the message path to filter for a marker with a specific ID, we can monitor a single marker over time. For another creative use of this panel, try changing the message path to point to a primitive value (e.g. `/markers/annotations.markers[:]{id==28114}.pose.position.x`) for a single value monitor.

## Further exploration

This sample dataset and layout was built to showcase the breadth of Foxglove Studio's newest features and to inspire you to visualize your own robotics datasets. This nuScenes example will be the first of many upcoming demos in Studio, all tailored to different verticals in robotics.

In the spirit of our open source philosophy, the code we used to convert the nuScenes data into ROS bag format is [available on GitHub](https://github.com/foxglove/nuscenes2bag). While this was our approach, there are many ways this conversion could be extended – visualizing nuPlan trajectories, upgrading the car marker to a 3D mesh marker, or segmenting lidar points to different topics for each object classification. There is also additional nuScenes data that we could have pulled in to provide even richer visualizations.

If you have questions or ideas on how to visualize your data with Foxglove Studio, join us in our [Slack community](/slack)'s #lounge channel.



Since [v0.21](https://github.com/foxglove/studio/releases/tag/v0.21.0), Foxglove Studio has supported loading [mesh resource markers](http://wiki.ros.org/rviz/DisplayTypes/Marker#Mesh_Resource_.28MESH_RESOURCE.3D10.29_.5B1.1.2B-.5D) in the 3D panel. This means that you can now visualize custom models in the context of your 3D scene, alongside the other players and perceived objects present in your robot's world.

So how can you start loading 3D models into your scene?

## Write a Python publisher

First, you’ll need to publish a marker topic, following the message definition outlined in ROS 1’s [visualization_msgs/Marker](http://docs.ros.org/en/noetic/api/visualization_msgs/html/msg/Marker.html) or ROS 2’s [visualization_msgs/msg/Marker](https://github.com/ros2/common_interfaces/blob/master/visualization_msgs/msg/Marker.msg). In both schemas, you'll notice a `mesh_resource` field, where you can specify the file containing your 3D model. Studio currently supports reading an HTTP, HTTPS, or ROS package URL from this field to load glTF (`.glb`), COLLADA (`.dae`), and STL (`.stl`) files.

On a computer with a working [ROS](https://ros.org/) setup, create a Python file (e.g. `mesh_marker.py`) containing a simple publisher that outputs marker messages with a currently empty `mesh_resource` field:

```python
#! /usr/bin/env python

import rospy
from visualization_msgs.msg import Marker

rospy.init_node('mesh_marker')

marker_pub = rospy.Publisher("/avocado", Marker, queue_size = 2)

marker = Marker()

marker.header.frame_id = "base_link"
marker.header.stamp = rospy.Time.now()
marker.ns = ""

# Shape (mesh resource type - 10)
marker.type = 10
marker.id = 0
marker.action = 0

# Note: Must set mesh_resource to a valid URL for a model to appear
marker.mesh_resource = ""
marker.mesh_use_embedded_materials = true

# Scale
marker.scale.x = 10.0
marker.scale.y = 10.0
marker.scale.z = 10.0

# Color
marker.color.r = 0.0
marker.color.g = 0.0
marker.color.b = 0.0
marker.color.a = 1.0

# Pose
marker.pose.position.x = 3
marker.pose.position.y = 0
marker.pose.position.z = 0
marker.pose.orientation.x = 0.0
marker.pose.orientation.y = 0.0
marker.pose.orientation.z = 0.0
marker.pose.orientation.w = 1.0

while not rospy.is_shutdown():
  marker_pub.publish(marker)
  rospy.rostime.wallsleep(1.0)
```

Now that we've written our publisher, let's try loading a few different types of mesh resources in Foxglove Studio's 3D panel!

## Load a glTF (`.glb`) file with an HTTPS URL

Let's load a glTF file in our published marker's `mesh_resource` field. In your `mesh_marker.py` file, replace the empty field value with the following URL:

```python
marker.mesh_resource = "https://storage.googleapis.com/foxglove-public-assets/Avocado.glb"
```

Run the following command in your terminal to **start publishing marker messages on the `/avocado` topic**:

```bash
python3 mesh_marker.py
```

To verify that your newly published messages include a link to the `.glb` file, **inspect the `/avocado` topic in a Raw Messages panel in Foxglove Studio**. If everything is working as expected, you’ll see that the `mesh_resource` field points to [an HTTPS URL for a glTF asset](https://storage.googleapis.com/foxglove-public-assets/Avocado.glb).

![Topic in Foxglove Studio's Raw Messages panel](/images/blog/how-to-visualize-ros-mesh-markers/raw-messages.png)

Next, **add a 3D panel to your layout to see the mesh marker in 3D space.** Open the topic picker to toggle on your mesh marker topic (`/avocado`). You’ll see a 3D avocado appear floating in the scene.

![Avocado mesh marker in Foxglove Studio's 3D panel](/images/blog/how-to-visualize-ros-mesh-markers/avocado.png)

To see this working immediately, you can also drag and drop this [meshmarkers.bag](https://drive.google.com/file/d/16RR4ijj7_mMr8l3-wvjMmQtXnk8FF3y2/view?usp=sharing) into Foxglove Studio to see the same results.

## Load a COLLADA (`.dae`) file with an HTTPS URL

Now, let’s try loading a different mesh resource. Replace the `mesh_resource` field value to point to an HTTPS URL for a COLLADA (`.dae`) asset:

```python
marker.mesh_resource = "https://raw.githubusercontent.com/gundam-global-challenge/gundam_robot/master/gundam_rx78_description/meshes/rx78_object_005-lib.dae"
```

In your 3D panel, you should now see a [Gundam](https://en.wikipedia.org/wiki/Gundam) robot part where your avocado used to be:

![Gundam robot part in Foxglove Studio's 3D panel](/images/blog/how-to-visualize-ros-mesh-markers/gundam.png)

## Load an STL (`.stl`) file with a ROS package URL

Finally, let’s try a link to a file included in a ROS package.

First, install the [`turtlebot3_description`](http://wiki.ros.org/turtlebot3_description) ROS package on your machine. If you are using ROS Noetic, run the following command in your terminal:

```bash
$ sudo apt install ros-noetic-turtlebot3-description
```

Source your ROS setup file, then grab the correct package path:

```bash
$ source /opt/ros/noetic/setup.zsh

$ echo $ROS_PACKAGE_PATH
/opt/ros/noetic/share
```

Next, open the Preferences sidebar in Foxglove Studio to set your [`ROS_PACKAGE_PATH`](http://wiki.ros.org/ROS/EnvironmentVariables#ROS_PACKAGE_PATH) environment variable to your package path (e.g. `/opt/ros/noetic/share`). This makes it so that all ROS `package://` URLs referenced in the app will now use your `ROS_PACKAGE_PATH` to find the relevant package assets (e.g. `/opt/ros/noetic/share/turtlebot3_description`). _Note that this will work only on the desktop app, as we are referencing files in our local file system._

Finally, replace the same line of code in your `mesh_marker.py` file with the following:

```python
marker.mesh_resource = "package://turtlebot3_description/meshes/bases/burger_base.stl"
```

If your `ROS_PACKAGE_PATH` was set to `/opt/ros/noetic/share`, the mesh_resource above will look in `/opt/ros/noetic/share/turtlebot3_description/meshes/bases/burger_base.stl` for the file in question.

The [STL file in question](https://github.com/ROBOTIS-GIT/turtlebot3/blob/master/turtlebot3_description/meshes/bases/burger_base.stl) contains the following model:

![Turtlebot base](/images/blog/how-to-visualize-ros-mesh-markers/turtlebot-base.png)

You should now see it displayed in your 3D panel.

## Show us what you’ve got!

We hope this new feature helps you build ever-richer representations of the world your robots navigate, to better analyze their behavior and improve their performance. If you would like more file formats (beyond `.glb`, `.dae`, `.stl`) to be supported in the future, please let us know! You can open a [GitHub issue](https://github.com/foxglove/studio/issues) or message us directly in our [Slack community](https://foxglove.dev/slack).

This was a long-anticipated feature, so we’re excited to see what your Foxglove Studio dashboards look like now! Share images of your models in our Slack community’s #lounge channel or on Twitter (don't forget to tag us [@foxglovedev](https://twitter.com/foxglovedev)). We’re so excited to see the workflows that this feature unlocks for your team.


Accelerate your
robotics development

Use Foxglove Studio's rich interactive visualizations to analyze live connections and pre-recorded data.

Use the Foxglove Data Platform to manage and organize your data.

Lean on robust community support to get the most out of our extensible tools.

Follow our blog

Want product news and updates?

Ready to get started?Download today on Linux, Windows, or macOS.

Accelerate yourrobotics development