Import and Export Robotics Data Using the Foxglove CLI

Simplify robotics development by managing data from the command line

Esther Weon ·

6 min read

Published December 19, 2023

Import and Export Robotics Data Using the Foxglove CLI

In robotics development, importing data from your robots and sharing it across your organization efficiently is critical. Foxglove offers a variety of tools to streamline this process – the web UI allows anyone with a browser to manage team data from a central repository; the Python library allows developers to write scripts that retrieve and manipulate data from this same repository. While these are popular options, the Foxglove CLI provides yet another lightweight, yet powerful, alternative.

In this tutorial, we will explore how to leverage the Foxglove CLI to import and export data, so that your team can spend less time wrangling data and more time building robots.

Understanding the use cases

Unlike the web UI, the Foxglove CLI provides a reliably consistent interface that developers can rely on to interact with data. And unlike the Python library, the CLI provides developers with the direct feedback of terminal commands – no need to create a file, install the relevant packages, and run the script.

The CLI becomes particularly useful in scenarios where automation and scripting play a crucial role. For instance, if you are dealing with setting up large-scale data processing tasks or integrating Foxglove into a broader automation pipeline, the CLI can be a powerful option.

Simplifying data management with the CLI

To get started, create a free Foxglove account – this will allow you to import data to the Foxglove cloud.

Next, install the latest release of the Foxglove CLI for your platform, and use it to sign in to your Foxglove account from the command line:

$ foxglove auth login

Alternatively, you can configure the CLI to use an API key.

Once you’re logged in, you’re ready to start importing and exporting data!

Import data

Foxglove supports importing ROS 1 bag files and MCAP files.

If you’re already using ROS 2 .db3 files, you can convert them to the MCAP format with the MCAP CLI. Starting from the ROS 2 Iron Irwini release, rosbag2 now records data directly to MCAP by default, for easy analysis with third-party tools like Foxglove.

Once you have data files to import, use the CLI to create the devices (i.e. robots) on which they were recorded:

$ foxglove devices add --name RobotA
  Device created: dev_flm75pLkfzUBX2DH

$ foxglove devices add --name RobotB
  Device created: dev_Wm1gvryKJmREqnVT

If you create more than one device, you can retrieve a list of all their names and IDs for easy reference:

$ foxglove devices list
  | ID                    | NAME    | CREATED AT            | UPDATED AT            |
  | --------------------- | —------ | --------------------- | --------------------- |
  | dev_flm75pLkfzUBX2DH  | RobotA  | 2023-12-10T17:20:55Z  | 2023-12-10T17:20:55Z  |
  | dev_Wm1gvryKJmREqnVT  | RobotB  | 2023-12-10T17:22:55Z  | 2023-12-10T17:22:55Z  |

Next, it's time to import a local file associated with one of these newly created devices! If you don't have a file to import, download this example MCAP file to continue with the tutorial:

$ curl -X GET https://assets.foxglove.dev/NuScenes-v1.0-mini-scene-0061-df24c12.mcap -o recording.mcap

Next, associate your local file to the device that recorded it:

# For MCAP file recorded by RobotA
$ foxglove data import /path/to/recording.mcap --device-name RobotA

# For ROS 1 file recorded by RobotB
$ foxglove data import /path/to/recording.bag --device-name RobotB

To verify that you’ve successfully imported these data files to the Foxglove cloud, retrieve a list of all imported data:

$ foxglove recordings list
-[ RECORD 1 ]--+----------------------------
Recording ID   | 5m88YzSVGPVBDax2
Path           | recording.bag
Size           | 67.1 MiB
Message Count  | 1606
Created At     | 2023-12-10T19:06:55.425Z
Imported At    | 2023-12-10T19:06:55.425Z
Start          | 2018-07-25T03:28:47.604844Z
End            | 2018-07-25T03:29:06.797517Z
Import Status  | complete
Site ID        | aCQa2Ks3AyNvQo4H
Site Name      | Foxglove Cloud
Edge Site ID   |
Edge Site Name |
Device ID      | dev_Wm1gvryKJmREqnVT
Device Name    | RobotB
Metadata       |
-[ RECORD 2 ]--+----------------------------
Recording ID   | 5m88YzSVGPVBDax2
Path           | recording.mcap
Size           | 488.5 MiB
Message Count  | 33886
Created At     | 2023-12-10T18:06:55.425Z
Imported At    | 2023-12-10T18:06:55.425Z
Start          | 2018-06-24T03:28:47.604844Z
End            | 2018-06-24T03:29:06.797517Z
Import Status  | complete
Site ID        | aCQa2Ks3AyNvQo4H
Site Name      | Foxglove Cloud
Edge Site ID   |
Edge Site Name |
Device ID      | dev_flm75pLkfzUBX2DH
Device Name    | RobotA
Metadata       |

If you don’t see your recordings, they may still be listed as pending imports:

$ foxglove pending-imports list
  -[ RECORD 1 ]-+-------------------------------------
  Created At    | 2023-12-10T18:06:55.425Z
  Imported At   | 2023-12-10T18:06:55.425Z
  Org ID        | 1329pe9f-cf63-92ec-7e19-93db76a7103b
  Filename      | recording.mcap
  Pipeline stage| ingestion
  Request ID    | 9831ab13-31e9-450f-bfb0-2250d7c6d2f6
  Device ID     | dev_flm75pLkfzUBX2DH
  Device name   |
  Import ID     |
  Site ID       |
  Status        | ready for indexing
  Error         |
  ...

Export data

Now that you have imported data, you can export meaningful segments of it for downstream sharing and analysis.

Query your imported data by device ID, recording time range, and list of topics. For example, if you wanted to calibrate the sensors of a robot that crashed into a wall, pull just its camera and LIDAR messages around the time of the incident. If you don’t provide a list of topics, the CLI will return all topics for the device within the time range.

Once you’ve identified the data you want to export, you can output it as an MCAP file (default), as a ROS 1 bag file (--output-format bag1), or directly to the console (--output-format json):

# MCAP files
$ foxglove data export --device-id dev_flm75pLkfzUBX2DH \
  --start 2021-01-01T00:00:00Z --end 2022-01-01T00:00:00Z \
  --topics /gps/fix,/gps/fix_velocity > output.mcap

# ROS 1 bag files
$ foxglove data export --device-id dev_flm75pLkfzUBX2DH \
  --start 2021-01-01T00:00:00Z --end 2022-01-01T00:00:00Z \
  --output-format bag1 \
  --topics /gps/fix,/gps/fix_velocity > output.bag

Next, let's verify that you’ve exported the correct data.

For MCAP files, download the MCAP CLI, and run mcap info on your output file:

$ mcap info output.mcap
  library: mcap go #(devel); fg-data-platform-db07abe7
  profile: ros1
  messages: 6728
  duration: 5m39.304931438s
  start: 2021-03-22T08:03:38.473036858-07:00 (1616425418.473036858)
  end: 2021-03-22T08:09:17.777968296-07:00 (1616425757.777968296)
  compression:
          lz4: [1/1 chunks] (86.05%)
  channels:
          (1) /gps/fix           3364 msgs (9.91 Hz)   : sensor_msgs/NavSatFix [ros1msg]
          (2) /gps/fix_velocity  3364 msgs (9.91 Hz)   : geometry_msgs/TwistWithCovarianceStamped [ros1msg]
  attachments: 0

For ROS 1 bag files, download rosbag to run similar commands:

$ rosbag reindex output.bag
$ rosbag info output.bag
  path:         output.bag
  version:      2.0
  duration:     5:39s (339s)
  start:        Mar 22 2021 08:03:38.47 (1616425418.47)
  end:          Mar 22 2021 08:09:17.78 (1616425757.78)
  size:         328.6 KB
  messages:     6728
  compression:  lz4 [1/1 chunks; 12.87%]
  uncompressed:   1.8 MB @ 5.5 KB/s
  compressed:   240.0 KB @ 0.7 KB/s (12.87%)
  types:        geometry_msgs/TwistWithCovarianceStamped [b00b6ce36bf21f646151de97da2c485c]
                sensor_msgs/NavSatFix                    [7f6e605ad1e52d05162190ff17be80b6]
  topics:       /gps/fix            3364 msgs    : sensor_msgs/NavSatFix
                /gps/fix_velocity   3364 msgs    : geometry_msgs/TwistWithCovarianceStamped

You can also drag and drop output ROS 1 bag and MCAP files into Foxglove to visualize their playback – we recommend adding a Raw Messages panel to view the raw output of one of the topics you exported.

Add events

Knowing how to import and export data is just the beginning. To navigate your growing collection of data recordings, you can annotate them with events. Events denote points of interest, and can occur instantaneously – like for a robot collision – or over a range of time – like the time a robot spends over a certain speed.

To create an event, you must provide it with a device-id and time (start and end – these values should be identical for instantaneous events) – colon-separated key-value pairs for metadata is optional:

# Instantaneous event
$ foxglove events add --device-id dev_flm75pLkfzUBX2DH \
  --start 2021-01-02T00:00:00Z --end 2021-01-02T00:00:00Z \
  --metadata collision:true
  Created event: evt_N6doUtPYh8i7iZxf

# Time range event
$ foxglove events add --device-id dev_Wm1gvryKJmREqnVT \
  --start 2021-01-02T00:00:00Z --end 2021-01-02T00:05:00Z \
  --metadata speeding:true --metadata mph:60
  Created event: evt_idMGJImlICYP4dcy

Retrieve a list of all events to ensure your event was created successfully:

$ foxglove events list

| ID                   | DEVICE ID            | START                | END                  | CREATED AT               | UPDATED AT               | METADATA                       |
| -------------------- | -------------------- | -------------------- | -------------------- | ------------------------ | ------------------------ | ------------------------------ |
| evt_N6doUtPYh8i7iZxf | dev_flm75pLkfzUBX2DH | 2021-01-02T00:00:00Z | 2021-01-02T00:00:00Z | 2023-12-13T13:22:44.263Z | 2023-12-13T13:22:44.263Z | {"collision":"true"}           |
| evt_idMGJImlICYP4dcy | dev_Wm1gvryKJmREqnVT | 2021-01-02T00:00:00Z | 2021-01-02T00:05:00Z | 2023-12-13T13:26:37.080Z | 2023-12-13T13:26:37.080Z | {"speeding":"true","mph":"60"} |

Conclusion

The Foxglove CLI unlocks new ways to interact with your robots’ data and simplifies your team’s development workflows. Its consistent interface for creating, exporting, and retrieving resources makes it a valuable addition to every roboticist’s toolbox – especially when they want to write scripts to automate their workflows.

Check out the docs to learn how you can leverage the CLI for other development workflows – like viewing pending imports, publishing Foxglove extensions, and more.