The Foxglove Isaac Sim extension enables real-time visualization of robotics simulation data directly in Foxglove.
The Foxglove Isaac Sim extension enables real-time visualization of robotics simulation data directly in Foxglove. In this post, we'll take a deeper look into the extension's code to understand how it works and explore ways it can be expanded upon.
In case you missed our previous post on the extension, we released an NVIDIA Isaac Sim extension that enables seamless visualization of simulation data in Foxglove. The extension automatically detects all cameras, IMUs, and articulations in the simulation stage, making the data—along with the complete Transform Tree—available in Foxglove.
You can install the extension directly from Isaac Sim’s extension manager, and the source code is available on GitHub.
foxglove.tools.ws_bridge/
├─ config/
│ └─ extension.toml
├─ data/
│ ├─ icon.png
│ └─ preview.png
├─ docs/
│ ├─ CHANGELOG.md
│ └─ README.md
└─ foxglove/tools/ws_bridge/
├─ json_schemas/
├─ README.md
├─ __init__.py
├─ schemas.py
├─ **data_collection.py**
├─ **extension.py**
├─ **foxglove_wrapper.py**
└─ **ui_builder.py**
The code follows the same structure as other Isaac Sim extensions and was built on top of the UI extension template generated by Isaac Sim (available under Isaac Utils > Generate Extension Templates in the toolbar).
The extension operates primarily through four main files:
ui_builder.py.IsaacSensor and DataCollector classes, which track active sensors in the simulation and interface with the Foxglove wrapper.Since extension.py and ui_builder.py were mostly adapted from the template code, this post will focus on the custom functionality in data_collection.py and foxglove_wrapper.py.
💡 Useful resources
The purpose of this extension is to automatically detect the sensors in an Isaac Sim project, retrieve their data, and send it to Foxglove for visualization. To better understand the code, let’s start by looking at how sensors are defined.
In Isaac Sim, sensors are characterized by the following:
The Foxglove extension adds two more attributes:
The FoxgloveWrapper class (defined in foxglove_wrapper.py) manages the Foxglove Server: it starts and stops the server on a configurable port, adds or removes channels based on the active sensors in the simulation, and publishes messages on those channels.
class FoxgloveWrapper():
def __init__(self, data_collector: DataCollector): ...
def start(self, port: int, sensors : dict): ...
async def _run_server(self, port : int, sensors : dict): ...
async def init_channels(self, sensors : dict): ...
def add_channel(self, sensor): ...
async def _add_channel(self, sensor): ...
def remove_channel(self, sensor_path : str): ...
async def _remove_channel(self, sensor_path : str): ...
def send_message(self, data : dict): ...
async def _send_message(self, data : dict): ...
def close(self): ...All server actions run on a dedicated thread, allowing them to execute in parallel with the simulation without blocking it. In the code, functions prefixed with _ are asynchronous (async) versions of the main functions, enabling non-blocking calls from other parts of the code.
def start(self, port: int, sensors : dict):
loop = asyncio.get_event_loop()
self.server_task = loop.create_task(self._run_server(port, sensors))
async def _run_server(self, port : int, sensors : dict):
async with FoxgloveServer("0.0.0.0", port, "isaac sim server") as self.server:
...
await self.init_channels(sensors)
print(f"[Foxglove Info] Foxglove server started at ws://0.0.0.0:{port}")
...
async def init_channels(self, sensors : dict):
for sensor in sensors.values():
await self._add_channel(sensor)
def close(self):
if self.server:
self.server_task.cancel()
self.server = None
print(f"[Foxglove Info] Foxglove server closed")start() calls the async function _start(), which creates a new asynchronous task that runs the server on its own thread.close() cancels that task, thus shutting down the server.def add_channel(self, sensor):
loop = asyncio.get_event_loop()
if self.server:
loop.create_task(self._add_channel(sensor))
async def _add_channel(self, sensor):
schema_name, schema, encoding, schema_encoding = ...
topic_name = ...
self.path2channel[sensor.path] = await self.server.add_channel(
{
"topic": topic_name,
"encoding": encoding,
"schemaName": schema_name,
"schema": schema,
"schemaEncoding": schema_encoding,
}
)
self.channel2path[self.path2channel[sensor.path]] = sensor.path
def remove_channel(self, sensor_path : str):
loop = asyncio.get_event_loop()
if self.server:
loop.create_task(self._remove_channel(sensor_path))
async def _remove_channel(self, sensor_path : str):
await self.server.remove_channel(self.path2channel[sensor_path])
chan_id = self.path2channel.pop(sensor_path)
self.channel2path.pop(chan_id)To create a new channel on the Foxglove Server (see _add_channel()), five arguments are required:
topic: The arbitrary name of the channel under which messages will appear in Foxglove.schemaName: The name of the message type, which Foxglove uses to determine how to display the message. Supported message types in Foxglove can be found here, but the schema name can also be customized if no existing type fits your requirements.schema: The detailed format information for the message type. encoding & schemaEncoding: The type of encoding used for the message (e.g. JSON, Protobuf, etc.).In the extension, the mapping from sensor type to schema is defined in schemas.py.
When a new channel is created, it is assigned a unique ID used when publishing a message or removing the channel (see _remove_channel()). In the extension, channels are identified by their corresponding sensor’s path, so we maintain a mapping from sensor path to channel ID (and vice versa).
def send_message(self, data : dict):
loop = asyncio.get_event_loop()
if self.server:
loop.create_task(self._send_message(data))
async def _send_message(self, data : dict):
for path, payload in data.items():
if self.server and payload:
await self.server.send_message(
self.path2channel[path],
time.time_ns(),
payload,
)To send a message to the Foxglove Server (see _send_message()), three arguments are required:
The extension tracks the active sensors of the simulation using the IsaacSensor and DataCollector classes, defined in data_collection.py.
class IsaacSensor():
def __init__(self, sensor_type : str, sensor_path : str, ...):
self.type = sensor_type
self.path = sensor_path
...
def enable(self): ...
def disable(self): ...
def update_cam_resolution(self, width : int, height : int): ...
def collect(self): ...
def cam_collect(self): ...
def imu_collect(self): ...
def articulation_collect(self): ...
def tf_tree_collect(self): ...The extension introduces a custom IsaacSensor class to interface with actual Isaac Sim sensors. This class has three primary attributes: type and path (as described earlier in this post) and _sensor, which represents the corresponding Isaac Sim sensor object. It also maintains a boolean attribute, enabled, which is set to True when a Foxglove Client subscribes to the sensor’s channel on the Foxglove Server, preventing unnecessary data collection for sensors that aren’t being displayed. Lastly, the class includes a collect() method for querying data from the real sensor.
The _sensor attribute is initialized based on the sensor type, using various methods of the Omniverse API (see __init__()). Similarly, the collect() method uses type-specific functions to handle data collection for each sensor type. Each function returns the pre-formatted payload that aligns with the schema defined for this data type—the same schema used when creating the corresponding server channel.
In the current version of the extension, Camera Images and Transform Trees are sent as protobuf messages (respectively foxglove.CompressedImage and foxglove.FrameTransforms), while IMU data and Joint States are sent as custom JSON messages.
class DataCollector():
def __init__(self):
...
self.fox_wrap = FoxgloveWrapper(self)
def init_sensors(self): ...
def update_sensors(self): ...
def add_sensor(self, prim : Prim, ...): ...
def remove_sensor(self, sensor_path : str): ...
def set_cam_resolution(self, width : int, height : int): ...
def update_tf(self, new_tf_root): ...
def collect_data(self): ...
def cleanup(self): ...Sensors are tracked using the DataCollector class. This class regularly checks for updates in the Isaac Sim stage and automatically adds/removes sensors from its tracking list.
When a new sensor is detected, its type is automatically inferred based on criteria that can vary from one sensor to the next. Cameras can be identified with a simple class check, IMUs correspond to a certain type, and Articulations have a specific property:
if sensor.IsA(UsdGeom.Camera): # class check
sensor_type = "camera"
elif sensor.GetTypeName() == "IsaacImuSensor": # type check
sensor_type = "imu"
elif "PhysicsArticulationRootAPI" in sensor.GetAppliedSchemas(): # property check
sensor_type = "articulation"
# If new sensor types were added, they would likely require
# yet another identification methodAt every simulation step, the DataCollector class queries each enabled sensor in its tracking list for its data. It then packages everything into a dictionary mapping the sensor paths to the corresponding data message payload, before sending it to the Foxglove wrapper for transmission.
In this last section, let’s recap what would be needed to add a new kind of sensor to the extension base on what has been covered in this post.
Inside schema.py:
type2schemaInside data_collection.py:
add_sensor()IsaacSensor class __init__() functioncollect() in the IsaacSensor class, making sure it returns the proper payload based on the chosen schemaInside ui_builder.py [Optional]:
We had a lot of fun with this and hope you can too. If you haven’t already explored Foxglove for all your robotics data visualization needs, sign up here and check it out! This extension will appear in the Extension Manager within Isaac Sim, but behind the scenes, when you click Install, it pulls from this repository. If you have any questions be sure to join our community and let us know.
