Announcing: Data Loader in private beta.

Loading custom file formats in Foxglove is now in private beta.

If you have data in a format that Foxglove can't understand, you can now write a small WASM binary that runs directly in the app, and teaches Foxglove how to read your data. Where previously you would need to convert everything to MCAP, you can now drag these files straight into the Foxglove app.

Get going faster.

At Foxglove we love MCAP. However, with buckets and buckets of logs and recordings, it can be a big investment to convert everything before you can even open your files in the Foxglove app.

The Data Loaders extension API is a perfect middle-ground. Using C++ or Rust, your code will integrate deep inside the Foxglove app and serve up all the data needed to populate plots and panels during playback.

Resurrect that ancient format that nothing supports anymore. Or just add support for your proprietary logs, and iterate a little faster.

Publish your extension to your Foxglove organization, and now your entire team can open all your files, seamlessly.

To give you an idea of how to implement a data loader, let's implement a CSV reader that would allow you to load your data seamlessly. The same process can apply to any custom type - if you can read it, we can visualize it.

Build a csv data loader.

Since our SDK uses all the standard Rust traits, it’s straightforward to glue together libraries from the Rust ecosystem to build up your data loader. Let’s use the popular csv crate to read our file and teach Foxglove how to treat it.

Let's start by implementing the DataLoader trait and add a method for initialization. Here we’re reading the first line of the CSV and creating a new channel for each of the columns:

impl DataLoader for CsvDataLoader {
    fn initialize(&mut self) -> Result<Initialization, Self::Error> {
        // Open a new CSV reader
        let mut reader = csv::ReaderBuilder::new()
            .has_headers(true)
            .from_reader(reader::open(&self.path));

        // Load the first line of the CSV as the header 
        let headers = reader.headers()?;

        let mut init = Initialization::builder();

        // Add a Foxglove channel for each column in the CSV header row
        for column in headers {
            init.add_channel(column).message_encoding("json");
        }

        Ok(init.build())
    }
}

When Foxglove needs to read back data, it goes via the MessageIterator trait. Here we’re reading each row and creating a Foxglove message for each cell. We’re returning JSON for simplicity here, but you could return any message encoding supported by the Foxglove SDK.

impl MessageIterator for CsvMessageIterator {
    fn next(&mut self) -> Option<Result<Message, Self::Error>> {
        loop {
            // Emit all the messages read from the current row one at a time
            if let Some(message) = self.current_row.pop() {
                return Some(Ok(message));
            }

            let Some(column) = self.reader.next() else {
                // If we've got no columns
                return None;
            };

            // Treat the first row as
            let Ok(log_time) = column[0].parse::<u64>() else {
                return Some(Err(anyhow!("failed to parse log time")));
            };

            // Skip if a later timestamp was requested
            if log_time < self.start_time {
                continue;
            }

            // Push all the cells from the column to the current row
            for (channel_id, cell) in column {
                self.current_row.push(Message {
                    log_time,
                    publish_time: log_time,
                    channel_id: channel_id as _,
                    data: format!(r#"{{"value":"{cell}"}}"#).as_bytes().to_vec(),
                })
            }

            continue;
        }
    }
}

Once you’ve completed the implementation, package it up as a Foxglove extension and install it in the app. Now you’ll be able to drag your CSVs into Foxglove and get visualizing:

For a more fleshed out version of a csv data loader, check out our examples repo.

Join the private beta now.

Data Loaders is an experimental API that will become generally available in the app soon. In the meantime, check out our guide on building a data loader. If you’re interested in using the API now, contact us at support@foxglove.dev to join the private beta.