Message path downselection.

One of the most commonly used tools within Foxglove is the Plot panel. Whether you’re tuning a control loop, or tracking down a flaky connection, numeric line plots are invaluable. 

If you’ve much spent time plotting values from your messages in Foxglove, you may have asked yourself this question:
Why is this plot taking so long to load?

The correct answer is “it depends”. We’ve spent a lot of time and effort improving the performance of our plotting frontend over the last year.  However, over the last 6 months, we’ve also been improving our cloud backend for plotting workloads.

Consider the problem of checking timing jitter in a PointCloud topic. A good way to observe timing problems visually is to plot the header timestamp of each message against its log time. Previously to draw this plot, Foxglove would download each pointcloud, copy the timestamp value, and throw away the point data. You might download gigabytes of points to display a few thousand timestamps! Depending on your connection, this might be fine, but for most of us, it’s not.

In order to solve this, we’ve built message-path downselection into our cloud service. This means for most supported serialization formats, when plotting values using message path syntax, you only download the bytes you need.

To get this to work at scale, we’ve had to build custom high-performance deserializers for several formats in Rust. This was grueling work, and we’ve left a trail of upstream bug tickets in our wake. However, the results are already paying off, and we’re seeing up to 80% decreases in plot load times. We’re also looking forward to leveraging this code in other parts of our application.

If you store your recordings in Foxglove Cloud, you’re already seeing the benefits of this work. If your organization manages its own Foxglove Primary Site, you may need to upgrade to take advantage of this feature.

PS. If this sounds like the kind of feature you’d like to build, we’re hiring!