Hi, We're testing the sparkplug_b input plugin with devices that use protobuf extensions on metric.metadata (custom fields like sub-ms timestamp info). Currently the plugin decodes the standard payload fine, but silently drops any extension data. Extensions are a standard part of the Sparkplug B proto2 spec and are being expanded further in Sparkplug 4.0 (eclipse-sparkplug#224). Is anyone else here using Sparkplug B extensions that they'd like to be able to access? kind regards Pascal

@Aaron is currently working on SparkplugB

Hi @pascaldesmet., could you give us some more context on which extensions you want to use and what data is being dropped?

Hi @Aaron , here's the specifics: We use two protobuf extensions defined in our proto : 1. MetaData extension (extend sparkplug_b.Payload.MetaData, field 9): timestamp_ns (uint32) — nanosecond component to complement the standard ms-resolution SparkplugB timestamp. (and some other specific additional info like txstate, seq, ack, states[] / bits[] for state machines) 2. MetricValueExtension (extend sparkplug_b.Payload.Metric.MetricValueExtension, field 1): (Custom value types for machine-specific data: for CAN bus messages, IO configuration lists, error info, etc. ) What gets dropped: The benthos-umh sparkplug_b input plugin currently decodes the standard SparkplugB payload correctly (basic metric values like UInt16, UInt32, Float etc. all work fine). But both extension blocks seem to be silently discarded during protobuf decoding (which completely makes sense given the fact that no extensions definition is declared) What we'd like: At minimum, access to the MetaData extension fields as additional metadata headers (especially timestamp_ns for precise timestamping). Ideally the MetricValueExtension would also be accessible, either as raw bytes or decoded if a proto descriptor is provided. One could argue that it might not have been the right choice to make use of extensions, but that was a design choice at that time and is out of our scope, so we would like to be able to deal with it. Our proto files are based on the standard sparkplug_b.proto extensions mechanism (proto2), which SparkplugB explicitly supports. We are aware that the use of extensions in proto3 is no longer advised, but as far as we know sparkplugB still implements proto2 and we don't assume this will change.

@Pascal Desmet, I could add an opt-in option that attaches the unknown fields to each message as base64 metadata, split by source. You decode it downstream with your own proto. For timestamp_ns that's just a proto.Unmarshal into your type. I would rather not decode your extensions in the plugin itself, since that bakes your proto into our host. So it's raw bytes you decode on your side, not ready-made headers. Would that work for you?

@Aaron the opt-in option for attaching the unknown bytes seems like a reasonable solution, but by "decoding downstream" i suspect you mean outside benthos, not something that could be done using bloblang I suppose? I fear that would mean a devaluation for the use of UMH-core as the central system . The other more complex option of course with the requirement to keep things generic, would be to allow the user to add the descriptorSet as a file (or even the extensions.proto, but that would need a proto compiler inside the application...), it get's read into Go's protoregistry and proto.GetExtension() reads and maps the data to the corresponding metadata fields (e.g. spb_ext_...)

there is an option to decode protobuf with the benthos protobuf decoder, maybe we can find something using that? so expose it, and then use a seeprate standalone or something to extract it fully?

ok, using this https://docs.redpanda.com/connect/components/processors/protobuf/ in a data-flow of type "stand-alone"? With input and output both uns for instance. Possibly, yes that could be a solution for certain types of messages.

we were also thinking od addign a protobuf.encode/decode to nodered_js, so it works perfectly using bridges / in the UI as well. See also https://docs.umh.app/benthos-umh/processing/javascript-api

So to summarise @pascaldesmet. what we could to is 1: Attach the extension data to the sparkplug messages using an opt-in flag and 2: to add a new decode function to our nodered_js plugin such that you can decode the extension data in the same bridge. What do you think about that?

Yes, that seems like a good solution. Keeping it in the same bridge keeps it lean and efficient. It allows to filter on some specific messages where we want to extract extension data. for all the other messages we waste no decoding power and just dump the data before shipping to uns.

Alright. I will take care of it then

I did a first limited test and it works flawless. Thanks!

Great to hear 😄