Issue: Out of order MQTT messages in uplink ?

Already tried that comparison with Kafka? It’s exactly one of the reason we are using that at the core: Guranteed processesing + right order of messages

I am using UMH Lite...no kafka....

So it’s only about: reading from an API (like OPC/UA or other MQTT broker) -> Push it to UMH classic. Right?

processing performance testing.... Python -> MQTT broker -> UMH Lite -> MQTT broker -> Python

Ok. But the limit (in our experience) might likely be either the MQTT client libary / broker on that test

at 1 msg/sec it stays in order, at 10 msg/sec it stays in order, at 100 we get a few that get out of order...200 the out of order messages double

MQTT is not designed for “in order processing”

understood...but why can the same broker, same test, with node red...work fine

Is it about getting out of order. 1,2,,3,4 -> 1,4,2,3 or doubling / loosing messages

out of order

KK - TBH interesting behavior that node red handles that

I get it...it doesn't have a mechanism to maintain order....but at the same time....only 100 msg/sec gets it shuffled...

At the DB level it gets ordered anyway via the timestamp property

yes it gets ordered...but is the query mechanism efficient? or because of the timestamps out of order....causing performance inefficiencies?

It gets batched over the Intervall of one second and inserted in the batch order. You “order” it actually via the sort by mechanism when fetching it from the DB

got it....

The only way we see to reliably get messages on the event level in the order you put it in is actually Kafka / Redlands

maybe it's not an issue for data transport....really my goal was to see if I was missing messages....and monitoring performance....I will re-vamp my test to tolerate out of order sequence

This is what is “normal” in MQTT, and often an accepted trade off to speed up the total amount of messages. Theoretically, you could reduce the total amount of messages per batch, and increase the QoS and therefore achieve more order and less lost messages. But you will never be able to achieve no message loss, that is theoretically impossible. For this you need to have a different approach such as Kafka

I am really just trying to get a performance measurment comparison between node-red and umh-lite.... I can tolerate out of order messages...but I wanted a way to measure performance and find the limit by monitoring for message loss...

Then I recommend you to use UMH Classic as this is where we conducted all our benchmarks on it. There is for example an automated tests that sends in 10k msg/sec and checks for a certain duration if all messages arrive in the database as well

I appreciate the recommendation...but it doesn't test it on low power hardware....because classic can't run on a pi...which is what we are attempting to use as an "edge device".

I can fully understand that :/ it seems that the current approach of UMH Lite has failed because of the MQTT approach and the missing store and forward capabilities. What we could do is design together UMH Lite 2.0, where: - the protocol converter first write into the local Redpanda - from there on, one can bridge it via a Kafka to Kafka bridge to other UMH instances - and it needs to run on arm and in a docker container

For reference....here is a 1 min test, looks like my current setup maxed out at 3400 msg/sec Which honestly is pretty good that gets me 3 assets up to 1000 msg/sec.... or 100 data points every 0.1s