https://www.umh.app logo
Join Discord
Powered by
Issue (Closed): Out of order MQTT messages in
# umh-support
t
Issue: Out of order MQTT messages in uplink ?
So I decided I wanted a performance comparison between node red and umh-lite for processing mqtt data. So I am literally just publishing an incrementing number and watching for when it is not following the sequence....looking for missed messages...
I discovered it publishing out of order...
Seems like it shouldn't be doing this at such a low rate of messages....and it's ideal for ingestion to be FIFO....to keep timestamps in order...
a
Already tried that comparison with Kafka? It’s exactly one of the reason we are using that at the core: Guranteed processesing + right order of messages
MQTT we currently see more like a connector / ingress
(And API for AGVs)
t
I am using UMH Lite...no kafka....
this is UMH Lite, running in docker, on a Raspberry Pi 4 device....
Also, at 5000 msg/sec it is eating CPU......87%.....where node red handles this all day at less than 70%...with some spikes up to 75%
a
So it’s only about: reading from an API (like OPC/UA or other MQTT broker) -> Push it to UMH classic. Right?
To check the performance ?
t
processing performance testing.... Python -> MQTT broker -> UMH Lite -> MQTT broker -> Python
no UMH classic in the mix....literally only publishing from a python script with a "messsage_id" that is incrementing in sequence....then when received back at the python script, it checks that its is following a sequence....
a
Ok. But the limit (in our experience) might likely be either the MQTT client libary / broker on that test
t
at 1 msg/sec it stays in order, at 10 msg/sec it stays in order, at 100 we get a few that get out of order...200 the out of order messages double
a
MQTT is not designed for “in order processing”
t
understood...but why can the same broker, same test, with node red...work fine
?
on same device....
less cpu...
a
Is it about getting out of order. 1,2,,3,4 -> 1,4,2,3 or doubling / loosing messages
t
out of order
a
KK - TBH interesting behavior that node red handles that
t
I get it...it doesn't have a mechanism to maintain order....but at the same time....only 100 msg/sec gets it shuffled...
many timeseries DB work better to ingest data with timestamps in order....so if possible...this is best....
a
At the DB level it gets ordered anyway via the timestamp property
t
yes it gets ordered...but is the query mechanism efficient? or because of the timestamps out of order....causing performance inefficiencies?
it likely gets ordered in the query...not at insert
a
It gets batched over the Intervall of one second and inserted in the batch order. You “order” it actually via the sort by mechanism when fetching it from the DB
(Thin ICE for me as I didn’t wrote that code) 😄
But performance is no issue
t
got it....
a
The only way we see to reliably get messages on the event level in the order you put it in is actually Kafka / Redlands
Redpanda
t
maybe it's not an issue for data transport....really my goal was to see if I was missing messages....and monitoring performance....I will re-vamp my test to tolerate out of order sequence
j
This is what is “normal” in MQTT, and often an accepted trade off to speed up the total amount of messages. Theoretically, you could reduce the total amount of messages per batch, and increase the QoS and therefore achieve more order and less lost messages. But you will never be able to achieve no message loss, that is theoretically impossible. For this you need to have a different approach such as Kafka
Or you can tune the values a little bit, and reduce the amount of total messages you can send
t
I am really just trying to get a performance measurment comparison between node-red and umh-lite.... I can tolerate out of order messages...but I wanted a way to measure performance and find the limit by monitoring for message loss...
j
Then I recommend you to use UMH Classic as this is where we conducted all our benchmarks on it. There is for example an automated tests that sends in 10k msg/sec and checks for a certain duration if all messages arrive in the database as well
Happy to sit together with you at the beginning if next week and set something up
t
I appreciate the recommendation...but it doesn't test it on low power hardware....because classic can't run on a pi...which is what we are attempting to use as an "edge device".
j
I can fully understand that :/ it seems that the current approach of UMH Lite has failed because of the MQTT approach and the missing store and forward capabilities. What we could do is design together UMH Lite 2.0, where: - the protocol converter first write into the local Redpanda - from there on, one can bridge it via a Kafka to Kafka bridge to other UMH instances - and it needs to run on arm and in a docker container
Other suggestions?
The tests with Redpanda are quite good, we have 10k msg/sec test systems in our office where it is running good
One more idea: with custom data flows you could implement the store and forward yourself! There is caching functionality in there, and you could tune the MQTT output parameter to your liking (e.g. increases QoS)
t
For reference....here is a 1 min test, looks like my current setup maxed out at 3400 msg/sec Which honestly is pretty good that gets me 3 assets up to 1000 msg/sec.... or 100 data points every 0.1s
it missed some messages in this test...but you can see the "received rate" was ranging from 3300-3400....
call it 3000 msg/sec with some headroom.
It does eat up cpu though....85%
Connecting straight to Kafka for all of the benefits from UMH lite would be great as long as we still get a way to bridge Kafka to MQTT, even from the lite instance
Yes store and forward or cache capability that is configured could be useful
PreviousNext
Powered by