UMH Lite 2.0
# generalj
Jermuk
10/21/2024, 8:50 AMUMH Lite 2.0
Jermuk
10/21/2024, 8:53 AM- Reliability and Data Loss Prevention: Users like @trentc and @DanielH raised concerns about UMH Lite’s lack of store-and-forward capabilities, which could lead to data loss during disconnections.
- Performance Issues with MQTT: There were observations about MQTT messages arriving out of order and performance limitations when using UMH Lite on low-power devices like Raspberry Pi.
- Role of Node-RED vs. UMH Lite at the Edge: Discussions on whether Node-RED could suffice at the edge and how UMH Lite fits into the ecosystem.
So we have thought about the next version of UMH Lite (UMH Lite 2.0):
- Redpanda Integration for Store-and-Forward: Implement Redpanda as the local data streaming platform to buffer data during network outages, ensuring no data loss.
- Include Node-RED as a versatile tool for quick integrations and edge logic, serving as a “Swiss Army knife” for edge computing needs.
- Configure UMH Lite to use KAFKA_UPLINK by default instead of MQTT_UPLINK for more reliable and ordered data transmission.
- Allow UMH Classic instances to expose their Kafka port by default when using KAFKA_UPLINK, facilitating seamless communication between UMH Lite and Classic.
What are your thoughts?
d
Diederik
10/21/2024, 9:04 AMDiederik
10/21/2024, 9:05 AMCould you also share UMH thoughs on crossing DMZs? (how will that work?)
in a real setup UMH lite instances and the UMH classic will not be in the same network
d
DanielH
10/21/2024, 9:06 AMSeems like the right direction. I would however like to still perhaps use mqtt for the UMH lite but with a small local DB that i case of network fails stores data locally and pushes the data when network gets back.. Also to have the ability to store data locally and do calculations at interval and then send only calculated data and specific times.
d
Diederik
10/21/2024, 9:07 AM@DanielH what do you work this way? you're combining 2 tools to get the functionallity that kafka brings by default? (what am i missing?-
Diederik
10/21/2024, 9:07 AMok, the calculated value at itnerval would indeed be nice
d
DanielH
10/21/2024, 9:08 AMyou can have edge nodes working independently and only send useful data.
j
Jermuk
10/21/2024, 9:14 AMit will work like this 🙂 each UMH Lite only has an ongoing connection, which is fine for most security people. then you have the UMH Classic inside of the DMZ, which you can point your lite instance to.
Jermuk
10/21/2024, 9:17 AMI need to agree with @Diederik. it would be combining two tools together, with an increased risk of failures. and you would still have the issue with lost data. This is something that is simply built into the MQTT protocol and you cannot fix easily. I'll be writing in one of my next articles on data modelling on it, but basically MQTT only gaurantees the devliery of messages to the broker, not to the receiving applications (the subscriber).
by having redpanda in there, it acts as a reliable cache (even through power outages). you could then apply all the standard stream processors to process the data / downsample it, etc. as you now it from teh UMH Classic. and then you could bridge only the topics you think are reelevant (e.g., only the downsampled data)
d
DanielH
10/21/2024, 9:31 AMyes sure. I understand that part. It was most for the ease of use to have mqtt everywhere. What i am meaning is that if you have a local DB on the edge you could for example do calculations and only send useful data but you could also compare data between local and cloud db and push whatever is missing to the cloud db. this could be done using mqtt. But you could for sure use redpanda but have a "converter/bridge" on the edge so if you push data via nodered on the edge to mqtt you would intercept and switch to kafka. hmm..
d
Diederik
10/21/2024, 9:47 AMhow can/will the kafka message "finds its way" to the server in the DMZ? ports will have to be opened on the firewalls/switches in between right?
j
Jermuk
10/21/2024, 11:09 AMexactly, this is what needs to be done. but opening a port in the DMZ, so that devices from the protected area can send data top that, is best practice in IT. i would not worry about it.
the other way around (the server accessing the clients) is what makes security guys scared
Jermuk
10/21/2024, 11:10 AM> i am meaning is that if you have a local DB on the edge you could for example do calculations and only send useful data but you could also compare data between local and cloud db and push whatever is missing to the cloud db
this is all basically baked into kafka :/ calcualtions you could do in the edge kafka to kafka
Jermuk
10/21/2024, 11:11 AM> It was most for the ease of use to have mqtt everywhere
MQTT has a lot of advantages, but it is in my opinion designed not for industrial use, it is designed for receiving telemtetry data. Kafka is not ideal as well.
b
Brian Pribe
10/21/2024, 12:44 PMNeither are ideal and this is why they are being brought together. MQTT is just too bare to fulfill all of the requirements in regards to state and events. Kafka should go as far as being on the gateway device.