That was quite a long discussion you had with <@89...
# general
j
That was quite a long discussion you had with @Brian Pribe 😄 Here some comments: - go check out benthos-umh (it is also included in UMH Lite), as it will have the required connectivity from s7 to mqtt, kafka, postgres/timescale, etc. available as a docker container, or simplty as a binary (I know @Diederik is using it even on a windows machine) - you can also send multiple tags per MQTT message, see also our _historian schema/datacontract https://umh.docs.umh.app/docs/datamodel/messages/_historian/ - it is perfectly fine to send data in a high frequency into MQTT and Kafka, you just have to know how. What I did a couple of years back was connecting a sound sensor using benthos to MQTT. The trick was to buffer the data for a second and then send it out, so that you dont have 10kHz put instead only 1 message per second. Then FFT etc. could work on the data for a second.
also, lets start a thread, to keep the overview 🙂
h
Right, I forgot that discord has threading. Thanks!
I'll definitely look into how th s7 and beckhoff ADS implementations are handled in benthos-umh and use that to inform how I manage the data connection
d
I'm indeed using it as a binary (.exe) on windows machines) It was perfect when my costumer was using it for a 100 tag folders (+-20 tags/sec) now he switched to 1000 tag folders... (+-50 tags/s) and now i'm facing scalability issues (we're testing frankenstein gateway as an alternative atm)
h
Do you think the scalability issues you are seeing are related to benthos-umh or to the binary or to some other component? From the little bit of reading I've done, it looks like benthos should happily handle that message volume?
d
well that's still unclear indeed it should handle it easily. maybe its because of the binary maybe its user error maybe its my opcua server still investigating :p