https://www.umh.app logo
Join Discord
Powered by
Hi everyone, I am new to the UMH stack
# umh-support
s
Hi everyone, I am new to the UMH stack and slowly piecing together how it everything works. I am running into a couple of issues particularly with protocol converters. I am trying to use bloblang for the processing. In the bloblang script I am attempting to parse the conent as json, and extract the relevant fields from the payload. As there is only one topic to which I can subscribe I need to generate the tag directly from one of the payload fields as the same topic to which I am subscribing has multiple messages from different sensors all with varying number of fields. My bloblang script is unable to parse json and results in Null Values. I have tested the script and multiple variants in benthos studio and redpanda playground. I have attached a screenshot of the script in benthos studio which I have been using to debugg. Furthermore I am receiving a variety of warnings within the connections and data flow listed below: 1. Logs found with error indicators for the pod dfc-9c1e0bc3-3ac2-41bf-9eda-bd17cec3bbd1-test-7b848b6ddf-hpz6k. Logs: level=info msg="Running main config from specified file" @service=benthos benthos_version="" path=/benthos.yaml 2. level=error msg="Failed to connect to mqtt: network Error : EOF" @service=benthos label="" path=root.input Input YAML: mqtt: urls: - ssl://dummyurl.emqxsl.com:8883 topics: - fortes/fs24-011/+/up/telemetry client_id: ClientID user: 'user' password: 'password' tls: enabled: true skip_cert_verify: true https://cdn.discordapp.com/attachments/984083100520808478/1313745386954821703/Benthos_Studio_Screenshot?ex=6751403c&is=674feebc&hm=16c9d9a755de85bfa754d669752c35635acc9996fb533cce6a05d24a2e2f9175&
d
so you want to subscribe to the entire gateway instead of only a specific device? try extract the payload using the keyword "this" https://cdn.discordapp.com/attachments/1313745387370188851/1313776840762064926/image.png?ex=67515d87&is=67500c07&hm=7c6b361bdd0b8b2b7bbc66814b5d0a6935ed33e608d38582755fdd2aeabe4259&
I see you are using loraWAN with milesight. are you using the inbuilt functionallity if the gateway? Have you tried chirpstack to manage the devices ?
j
I am looking through your code, and interestingly you were using benthos studio (which came up a coupleo f days internally here as well) My first though is that you do notneed to parse the paylaod as a json, you should be able to access it directly
let me give you an example
s
Hi @DanielH thank you for reaching out. I have tried the use of the explicit "this." to focus the context to the input. Sadly it does not alter the output. One of the challenges we are facing is that on the infrastructure side we essentially have no access as that was positioned by a third party. Essentially all we have access to is the broker and one specific topic through which we can subscribe to use for all the various sensor data.
@Jermuk Thank you very much, an example would be of great assistance. I tried the refactored version based on your suggestion to access it directly. This resulted in an improvement, The payload is being accessed directly as Json as you can see below. However the extraction of the fields within the payload still result in null values. *******new code******* # these variables are autogenerated from your instance location let enterprise = "GBTB" let site = "BaySouth" let area = "SB3" let workcell = "zone 4" let schema = "_historian" # Directly access the fields from the payload let tagname = this.deviceName let value = this.Temp_Soil let timestamp = this.time # Return the extracted tagname and value along with the raw content for debugging root = { "payload_debug": this, # Log the entire payload for inspection "raw_content_debug": content().string(), # Log the raw content for debugging "tagname": tagname, "value": value, "timestamp": timestamp } ******Output****** { "payload_debug": { "GatewayEUI": "24e124fffef9e7c1", "Temp_Soil": "22.30", "applicationID": 1, "battery": 100, "devEUI": "a8404131e1593054", "deviceName": "SPH01-LB-09", "pH_Soil": "5.75", "rssi": -85, "snr": 11.2, "time": "2024-11-30T05:30:18.110701Z" }, "raw_content_debug": "{\r\n \"GatewayEUI\": \"24e124fffef9e7c1\",\r\n \"Temp_Soil\": \"22.30\",\r\n \"applicationID\": 1,\r\n \"battery\": 100,\r\n \"devEUI\": \"a8404131e1593054\",\r\n \"deviceName\": \"SPH01-LB-09\",\r\n \"pH_Soil\": \"5.75\",\r\n \"rssi\": -85,\r\n \"snr\": 11.2,\r\n \"time\": \"2024-11-30T05:30:18.110701Z\"\r\n}", "tagname": null, "timestamp": null, "value": null }
j
Copy code
let enterprise = "GBTB"
let site = "BaySouth"
let area = "SB3"
let workcell = "zone 4"
let schema = "_historian"

let mqtt_topic_appendix = "hello123"
let current_timestamp = (timestamp_unix_nano() / 1000000).floor()

let tagname = $mqtt_topic_appendix

let invalid_payload = this.catch(true)
let payload = match {
    $invalid_payload == true => deleted(),   # Delete if empty payload
    _ => this                             # Keep the message if empty payload
}

# add timestamp_ms to comply with historian schema
let payload = $payload.merge({"timestamp_ms":$current_timestamp})

root = {
  "payload": $payload,
  "tagname": $tagname,
}
based on the examples here: https://learn.umh.app/course/mastering-data-transformation-with-benthos-and-bloblang-in-19-examples-2 this is somehow ridicilous hard, let us think how we can make this easier
(the root is obviously just for benthos studio)
what I noticed is that in order to access a variable you need a dollar sign in front of it
s
This is interesting, so essentially even when working with the root document and using the this keywoord you have to use the dollar sign to point to the root explicitly for each variable
Thank you very much @Jermuk another thing I am gathering is the timestamp has to match the format of "(timestamp_unix_nano() / 1000000).floor()" so even if I extract the timestamp from the payload it will have to be processed to fit the schema.
j
if you want to store it in the historian, then exactly!
if you want to use the incldued timestamp, feel free to checkout https://docs.redpanda.com/redpanda-connect/guides/bloblang/methods/#timestamp-manipulation
ts_unix_milli()
s
Hi everyone, i'm currently subscribing to an external broker for telemetry readings. However after processing in Node red we are not able to publish via the same broker as the third party will not enable publishing privileges for the clients. I know part of the UMH stack is a HiveMq broker but I don't know how to access the server details, authentication or how to configure the HiveMq broker through UMH. Any guidance to walk me through the process to get connected to the UMH HiveMq broker
j
The broker is by default available on port 1883
In the management console you will also find an example connection, which connects to the local MQTT broker using it’s internal DNS
Configuration is currently through the helm chart
But there are also tutorials for concrete things that you might want to configure
What do you want to configure exactly?
s
As it stands right now hopefully won't need to configure anything in the Helm Chart but will need to go through the docs in detail before any attempt. The moment you mentioned the Internal DNS it clicked. I replace the server details in node red with "tcp://internal DNS" However what I am noticing is that it actively connects and disconnects. https://cdn.discordapp.com/attachments/1313745387370188851/1315987948365152386/20241210-1018-11.2429296.mp4?ex=675968c8&is=67581748&hm=f6caaf30204c350e1fb2d71c0318517a39e33f2686d238ef9879f3c822e14926&
j
can you give nodered a unique client id? i remember this can be issues
s
I provided a unique client id in Node red and another unique client id in the protocol converter. Health checks pass but getting warnings: Warning (protocol converter) Logs found with error indicators for the pod dfc-e1cc3434-be07-4011-8cd1-980708b38f16-testnode-6d6677f5gd6z5. Logs: level=info msg="Running main config from specified file" @service=benthos benthos_version="" path=/benthos.yaml level=info msg="Listening for HTTP requests at: http://0.0.0.0:4195" @service=benthos level=info msg="Output type kafka is now active" @service=benthos label=kafka path=root.output_resources level=info msg="Input type mqtt is now active" @service=benthos label="" path=root.input level=info msg="Launching a Benthos instance, use CTRL+C to close" @service=benthos level=error msg="Connection lost due to: EOF" @service=benthos label="" path=root.input level=error msg="Connection lost due to: EOF" @service=benthos label="" path=root.input level=error msg="Connection lost due to: EOF" @service=benthos label="" path=root.input
t
@Ferdinand do you have any thoughts on what could be causing this frequent disconnect / reconnect cycle?
f
@Sami Can you give me the full data flow component config (minus any authentification) ?
Are there any logs on your MQTT server regarding these connection/disconnect events ?
PreviousNext
Powered by