https://www.umh.app logo
Join Discord
Powered by
Deploy failed on custom bridge DFC
# umh-support
t
I want a _local schema kafka to mqtt bridge....so I can see my results before I move it to the _historian schema.....how do I do that? I went in to set up my own custom DFC and started with the kafka_to_mqtt_bridge template.....then changed the schema to _local....but I get deploy failed...it says it can't connect to the broker....I didn't change this, I assumed it was the internal address to the broker.
j
@Denis moving the discussion in here. Having the code would be nice and the exact error message
t
Ok, thank you for replying. I did get the custom DFC bridge to work from kafka to MQTT with the _historian schema.... For some reason, even though the MQTT consumer DFC is working, the kafka to MQTT bridge is working....but the Kafka to postgresql is not working...I'm not getting any data in the tag or tag_string tables...and there is no "throughput"
payload that the kafka to mqtt bridge is passing... {"MQTT.Actual_diameter":241,"bridged_by":["protocol-converter-hetzner-umh-proveit-broker-1","kafka_to_mqtt_bridge_historian_default"],"timestamp_ms":1737117534543}
j
@Athavan kanapuli can you check?
t
The IOT sensors simulator works..... BTW, why is the template set to use content().string() when the simulator is generating numbers?
is it the topic having a space? umh/v1/Enterprise/Dallas/Press/Press 103/_historian/MQTT
ok, yes, you cannot have a space in the enterprise,site,area,line,workcell values
now I just have to figure out why it's treating all of my values as strings...when they are numbers...
I replaced all spaces with "_", this helped....but the prostgresql bridge needs to handle this gracefully, as there is no notification that this is an issue, other than it doesn't work. I got this figured out.....do not put a . in the tagname variable When you do, it bridges to MQTT, but doesn't go to the historian. If you leave it out, then it goes to historian, but doesn't get bridged to mqtt. I put the prefix I wanted under tagname, and my tag name in payloadTagname.... payloadTagname variable needs better documentation, it's not mentioned in the docs on the right of the mqtt protocol converter.
Here is what I ended up with that works....
Copy code
# Get MQTT Topic, replacing spaces in topic with _
let topic = meta("mqtt_topic").replace_all(" ", "_")
# Set Hierarchy from topic
let enterprise = $topic.split("/").index(0)
let site = $topic.split("/").index(1)
let area = $topic.split("/").index(2)
let line = $topic.split("/").index(3)
let schema = "_historian"

let tagname = "MQTT"
let mqtt_topic_appendix = $topic.replace_all($enterprise + "/" + $site + "/" + $area + "/" + $line + "/MQTT/","")
let mqtt_topic_appendix_sanitized = $mqtt_topic_appendix.replace_all("/", ".")
let payloadTagName = $mqtt_topic_appendix_sanitized

let sanitized_content = content().string()

let value = match {
  # Remove null/empty values
  $sanitized_content == null => deleted()
  $sanitized_content == "" => deleted()
  
  # Handle special string cases
  $sanitized_content == "True" || $sanitized_content == "true" => 1
  $sanitized_content == "False" || $sanitized_content == "false" => 0

  # Try number conversion with error handling
  this.string().trim().catch(this).number().catch($sanitized_content) != null => this.string().trim().number().catch($sanitized_content)
  
  # Keep non-numeric strings as is
  _ => $sanitized_content
}
j
Hi @trentc , thanks for providing your experience here, we are going to have a look on the documentation regarding 
payloadTagname
and maybe mentioning characters, which should be avoided explicitely somewhere more intuitive to improve user-experience.
Regarding that issue we documented the allowed characters here. This should be linked in sidebar as well. https://umh.docs.umh.app/docs/datacontracts/ On top of that you could use the Topic validator above to check on that.
t
Thanks! It seems that using payload tag name and tag name works fine when I only want to have two levels of hierarchy But if I wanna have multiple levels of hierarchy, what do I do? When I use a dot, it breaks the bridge, when I use a / it just keeps it in the tag name and doesn’t treat it as another level if I put a dollar sign then my grafana Tag browser builds the hierarchy, but the hierarchy is broken in the umh console browser, and it just keeps the dollar sign but also the generated queries are wrong
I understand i need to avoid special characters such as in the docs.... however, if I have multiple levels of mqtt topic hierachy....how is that intended to be handled?
j
When using protocol converters you are using Kafka, so instead of “/“ use “.”
t
when I do that, it breaks the historian
when there is a dot in the payloadTagname, it doesn't make it to my historian...
j
What do you mean it breaks?
t
it just doesn't pass through the bridge
never makes it to postgres
I take that back, it makes it to the historian but treats it like a string....and I get null values...
j
@Ferdinand can you take a look here? this looks like a bug in our bridges
f
@trentc Can you show me a sample input you are sending ?
t
Copy code
# Get MQTT Topic, replacing spaces in topic with _
let topic = meta("mqtt_topic").replace_all(" ", "_")
# Set Hierarchy from topic
let enterprise = $topic.split("/").index(0)
let site = $topic.split("/").index(1)
let area = $topic.split("/").index(2)
let line = $topic.split("/").index(3)
let schema = "_historian"

let tagname = $topic.split("/").index(4)
let mqtt_topic_appendix = $topic.replace_all($enterprise + "/" + $site + "/" + $area + "/" + $line + "/" + $tagname +"/","")
let mqtt_topic_appendix_sanitized = $mqtt_topic_appendix.replace_all("/", "$")
let payloadTagName = $mqtt_topic_appendix_sanitized

let sanitized_content = content().string()

let value = match {
  # Remove null/empty values
  $sanitized_content == null => deleted()
  $sanitized_content == "" => deleted()
  
  # Handle special string cases
  $sanitized_content == "True" || $sanitized_content == "true" => 1
  $sanitized_content == "False" || $sanitized_content == "false" => 0

  # Try number conversion with error handling
  this.string().trim().catch(this).number().catch($sanitized_content) != null => this.string().trim().number().catch($sanitized_content)
  
  # Keep non-numeric strings as is
  _ => $sanitized_content
}
Note that the values coming in are not json....they are flat mqtt with just a value, some are strings, some are numbers.
f
Ok, so you are sending non json values to the 
_historian
?
t
i'm using the protocol converter....it should be json coming out....but i cannot tell because it's not bridging to mqtt
f
Yes the protocol converter will always output json
t
last I had that working, it was json when bridged kafka to mqtt
wait...it is working currently....mybad
but only for some topics
I think if I put the '.' back in, the bridge works....but then all of my tags are treated as strings....
f
Ok, but thats a different problem. So let's use the dots
t
i'll use the dots...
f
So this Line.Waste entry that is shown as an number on mqtt is now inserted as a string into postgres/timescale ?
t
Line.Waste isn't the problem....it comes through fine...
Look at the Line.OEE.{topic} values
the problem is the '.' in the payloadTagname var
you can see the '.' carries through into the historian as a string.
f
yes, null values are threated as string. Can you try replacing that dot with for example an - So like OEE-{topic} ?
t
see the name in the grafana query, but it doesn't match the payload below....
I had it replaced with a $ and it treats it as a number
I'll change to '-' for testing
f
Not sure how i would fix that, but ill add it as bug
t
I'm also available for a call if you want to walk through it together...
f
sure, let me find a room
For future reference: This issue was caused by our current kafka-to-postgres bridge not being able to handle dots inside the payloadTagName correctly
2 Views
PreviousNext
Powered by