I had with @DanielH a good discussion today on making the bloblang part more easy. We were thinking of something like this (I modified what we had in our meeting after discussing with @Mateusz ):

yaml
tagProcessor:
  defaultProtocolSettings: opcua

  customProtocolSettings: # alternative to specify more details to defaultProtocolSettings
  - tagFolder: meta("opcua_tag_group")
  - tagName: meta("opcua_tag_name")
  - tagType: meta("opcua_tag_type")

  enterprise: "enterprise-of-kings"

  groupConditions:
  - if: always()
    groupTo: 
      - site: "jeremy-vm"
  - if: meta("modbus_slave_id") == 10
    groupTo:
      - area: "daughterCompanyA"
        line: "line_ABC"
  - if: meta("modbus_address") == "B123"
    groupTo:
      - tagName: "temperature"

What are your thoughts on this all?

something like this:

yaml
tagProcessor:
    protocol: opcua

    setTagOverride:
        - tagFolder: "opcua_tag_group" # optional
        - tagName: "opcua_tag_name" # optional 
        - tagType: "opcua_tag_type" #optional 
        - tagMetadata: # optional 
            - serial_number: "123"
    renaming:
        - {
        "thisIs_a_very_4234VB_long.OPCUA.stupidVendor_notMakeAnySenseName": "tempInCabinet"
        "thisIs_another_very_4fa99434aB_long.OPCUA.stupidVendor_notMakeAnySenseName": "pressureInCabinet"
        }

    grouping:
        group:
           - from: * # group all
           - name: *
           - to: "site" # allowed is "site", "area", "line" "machine", To name if not used
           - name: "jeremy-vm"
        group: 
            - from: "modbus_slave_id"
            - ifMatched: 10 # optional, if not used all matches from - from: will be used
            - to: "area.line"
            - name: "daughterCompanyA.line_ABC"
        group: 
            - from: "modbus_address" 
            - ifMatched: "B123"
            - name: "temperature"
            
        group: 
            - from: "modbus_address" 
            - ifMatched: "A56"
            - name: "temperature"
            - jsonOutput: true # combines all values to a json struct 

  replaceAll:
   - tagFolder:
       - "/[^a-zA-Z0-9_]/g": "" # remove all special characters from tagFolder

We have now the attached file as proposal with the following reasons: - we need to include the meta() so that the user can match on metadata as well as the value itself with this.content(). It might not be that intuitive for OT people, but required to not cause confusion for IT people. I think overall we made an improvement for OT people. - we have basic grouping, basic renaming etc. all at the beginning - only then follows the advanced mode with conditions What are your thoughts @Brian Pribe @Diederik ? would you prefer this over the current bloblang implementation? would you give this to one of your customers for self-service? Also @Mateusz to give his input as well 🙂

For me currently it's difficult to see the value/why this should be changed. And certainly why this should be made more complex (my impression on first sight) I'm no bloblang expert (as seen by the issues i had last week) Maybe we should have a teams meeting with multiple people what/why/how/... I cannot give useful thoughts before understanding this all better. But obviously i'm interested to learn more! 😄

I’m in the same boat as Diederik. I’m working towards using bloblang, but I’m still at the POC stage of getting data in and doing rudimentary visuals. My biggest concerns are with installation and management of UMH and other services on-prem and at the edge. NodeRED is doing just enough for me to get by, but obviously scale will be an issue and this is where bloblang will come in.

i can confirm this is the way :p @Brian Pribe maybe you should take a look at my current issue before you go too far hahaha

Have you tried using a script instead? that could shorten everything. Or create subnodes.

Holy F*ck

everything has to be understandable by my costumer

i know you really don't like it, but currently I see no way around it without causing more confusion :/

Our concerns with installation and management is being able to do this at scale. Whether customers want to manage it themselves or we are managing their’s.

Let me add two more ideas: 1. batch messages of the last second into a single JSON payload (I think wish from @krgrsebastian , @Diederik, @DanielH ) 2. in the management console, don't show the full YAML. Instead show multiple text boxes e.g., a text box for conditions.

Took me a minute to fully understand what it's doing, but this gives you a way to normalize the topic structure of another broker into a UMH structured broker. If I understand correctly in this example, you can take all of the topics, configure the topic variables and their values that it will be comparing to transform, compare those topics with the variables to see if there's a match, and redistribute the topics into the new broker with added rules to validate the payload.

Exactly! But the code looks a little bit scary. But I think it would be possible easily to transfer the logic to the configuration I pasted, let me try out later

yaml
customProtocolSettings:         # Protocol Overrides
    tagFolder: "mqtt_topic"
       useMeta: true # default is true i omitted
       stringManipulation: "replace"
          type: "all" 
              - "/"
               - "."
    tagName: meta("mqtt_topic").split('/').slice(-1) # take the most right part of the topic as the tagName
    tagName: "mqtt_topic"
       useMeta: true # default is true i omitted
       stringManipulation: "slice"
              - "-1"
    tagType: string # convert everything to a json

How about something like this? i don't know if it makes any more sense or not?

Still no fan of the meta flag :). it I love the idea with the string manipulation, let me look at it tomorrow!

I think that if the processor is supposed to make everything easier and more intuitive the meta should not be used. If you have a optional useMeta: false which is default true then you could skip meta in case you are doing something special.. for ease of use just skip all programming mumbo jumbo and keep a clean an easy config.. in my opinion this is the easiest way for a "normal" user to understand it.. for anyone that is more advanced the bloblang should not be a problem either.. In my case i am always thinking about the next person that should maintain the system and what potential skills they have, and from my experience both from my current workplace and others the IT-level(programing) in the industry is quite low..

I’m in favor of configuration files, especially for templates and a standardized ui, because many of the end users will get frustrated with bloblang as the only way. My experience with bloblang isn’t great at the moment but ChatGPT, google, umh blog, and practice helped understand how it works, but I was getting frustrated, and I have more patience for these things than most. The configuration files can help to minimize the need to understand bloblang syntax and minimize errors on the output, with a common output ready for umh Uns to consume. This reminds me of telegraf, but I’m not a fan of toml. So I was thankful to find starlark. It’s much more in line with yaml, so if you’re used to homeassistant, or docker compose this feels familiar.