Pretty much. Just more visual way to diagnose performance and filter through what I’d like to see in regard to logs. Some logs are just spam, other are exactly what I’m looking for. Like console.log messages I’ve put directly in my pipeline I’d like to filter and see, rather than have a constant spam of other logs I didn’t configure that are just messages.

@Roger @Yannic UMH ?

Hey Brian, sounds like a very good feature to me! However, I think we currently don't have the engineering capacity to implement such a big feature at the moment. We are working hard on other exciting things 🙂 What we however want to tackle in the near future is the entire "logs experience". This includes cleaning up logs from the container as well as displaying them in the frontend. We know that it is quite messy at the moment and hard to read for everyone, so we will make improvements on this front. With these improvements, you should hopefully be able to better diagnose your flows until we are able to work on performance dashboards etc.

@Brian Pribe idea, would in make sense to add log errors to metadata so you could extract this metsdata in the historian bridge and save to db? Or do you want to have metrics like throughput as well?

See I was thinking something similar. Either a performance dashboard for the management console or in grafana.

Where would it make most sense to have this? Do you work daily with the management console or do you worl daily with grafana?

Well when I’m building pipelines it’s in the console. With grafana, I would want alerts.

How is accessing the console on mobile for y’all? Doesn’t seem to load anything (on the water, probably just service)

So best of both would then be to have something like a console log in the management console, like down in the bottom where you see all errors, and then for grafana you would perhaps have a separate log table where we store errors and warnings for easy querying and history? This idea i like, only thing is to have a good filter in the bridges to only see relevant errors and warnings.

We do a webinar on the ROI of Unified Namespace, go check it out: https://www.umh.app/events/webinar-calculating-the-roi-of-a-uns

I have a support question but I see the channels for that is achived so asking here if it is ok. I am trying to setup a basic modbus-> Sparkplug B flow using benthos-umh Here is the config:

input:
  modbus:
    controller: "tcp://192.168.0.1:502"
    slaveIDs: [1]
    timeBetweenReads: "1s"
    unifiedAddresses:
      - "output_frequency.input.24.INT16:scale=0.01"
      - "output_voltage.input.25.INT16"
      - "dc_bus_voltage.input.26.INT16:scale=0.1"
      - "output_power.input.27.INT16" 

pipeline:
  processors:
    - mapping: |
        if meta("modbus_tag_name") == "heartbeat" {
          root = deleted()
        } else {
          # Payload
          root.value = content().string().number()
          root.timestamp_ms = timestamp_unix_milli()

          # Metadata
          meta location_path = "None"
          meta virtual_path = "None"
          meta tag_name = meta("modbus_tag_name")
          meta data_contract = "_sparkplug"
        }

output:
  sparkplug_b:
    mqtt:
      urls:
        - "tcp://localhost:1883"
    identity:
      group_id: "Edge"
      edge_node_id: "benthos-umh"

Modbus reads great, pipeline works. MQTT connection is made to my broker and node data is sent. But no reading are sent every second to the broker? I am thinking I did something that does not meet the requirements for the sparkplug_b output i.t.o DDATA or something?

@Hanno i copied your message here and gave you an answer: https://discord.com/channels/700613971941785680/1512867994802454679

I saw y’all added a snowflake_put feature. What do y’all think about that integration and approach? How do you think it compares with other data lake options like data bricks or ducklake? Have have a stack built with ducklake, but just haven’t gotten around to inserting the metadata into the Postgres db.

I usually dont come across snowflake often in manufacturing, i've seen Databricks (and Azure fabric) way more

I see azure fabrics mentioned a lot considering it’s Microsoft. I think it makes a lot of sense plugging in a data lake to a UNS

My company is landing data in snowflake, it’s a common pattern that highbyte supports directly. I don’t know much about it other than a lot of people I’ve spoken to commented on the cost of using snowflake 🤷🏼‍♀️

with UMH you can use batching policies which will decrease the costs significantly

But how do you push data to snowflake? using the put or do you use the streaming?

Honestly not sure I am on the data source side but I think we write directly into tables in snowflake.

We are looking to add a broker in the middle

ah. Ok. so this is basically the same as the snowflake put.

Snowpipe, Snowpipe Streaming, and bidirectional SQL are supported. Snowpipe Streaming does go direct to tables. In the new version of the API, the steam channels can intercepted for steam processing use cases. This is probably more appropriate for raw telematics or clickstream feeds than manufacturing data.

What’s a clickstream feed?

Observing UX activity. Used for e-commerce, fraud detection, debugging/optimizing front end designs.

Ah ! I got it confused with clickhouse and thought it was related 😂

Is there anyone looking for AI/ML engineer ?

I am looking for a partner for a long-term collaboration. Hello everyone, I hope you are all doing well. I am a Senior Full-Stack AI Engineer with over seven years of development experience, currently working as a freelancer. I am seeking a long-term partner based in the Europe. Ideally, I am looking for a male collaborator in his 30s or older who is fluent in English (and preferably other languages) and possesses excellent communication skills. This is a paid, long-term partnership involving a monthly revenue-sharing arrangement. If you are interested, please send me a DM with a brief introduction (including your name, country, profession, age, etc.).

anyone that has tried a decent compare benthos(-umh) vs telegraf?

why would you recommend to use benthos over telegraf?

redpanda connect / benthos has reliable message procesisng, telegraf is kidn of optimized to ingest data into influxdb and doesnt habe that reliability guarantees (which is also correct tradeoff for timeseries data!), but usually you also want to make somehting like addOrder or soemthign where telegraf is not good at