https://www.umh.app logo
Join Discord
Powered by
And here is one more where an MQTT node tries to c...
# umh-support
r
And here is one more where an MQTT node tries to connect without success. It stays like this for ever.
j
Creating a new thread
Some things you could double check which I remember were a problem: - sending messages without username might result in a disconnect (try ia_nodered) - sending messages with the same client ID will cause clients to throw the other one out
Otherwise, could you provide us with the log of VerneMQ. I remember that under certain circumstances VermeMQ might have a LevelDB corruption (especially if forcefully shut down like turning down the power). It can be solved either by upgrading (using HiveMQ now) or by restarting VerneMQ by scaling down the replicas of the statefulset to 0, wait a couple of minutes, delete the persistent volume, and then scale back up to 1
r
Hi! Thanks for replying! I'll give the username and client ID parts a try. The VerneMQ logs can be retrieved with Lens, right? I´ll install it and inspect the stack to check the volumes and replicas. I don't know what UMH version we have, but can you ugrade to 0.9.10 from any version or do I need to do it in steps?
j
i recommend doing it always in steps
r
I wanted to try the username part but nodered stops responding after a few minutes without allowing any changes. I think the best option is to upgrade and I am even thinking of installing the new image with flatcar, because I think the step-by-step upgrade does not install the new OS, does it? I think the DCC still runs the k3OS version. Is there anything other than the nodered flows and the grafana dashboards that needs to be stored for such an upgrade?
j
If you want to re-install it, it would be great. If it is at the Warping Machine, we will also help you as we are planning to do a video anyway on that machine. We would recommend also using the same approach in Venice, which means removing OPNsense and Proxmox and doing everything in flatcar instead. Let me send you the template, and then you can fill it out with the data you need for the warping machine. I think you only need to backup the IP ranges in OPNsense (just put them into the config file), the Node-RED flows (incl. the list of installed plugins) and (just to be safe) the values in the Helm Chart.
Please provide me with the following information: - Serial Number of the Karbon300 --> can be seen in iPXE or on the backside of the K300, starts with E - Disk type. Is it sda or NVMe? if you do not know it, we need to try it out - Information on the IP for WAN (DNS, IP, Gateway) --> OPNSense - Information on the IP for LAN (IP range, static IPs and their MAC and IP addresses) --> OPNSense - Information on the IP for PLC Network (IP range, static IPs and their MAC and IP addresses) --> OPNSense - Whether sensorconnect, barcodereader should be enabled --> Helm Chart - the server IP with the dashboard --> Helm Chart (under the section kafka bridge) I'll then prepare on your deployment server the correct configuration file. Then you only need to use our iPXE image. It will automatically detect the serial number of the K300 and for security reasons you need to then insert a token during installation as well. Besides that, the entire system will then sets itself up automatically.
If you want tailscale to be enabled, you can use at the end of the installation this command https://learn.umh.app/course/installing-tailscale-on-flatcar/#step-3-authorize-the-device-with-your-tailscale-instance to provide us with the tailscale link, so we can add it to our organization.
r
Excellent, thank you! I actually wanted to update all IPCs on the shopfloor to the newest version because all of them have MQTT connection problems. I have blocked time for that on Thursday. I will report back then or at latest on Friday. When are you planning to record the video at the warping machine?
j
Shall we say Thursday starting from 13:00? Then I'll also block some time there to help you in real-time and deploy the configuration templates (as soon as I have the information from you). The video is currently aligned with Nadine, likely to be first week in May.
r
Would Thursday at 15:00 work? I have another meeting at 13:00.
j
does work for me!
r
Great. Do you need a calendar invite or will you block it for yourself?
j
already blocked it for myself
r
One more thing, we are trying to connect to the thermosetting machine to review the helm charts but the password we have in bitwarden for that does not work. It should be the 1-12-k3os, right?
We were able to retreive some of the information you need from the firewall: LAN * Firewall's IP: 182.16.39.1 * IP Range: 172.16.39.100 - 172.16.39.245 * Static addresses: (MAC) a6:be:8d:64:19:0c; (IP) 172.16.39.10; (hostname) dccaachen-2-12 PLC * Available range: 192.168.1.1 - 192.168.1.254 * No static IPs
j
Strange, you can try rancher
Maybe someone used a default password instead when setting the Maschine up (I remember we had a couple of tries)
let me grab my lunch and then I'll prepare the first configuration file for you. shall we start with the warping machine? This is also the machine which we will use during our video shooting.
ok, let me start to prepare the configuration files
I can prepare them, but you still need to go over them and confirm that they are ok
ah, just saw that you meant 15:00 and not 13:00. dont worry, i'll do other stuff then
Ping me if you have confirmed the following configuration parameters for the warping (and added missing information): - Name: dccaachen-2-4 - Serial Number of the K300: Exxxxxxx (TODO: pls fill out) - Disk Type of the K300: sda or nvme (TODO: pls identify) - WAN: static, 192.168.1.151, DNS (TODO), Gateway (TODO) - LAN: 172.16.42.1, DHCP server, static lease for 750-8101 with 172.16.42.13 (TODO: missing MAC address) - PLC: 192.168.1.1, DHCP server, static lease for SIMATIC_HMI with 192.168.1.151 and SIMATIC_S7-1200 with 192.168.1.150 - sensorconnect enabled to connect to AL1350 - Node-RED flows pre-installed, I can use our latest backup of it, which is 6 months ago or you can provide me the current version (TODO: decide) - Server IP: 192.168.1.198. (TODO: missing port. Can be found in the current helm chart under kafka-bridge) - Password: same as before for k3os (TODO: pls confirm) - Tailscale/VPN/Remote Access: enabled (TODO: pls confirm) - Secure Token, so that only you can fetch the configuration file (TODO: pls send to me via secure channel or I can create one for you)
r
Hi
Yep, the warping machine should be alright. I am preparing my workstation at the shopfloor at the moment
I'll give you the information as soon as I have it
I am not going to focus on the SN today. I would have to demount the device.
and it is nowhere to be seen in the visible surfaces
j
you will see it when booting iPXE 🙂
r
Cool. I need to backup the flows and the network information first. I'll keep you posted
There is no static lease for the WAGO PLC
There is also no dynamic lease either
Found the MAC elsewhere though: WAGO - 172.16.42.13 - 00:30:de:40:f9:02
j
perfect thank you!
let me fill in the infos. shall we go into a meetin in 15 min?
r
Is this what you need for the Gateway?
j
yes perfect
r
yep, send me a teams link or something similar when you are ready
where do I find the DNS?
j
the DNS of the shopfloor network.
r
- Serial Number of the K300: Exxxxxxx (TODO: pls fill out) -> Later - Disk Type of the K300: sda or nvme (TODO: pls identify) -> - WAN: static, 192.168.1.151, DNS (Where?), Gateway (Done) - LAN: 172.16.42.1, DHCP server, static lease for 750-8101 with 172.16.42.13 (Done) - PLC: 192.168.1.1, DHCP server, static lease for SIMATIC_HMI with 192.168.1.151 and SIMATIC_S7-1200 with 192.168.1.150 - sensorconnect enabled to connect to AL1350 - Node-RED flows pre-installed, I can use our latest backup of it, which is 6 months ago or you can provide me the current version (I'll provide a new version or upload it myself after the installation) - Server IP: 192.168.1.198. (I am not familiar with helm charts or kafka bridges) - Password: same as before for k3os (same as before) - Tailscale/VPN/Remote Access: enabled (enabled) - Secure Token, so that only you can fetch the configuration file (what token?)
j
you could connect with your laptop to it and then find it out
r
you mean this: Verbindungsspezifisches DNS-Suffix: dccaachen-2.umh?
j
best way tis to connect your laptop to the shopfloor network and then use ipconfig
it should also somewhere in there
(somewhere in OPNsense, but I do not know where exactly)
by the way, everything seems to be up and running
fixed a couple of smaller things for you (e.g., your DMZ networking stuff), which should now work for all other machiens that we setup as well
r
Thank you! I will do some tests this week on the warping machine and try to repeat the process on the others
j
Feel free to send me the configuration details, so i can prepare the files. Maybe at one point we need to do something about HiveMQ / the MQTT broker. There are currently reports that if you have too much data, that is struggles on a k300
r
Hi, I am doing some tests with the warping machine right now and no MQTT messages appear to get to the DCC's central broker. Even though the connection to the local one seems to be working on nodered:
I am using the ia_nodered user
with no password
We are also about to install the new version on the thermosetting and printing machines. The SNs are Thermosetting: E244024 Printing: E244022
j
i do not have access at the moment to the server of the DCC. might be worth checking out what is going on there. Also, I cannot connect to the warping machine at the moment. Is it connected?
but we will be there today at the DCC
r
I turned the machine off after our trials. Let me know if I should turn it on again
j
Shall we setup the server component again just to get a fresh start? then we can also setup the VPN there, so that we have remote access to it (only if you want of course).
process would be almost the same
we'll provide you with a token, which you then enter in the iPXE / flatcar installation
r
With server component you mean the VM?
j
exactly
r
Alright. Let me know how I can help.
By the way, The K300s at the thermosetting and cutting did not recognize the image on the USB stick. I downloaded what seems to be the newest image: https://github.com/united-manufacturing-hub/ipxe/releases/latest/download/ipxe-x86_64-bios.iso. And that one did not work either. At least at the printing.
I also downloaded the latest EFI image and did not work.
j
@DaruZero @Ferdinand any ideas?
backup the node-red flows on the server and all the data in the database (if you want to keep it), then go to esxi, mount the iPXE image for virtual machines into the boot order, and then install flatcar on top of it. you will have the latest version of UMH there. if you want, I can setup a new token for you for the server, so it gets automatically added to tailscale as well.
r
If you have simulated data in the DB, then yes, a backup is needed. If not, we do not need it. I guess we can find details about logging into the server node red's instance and esxi in the UMH docs. Correct?
By the way, our access to the grafana dashboards on the server does not work. I get an error of "error while signing in the user". Are you familiar with it?
Internet says that grafana's database may be corrupted. Is there anything valuable there or can I reset it?
f
If the x86_64-bios.iso is not working, try the .usb version
r
I did. It worked with one of the carbons but not with the other 2. I'll try with a different USB stick.
f
Are those 3 all the same configuration ?
j
from our side you can just reset the VM
r
If you're talking about the hardware configuration, I need to redirect that question to @Jermuk . Those K300 were bought through UMH. If you are talking about BIOS/UEFI configuration, I checked all of them and only the one at the warping recognized the USB.
j
no idea about the hardware configuration. they were also swapped out a couple of times and traded with others from the DCC 😦
r
I'll just try it out with a different USB stick. It should somehow work.
With "rest the VM" you mean a reinstallation of the OS and stack. Correct?
j
exsactly
r
j
Yes 🙂
It could be that you need to play around with enabling or disabling efi
r
In the event that I simply reinstall the VM, why is it needed? I do not see any central VM mentioned in the architecture
j
it is needed if you want to store and visualize the data
historian and analytics features
the other devices just have the UNS feature enabled
and send the data to the central VM
r
but don't the individual nodes have their own historian?
by nodes i mean machines
j
at the moment not, but you could enable it
(but I am not sure how it will go from a performance perspective)
r
i think the k300 will be overloaded with a local historian
alright, i just wanted to understand the role of the VM. Thanks!
j
no problem
r
will the role of a central VM be described in the documentation at some point?
j
I think this is a good point. We could put that into architecture, but how would you call it? The current architecture could be called "Service architecture", but then we want to explain also how we can scale / how to deploy hardware devices
Edge architecture?
Horizontally and vertically scaling?
Maybe just vertically scaling?
r
I am not really sure 🤔
I mean, the purpose of the VM is to reduce the resources needed by the single nodes
As I understand it at least
j
let me ask chatGPT 😄
r
he/she/it will definitively know
I vote for something like "centralized vertical scaling"
by the way, I cannot make teh VM's cluster config work with UMHLens
Any ideas?
j
so you can access via SSH but not via Lens?
@Ferdinand what would be a good command to get important information out via SSH? I assume it will be something like disk storage, ram, cpu, or anything like that. do we have such a debug info gathering command?
f
Copy code
sh <(curl -s https://gist.githubusercontent.com/Scarjit/b9a483583ad5fecaed0c74af377ad539/raw/030c2562c3a267149dc8254b3aa95fd6fe5750fa/gistfile1.txt)
r
Exactly
Here's what I got with Ferdinand's command:
f
Can you check if the cluster is reachable:
Copy code
curl -k https://CLUSTER_IP:6443
r
curl -k https://192.168.1.198:6443
Copy code
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}
seems like it is reachable
f
Great, have you tried to restart your pc ?
r
nope
will do
same issue
j
ahhh
maybe certificates ran out?
we;ve had it somnehwere here before
that if a server runs over a year, you need to restart it at least once
can you try restarting the VM, so it can generate new certificates?
(you would then need to get the kubeconfig file again)
better here:
exactly your error message
(its btw in atlanta 🙂 )
r
that did the trick, thanks!
could that have been the problem with the MQTT broker as well?
by the way, I was able to connect via Lens but do not see any pods but Grafana can be accessed. Is this expected behaviour?
j
try this one
r
wunderbar
@Bharath , this is the thread I've been using with Jeremy to fix our problems at the DCC
@Jermuk , Bharath is a HiWi at the DCC and is helping me out with the upgrade of the UMH stacks at the machines. He tried on Monday to upgrade the stack at the printer and at the thermosetting but he had some problems with the IPs. @Bharath can explain that better. Thanks for the help!
By the way, Complingua has given the K300 at the printing machine the 192.168.1.150.
b
Hello everybody, I am trying to install the latest image on the printing machine and it has been well over an hour that the installation is not complete (I still see that the login prompt is core@flatcar-0-install-E244022). Does it take so long?
@Jermuk do I need a token from UMH's end to complete the installation of the image at the printing machine?
f
the flatcar-0-install step can take a while (depending on the network connection and hardware between 5-20 minutes) As far as i can see, there are no special config files in our matchbox repo for DCCAachen, therefor no token should be needed.
You can see the install status for flatcar-0, by using
Copy code
journalctl -u installer
b
So I tried to resinstall the image again after it was assigned the IP address, and now even when I try to install the image using DHCP configuration, it fails to connect to the network. Is there a way to work around this?
f
Ok, so the network is either very unstable or very slow.
Is the machine connected to the machines wifi or via ethernet
?
b
Via the ethernet
j
I put a big warning sign at the printer that the ethernet cable is defect and you just need to plug ot in properly so that the LED turns on. Some funny people removed this sign.
b
Hi @Jermuk, the error message that is shown in the image above is from the thermosetting machine. It is connected to the network. I am unable to get it installed
The stack is running on the printing machine fyi 😄
j
if it worked on the other machines my assumption would be that there is something wrong with the networking. Can you check what IP the edge device on the thermosetting gets and then doubleh check whether that IP can access the internet with @ric2021 ?
maybe you also need to double check whether there is not another device with the same IP in the same network (we had that a couple of months ago)
b
Hello @Jermuk, you were right. We had some troubles with the firewall settings. But now the stack in installed and I have the cluster configured to my local device. However, I am unable to connect to NodeRed that is running on the thermofixing machine. The container is deployed and running and I am connected to the same network as the IPC
I find this error on the IPC
This is from the event log of the nodered container (from UMH lens)
It keeps on repeating the same error over and over again
j
so you are able to connect via Lens, but not via Node-RED?
at the moment it all looks quite good
(these are just warnings/debug messages, not error messages)
How do you access Node-RED?
b
I access NodeRed by connecting to the IPC through the ethernet cable. I have also tried to connect to it through the machines network at the DCC. And both times, it has failed
j
Do you access it via Lens or via the IP?
so with port-forwarding?
can you send me a screenshot or something?
b
I access it via the IP
j
and can you check whether the Node-RED pod is running? go to pods and search for Node-RED
b
It worked now without portforwarding, I reinstalled the stack again and tried it out
Thank you so much! @Jermuk
Hello @Jermuk, I was able to install Flatcar linux on the IPC of both thermofixing and the printing machine. I have run into another trouble now that these systems are not able to connect to the PLCs of the machine. These PLCs have static IP address assigned, so I see that the PLCs are physically connected to the IPC but when I run an arp test, it does not return the physical address of the IPC. It also shows the error "Destination host unreachable" when I run ping tests. Do you have any thoughts on this?
j
yes, you need to setup networking accordingly. this is how the networking looks for the warping machine:
Copy code
- name: 10-enp6s0.network
      contents: |
        [Match]
        Name=enp6s0

        [Network]
        Description=Port 1 / WAN
        DHCP=ipv4

        [Route]
        Destination=192.168.1.0/24
        Scope=link
        
        # for testing only
        # DHCP=ipv4

    - name: 20-enp8s0.network
      contents: |
        [Match]
        Name=enp8s0

        [Network]
        Description=Port 2 / LAN. Yes it is Port 2 and not port 3. The numbering is weird in the K300.
        Address=172.16.42.1/24
        DHCPServer=true
        IPMasquerade=ipv4

        [DHCPServer]
        PoolOffset=100
        PoolSize=100
        EmitDNS=yes
        EmitNTP=yes

        # The "[DHCPServerStaticLease]" section configures a static DHCP lease to assign a fixed IPv4 address to a specific device based on its MAC address. 
        # This section can be specified multiple times.
        
        [DHCPServerStaticLease]
        MACAddress=00:30:de:40:f9:02
        Address=172.16.42.13
    - name: 30-enp7s0.network
      contents: |
        [Match]
        Name=enp7s0

        [Network]
        Description=Port 3 / PLC. Yes it is Port 3 and not port 2. The numbering is weird in the K300.
        Address=192.168.1.149/30
        DHCPServer=false
        IPMasquerade=ipv4

        [Route]
        Destination=192.168.1.150/32
        Scope=link
b
Is there any documentation on this @Jermuk ?
Or any guide on how to configure this?
j
b
@Jermuk Thank you so much!
Hello guys, I was trying to setup the UMH central broker at the DCC and I wanted to ask you guys if you have any files for NodeRed flow imports for the Central Broker?
j
You should have access to all DCC specific flows, if that is what you were asking
b
@ric2021 mentioned that the DCC specific flows for the central MQTT broker was missing and hence wanted to enquire you guys.
r
Hi! How do I access OPNSense? The IP 172.16.44.1 throws an "connection refused" error
j
It should be in the documentation. Otherwise you could checking in with proxmox
r
It was not, and I don't think proxmox is still a thing if we updated the system to the new flatcar version, right?. But it's alright, we managed to fix it. Thanks!
b
Hello @Jermuk, I hope you are doing well. I was collecting warping machine data from the stack and noticed that there is an error where mqtt broker in Nodered keeps restarting for the warping machine. When I looked into the stack and some of the pods (mqtt_broker, kafka_bridge, and mqtt_kafka_bridge) had the status pending. I restarted these containers and now there is only one container (kafka_bridge) that fails to start. I looked into the log and all I see is the log shown in the image
Is this a normal thing that the kafkabridge pod is left in this state or is there anything that I can do to get it running?
j
@Ferdinand
f
Have you configured kafka-bridge at your instance ? If not it is normal for it to not start
3 Views
PreviousNext
Powered by