i run into a major problem. I needed to change the ip of the server.. Now i cannot edit the instance from the console to the new ip. How do i solve this ?

Maybe @Ferdinand can help here, as I remember changing the IP is what k3s does not really like. But I remember hostname was more of an issue

i tried to do the fix i made a long time ago with cluster-reset which fixed the kubernetes cluster at first. i can now access the cluster with kubectl but it is the wrong node so i cannot see the namespaces.

ah, the dual node issue? :/

you can try: - check which nodes you have (kubectl get nodes) - shut down k3s - edit the k3s.service (/etc/systemd/system/k3s.service) For ExecStart specify the tls ip's you want and the node-name (the name of your OLD node), like this:

ExecStart=/usr/local/bin/k3s \
    server \
    '--tls-san' \
    '116.203.96.176' \
    '--tls-san' \
    '2a01:4f8:1c1b:431b::1' \
    '--tls-san' \
    '100.96.246.1' \
    '--tls-san' \
    'fd7a:115c:a1e0::3301:f601' \
    '--kubelet-arg=config=/opt/kubelet-config.yaml' \
    '--node-name' \
    'ferdinand-hetzner-rocky' \

Afterwards try starting k3s again. Im not sure if it already re-genererates the TLS certificate without either manually wiping the tls folder or a cluster-reset

core@autoform /usr $ sudo kubectl get nodes $c NAME STATUS ROLES AGE VERSION autoform Ready control-plane,etcd,master 24m v1.29.5+k3s1

do you know what the original name was ?

originally flatcar-something , but i am not sure, it could be correct.. i do not remember if i got a better node-name for the new instance.. i checked the hostname before changing the ip and that was autoform

In your k3s.server does it have the --node-name entry ?

systemctl status k3s ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: disabled) Active: active (running) since Wed 2025-01-29 15:37:57 UTC; 18min ago Docs: https://k3s.io Process: 3792630 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS) Process: 3792632 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 3792633 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 3792634 (k3s-server) Tasks: 96 Memory: 1.0G CPU: 1min 4.543s CGroup: /system.slice/k3s.service ├─3788805 /var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/containerd-shim-runc-v2 -namespace k8s.io -id d1c3250c84ef45f35074a3b9b599ffacd7fbb1de700c965908eb3e34aaa759a0 -address /run/k3s/containerd/contain> ├─3788807 /var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin/containerd-shim-runc-v2 -name....... Jan 29 15:38:29 autoform k3s[3792634]: I0129 15:38:29.351080 3792634 replica_set.go:676] "Finished syncing" kind="ReplicaSet" key="kube-system/metrics-server-54fd9b65b" duration="5.040113ms" Jan 29 15:38:29 autoform k3s[3792634]: I0129 15:38:29.351532 3792634 replica_set.go:676] "Finished syncing" kind="ReplicaSet" key="kube-system/metrics-server-54fd9b65b" duration="35.316µs" Jan 29 15:38:29 autoform k3s[3792634]: I0129 15:38:29.357549 3792634 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager

If you do: cat /etc/systemd/system/k3s.service

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null' ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/opt/bin/k3s \ server \ '--tls-san' \ '192.168.20.18' \ '--kubelet-arg=config=/opt/kubelet-config.yaml' \

Ok, can you just adjust the ip there and restart k3s ? Also to prevent issues when/if the hostname ever changes you can add the --node-name param

still the same.. all namespaces are gone.

hm, the etcd might still use the old ip

it should not after the --cluster-reset but it could be that it is still something left.

Doesnt --cluster-reset wipe all state ?

it should not wipe all state. if i remember correct it should only wipe the etcd.. i have done it a couple of time before without any problem.. could i not use a snapshot to restore ?

If you have one

i also found the node-name in the journal from earlier today, it is flatcar-1-umh-5593-8816-8871-8086-7725-2286-94 so maybe i should change to that and restart k3s?

You can try

hmm. it still uses the new node..

Have you set it in the k3s.service or just when executing a manual k3s command ?

in the k3s.service

Maybe it only takes node name changes on new clusters

hmm. this is quite strange..

hm, they should be generated twice daily

the pv's are backed up.. it is not such a big deal since much stuff is not implemented yet. however there is of course some work put into it..

What you could try is to change the names in the etcd and then start k3s again

where ? i tried to do a restore now from the state.db which gave me most of the pods back but not the namespaces.

Not sure where exactly, but can you just search the etcd for incorrect entries ?

hmm. not really shure if this is fixable.. the cluster-reset should work fine when you have snapshots, but for some reason it looks like i don't have any snapshot.. So i guess it is totally broken.. It is however not a big deal since i have not set up so much and i have backups of the pvcs.. Only question is how i reinstall it and restore the pvcs the best way.

Have you activated the User Management feature flag ?

I don't think so. Where do i do that?

It should be disabled (for now).

I will check after lunch. Currently in a collegues office

I dont think there is a method in the install script to use existing data for a new cluster

ok. so what would be the easiet way from this point ? a complete purge of the k3s and a reinstall and maybe then copy the data from postgres into the new pvc? Most stuff i can just redeploy. Onlye thing that would be good to reuse is the postgresdata since i don't want to move it once again.

1. Backup the persistent volumes (especially postgres) 2. Wipe the k3s cluster (

/usr/local/bin/k3s-uninstall.sh

) 3. Create a new install 4. Shut down the new install and restore the postgres (and possible other) backup

so simply just copy/paste the content of the pvcs ?

by default we dont use certificates with postgres. As for passwords those are saved in the pv, so it should work after a restore

ok. then it should be fine. i also just discovered that IT had completly shut down the entire vlan which i moved the server to. no access to the outside.. i guess that was the main problem so i maybe did a fu*kup from start. 🙈 😂