eb 14 20:06:30 flatcar-1-umh k3s[720]: time="2023-02-14T20:06:30.270798502Z" level=info msg="Cluster-Http-Server 2023/02/14 20:06:30 http: TLS handshake error from 192.168.3.175:63853: remote error: tls: bad certificate" Feb 14 20:06:31 flatcar-1-umh k3s[720]: time="2023-02-14T20:06:31.303619711Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [flatcar-1-umh-f3d95ce2=https://192.168.3.236:2380], expect: flatcar-1-umh-f3d95ce2=192.168.3.22" Feb 14 20:06:32 flatcar-1-umh k3s[720]: time="2023-02-14T20:06:32.540043157Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1676400624: notBefore=2023-02-14 18:50:24 +0000 UTC notAfter=2024-02-14 20:06:32 +0000 UTC" Feb 14 20:06:32 flatcar-1-umh k3s[720]: time="2023-02-14T20:06:32.547273042Z" level=info msg="Cluster-Http-Server 2023/02/14 20:06:32 http: TLS handshake error from 192.168.3.175:63858: remote error: tls: bad certificate"

apparently it was installed with 192.168.3.236 and now has .3.22

i could check tomorrow if there is a way to fix it afterwards

when i change back to dhcp it works again.

With this you can change the IP of the certificate during the installation.

Maybe this also works afterwards?

So your goal is to change the IP address that you got via DHCP during installation to a different static ip correct?

Better

Check out this

There is even a tls-san example

Ah ok, it is only executed during install

These ones

It will pass through the arguments

i will try to rotate the certificates again

no.. it will not rotate since the node is not in the etcd

replicated

So, to fix it, try this: 1) SSH into the server 2) Edit the /etc/systemd/system/k3s.service add --tls-san=XXX.YYY.ZZZ.AAA to the ExecStart options (see image) 3) Delete the old secret

sudo kubectl -n kube-system delete secrets/k3s-serving

4) Remove the cert from disk

mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json

5) Restart k3s

systemctl restart k3s

Tested on flatcar k3s version v1.21.5+k3s2 (724ef700) go version go1.16.8

@Jermuk ill write a notion article for it

I also added the "development" option inside the article, to just ignore certificate errors

because if i change the ip and restart the network to apply the changes i can not access the cluster with kubectl to remove the secret, so i guess this operation needs to be performed before changing the ip

All of these commands where executed on the server

the cluster could not start correct since the tls handshake failed.

i also tried to delete the node before changing the ip but still same result

i still get the same problem.

i have however fewer errors now.

i followed the guide in flatcar docs to set static ip https://www.flatcar.org/docs/latest/setup/customization/network-config-with-networkd/

now i can not access my cluster

Failed to test data store connection: this server is a not a member of the etcd cluster. Found [flatcar-1-umh-d175f0b5=https://192.168.3.238:2380], expect: flatcar-1-umh-d175f0b5=192.168.3.22"

replicated your error

well, i managed to "fix" the error, but in doing so wiped my UMH cluster

:/

i tried to delete the cluster and rotate the certificates and then start up the cluster again but it did not help.. i have spent alot of time to trying to fix this problem..

nothing new here?

if i understand it should be able to reset the cluster but due to a bug in k3s it fails to remove the old cluster but this seems to be fixed with version 1.23 https://github.com/k3s-io/k3s/issues/4870

i am currently pulling an update.. i also notice that the upgrade-controller was not included with the flatcar installation..

YES!. finally..

So now i have a very easy solution to this problem..

1. first your update k3s to minimum v. 1.23. i used version: v1.26.1-k3s1 .. follow this guide https://docs.k3s.io/upgrades/automated

2. set your static ip using this guide https://www.flatcar.org/docs/latest/setup/customization/network-config-with-networkd/

3. once your ip is changed, ssh back to the server using the new ip

4. stop k3os with: sudo systemctl stop k3s

5. reset the cluster etcd with: sudo /opt/bin/k3s server --cluster-reset

6. start k3s again: sudo systemctl starts k3s

7. check status of server: sudo systemctl status k3s

8. change the ip for the cluster in the lens config..

BOM! you have successfully changed your server to static IP.

i have not double checked that nothing gets affected since my installation is fresh, but i think you guys can manage that part.. i guess it should be quite easy to write a small script that sets the static ip and reset the cluster.

i wrote a small script that handles the ip change.. just change chmod to +xu and run it in sudo.. i have change ip 2 times now and it seems to work but there are now guaranties. Also maybe add some wait time after changing ip before running the rest and some feedback from the cluster reset before starting the cluster again.

it looks like it leaves some backups when doing this so i guess one should add some cleaning in /var/lib/rancher/k3s/ i also saw something about taking a backup first and then reset the cluster using the backup..

Yes, ill will check that.

We make it easier to find in the new documentation: https://knowledge-3.ghost.io/course/how-to-fix-error-while-proxying-request-x509-certificate-is-valid-for-error/ (sneak preview only 🙂 )