NOTES: TODO 🔧 Checking if mgmtcompanion namespace exists... Error from server (NotFound): namespaces "mgmtcompanion" not found ✅ Installed UMH Helm Chart 🎉 UMH installed successfully! 🔧 Installing Management Companion... 🔍 Validating authentication token format... ✅ Authentication token format is valid. 🔍 Checking for existing MgmtCompanion installation... Error from server (NotFound): namespaces "mgmtcompanion" not found ✅ MgmtCompanion is not installed. 🔧 Installing MgmtCompanion... namespace/mgmtcompanion created ✅ Namespace mgmtcompanion created successfully. 🔧 Downloading MgmtCompanion manifests... /tmp/install.sh: line 1464: /usr/bin/curl --connect-timeout 10 --retry 5 --retry-delay 5 --retry-max-time 60 --retry-connrefused --retry-all-errors -sSL: No such file or directory ⚠️ Failed to download https://management.umh.app/static/kubernetes/configmap.yaml. Retrying with resume support... /tmp/install.sh: line 1470: /usr/bin/curl --connect-timeout 10 --retry 5 --retry-delay 5 --retry-max-time 60 --retry-connrefused --retry-all-errors -sSL -C - : No such file or directory ⚠️ Failed to download https://management.umh.app/static/kubernetes/configmap.yaml. Attempt 1 2 3 4 5 of 5. Retrying in 5 seconds... ❌ Error: Failed to download https://management.umh.app/static/kubernetes/configmap.yaml.

hmm. added some more dns server in the network config, ran the script again. now it comes a bit longer but error: W0203 10:26:24.954959 1 reflector.go:561] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.Namespace: Get "https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": net/http: invalid header field value for "User-Agent" E0203 10:26:24.955005 1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get \"https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0\": net/http: invalid header field value for \"User-Agent\"" logger="UnhandledError" and i cannot login to the cluster due to: E0203 11:27:15.643274 31456 proxy_server.go:147] Error while proxying request: x509: certificate signed by unknown authority x509: certificate signed by unknown authority

It also re-installed k3s, so the cert changed

for the mgmt i cet pvc error: Warning FailedScheduling 15m default-scheduler 0/1 nodes are available: persistentvolumeclaim "mgmtcompanion-pvc" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

yes. now it works with the 0.614 version

i also have error from the config: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 30m default-scheduler Successfully assigned united-manufacturing-hub/united-manufacturing-hub-kafka-configuration-g6zqh to server Warning Failed 30m kubelet Failed to pull image "management.umh.app/oci/redpandadata/redpanda-operator:v2.2.4-24.2.5": pull QPS exceeded Warning Failed 30m kubelet Error: ErrImagePull Normal BackOff 30m kubelet Back-off pulling image "management.umh.app/oci/redpandadata/redpanda-operator:v2.2.4-24.2.5" Warning Failed 30m kubelet Error: ImagePullBackOff Normal Pulling 30m (x2 over 30m) kubelet Pulling image "management.umh.app/oci/redpandadata/redpanda-operator:v2.2.4-24.2.5"

hmm. it is running, but with errors. in the container logs i have: {"log.level":"info","@timestamp":"2025-02-03T12:17:26.136Z","log.origin":{"function":"github.com/IBM/sarama.(*debugLogger).Printf","file.name":"[email protected]/sarama.go","file.line":129},"message":"client/metadata fetching metadata for all topics from broker united-manufacturing-hub-kafka-0.united-manufacturing-hub-kafka.united-manufacturing-hub.svc.cluster.local.:9092","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2025-02-03T12:17:26.136Z","log.origin":{"function":"github.com/united-manufacturing-hub/Sarama-Kafka-Wrapper-2/pkg/kafka/consumer/redpanda.NewConsumer","file.name":"redpanda/client.go","file.line":135},"message":"No topics found. Waiting for 1 second","ecs.version":"1.6.0"}

it could perhaps be the startup probe that fails: Warning Unhealthy 89s (x666 over 115m) kubelet Startup probe failed: Get "http://10.42.0.69:8086/live": dial tcp 10.42.0.69:8086: connect: connection refused

WARN 2025-02-03 10:22:28,573 [shard 0:admi] pandaproxy - service.cc:468 - mitigate_error: kafka::client::partition_error ({_schemas/0}, { error_code: unknown_topic_or_partition [3] })

hmm. i also ran into credential problems when restoring the /data folder for the timescale pvc. now mgmt console cannot access postgres and grafana will not start due to "Error: ✗ failed to create admin user: pq: could not access status of transaction 111830"