2023-06-21 10:28:26 UTC [1426]: [6492d0ca.592-1] [unknown]@[unknown],app=[unknown] [00000] LOG: connection received: host=10.42.0.83 port=41400 2023-06-21 10:28:26 UTC [1425]: [6492d0ca.591-1] [unknown]@[unknown],app=[unknown] [00000] LOG: connection received: host=10.42.0.83 port=41404 2023-06-21 10:28:26 UTC [1426]: [6492d0ca.592-2] grafana@grafana,app=[unknown] [00000] LOG: connection authorized: user=grafana database=grafana SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off) 2023-06-21 10:28:26 UTC [1425]: [6492d0ca.591-2] grafana@grafana,app=[unknown] [00000] LOG: connection authorized: user=grafana database=grafana SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off) 2023-06-21 10:28:26 UTC [1426]: [6492d0ca.592-3] grafana@grafana,app=[unknown] [00000] LOG: disconnection: session time: 0:00:00.007 user=grafana database=grafana host=10.42.0.83 port=41400 2023-06-21 10:28:26 UTC [1425]: [6492d0ca.591-3] grafana@grafana,app=[unknown] [00000] LOG: disconnection: session time: 0:00:00.007 user=grafana database=grafana host=10.42.0.83 port=41404 2023-06-21 10:28:35,842 ERROR: create_config_service failed Traceback (most recent call last): File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 950, in _create_config_service if not self._api.create_namespaced_service(self._namespace, body): File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper return getattr(self._core_v1_api, func)(*args, **kwargs) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper return self._api_client.call_api(method, path, headers, body, **kwargs) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api return self._handle_server_response(response, _preload_content)

File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response raise k8s_client.rest.ApiException(http_resp=response) patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '6f5e35c5-6cd4-4a63-80fd-2db8698b6307', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39505ebc-1855-4cd4-b4ac-6fd35f6229be', 'Date': 'Wed, 21 Jun 2023 10:28:35 GMT', 'Content-Length': '359'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \\"system:serviceaccount:united-manufacturing-hub:united-manufacturing-hub-timescaledb\\" cannot create resource \\"services\\" in API group \\"\\" in the namespace \\"united-manufacturing-hub\\"","reason":"Forbidden","details":{"kind":"services"},"code":403}\n' 2023-06-21 10:28:36 UTC [1435]: [6492d0d4.59b-1] [unknown]@[unknown],app=[unknown] [00000] LOG: connection received: host=10.42.0.83 port=41572 2023-06-21 10:28:36 UTC [1434]: [6492d0d4.59a-1] [unknown]@[unknown],app=[unknown] [00000] LOG: connection received: host=10.42.0.83 port=41574

the logs from timescale

@Ferdinand any idea?

logger=context t=2023-06-21T12:03:49.530314169Z level=warn msg="failed to look up session from cookie" error="dial tcp 10.43.188.6:5432: connect: connection refused" logger=context userId=0 orgId=0 uname= t=2023-06-21T12:03:49.531091768Z level=warn msg= error="dial tcp 10.43.188.6:5432: connect: connection refused" traceID= logger=context t=2023-06-21T12:03:49.531111568Z level=warn msg="failed to look up session from cookie" error="dial tcp 10.43.188.6:5432: connect: connection refused" logger=context userId=0 orgId=0 uname= t=2023-06-21T12:03:49.531235067Z level=info msg="Request Completed" method=GET path=/api/datasources/proxy/uid/P14116848B5300287/api/v2/factoryinsight/autoform/DefaultArea/DefaultProductionLine/G18/tags/custom/productionRate_actual status=403 remote_addr=10.42.0.1 time_ms=1 duration=1.511496ms size=15 referer="http://192.168.20.27:8080/d/8p4J1WnVe/g18?from=now-3h&orgId=1&refresh=10s&to=now" logger=context userId=0 orgId=0 uname= t=2023-06-21T12:03:49.543097439Z level=warn msg=Unauthorized error="dial tcp 10.43.188.6:5432: connect: connection refused" remote_addr=10.42.0.1 traceID= logger=context userId=0 orgId=0 uname= t=2023-06-21T12:03:49.543417938Z level=info msg="Request Completed" method=GET path=/api/datasources/proxy/uid/P14116848B5300287/api/v2/factoryinsight/autoform/DefaultArea/DefaultProductionLine/G18/tags/custom/productionRate_plannedAverage status=401 remote_addr=10.42.0.1 time_ms=12 duration=12.96407ms size=40 referer="http://192.168.20.27:8080/d/8p4J1WnVe/g18?from=now-3h&orgId=1&refresh=10s&to=now" handler=/api/datasources/proxy/uid/:uid/* logger=context userId=1 orgId=1 uname=admin t=2023-06-21T12:03:49.763282319Z level=error msg="Failed to update last_seen_at" error="dial tcp 10.43.188.6:5432: connect: connection refused" logger=context t=2023-06-21T12:03:49.794909044Z level=warn msg="failed to look up session from cookie" error="dial tcp 10.43.188.6:5432: connect: connection refused"

i get the error from both the umh-datasource and direct from postgres so i does not look as a data-source error..

i found some thread in a forum about rbac and patroni in timescale which seemed to be fixed in the newer versions.. it was something with the service roles.

could i be the load balancer for timescale?

so this was interesting.. i deleted the loadbalancer service for timescale and updatated the helmchart to the same version to recreate the loadbalancer and now it seems to work again.

still have some errors in timescale with create_config_service failed and also some in grafana with Unauthorized error="dial tcp 10.43.188.6:5432: connect: connection refused" remote_addr=10.42.0.1 traceID= and 0 uname= t=2023-06-21T12:31:04.13019385Z level=warn msg= error="dial tcp: lookup united-manufacturing-hub on 10.43.0.10:53: server misbehaving" traceID=

but at least the data displays correct now.