So, i got some new error again when IT rebooted th...
# umh-supportd
DanielH
11/03/2022, 7:46 PMSo, i got some new error again when IT rebooted the servers this evening. now Redis won't start correct so our loraWAN network server can not start correct due to the problem with redis so now i do not get any data from our machines.
j
Jermuk
11/03/2022, 7:52 PMMoving our private messages to the general discord here.
Jermuk
11/03/2022, 7:53 PMMy suggestion: Try scaling down redis (set statefulset replica to 0) and then scale it up again.
Jermuk
11/03/2022, 7:53 PM@Ferdinand for tomorrow if it is still open until then
Jermuk
11/03/2022, 8:00 PMcan you create a couple of screenshots how it is looking in lens? (or a video)
Jermuk
11/03/2022, 8:00 PMjust wanting to rule out diskpressure and stuff like that
Jermuk
11/03/2022, 8:02 PMyour error message (you send me via a PM) means that the error code was 137. this seems to be out of memory error (not enough RAM)
Jermuk
11/03/2022, 8:04 PMdoes it also say in Kubernetes / Lens something like OOM?
d
DanielH
11/03/2022, 8:05 PMDanielH
11/03/2022, 8:06 PMj
Jermuk
11/03/2022, 8:08 PMd
DanielH
11/03/2022, 8:09 PMDanielH
11/03/2022, 8:09 PMj
Jermuk
11/03/2022, 8:12 PMone thing you could check is what is set as memory in the helm chart. maybe there was a similar issue as with the ACL. let me find you the relevent entries
Jermuk
11/03/2022, 8:12 PMotherwise I'd suggest sitting together with @Ferdinand tomorrow
Jermuk
11/03/2022, 8:13 PM(the screenshots were btw good and ruled out that there is not enough physical memory)
Jermuk
11/03/2022, 8:14 PMredis.master.extraflags
Jermuk
11/03/2022, 8:14 PMredis.master.resources
Jermuk
11/03/2022, 8:14 PMredis.sentinel.resources
Jermuk
11/03/2022, 8:15 PMredis.replica.resources
Jermuk
11/03/2022, 8:16 PMyour system should have been originally on like 2 or 4GB. the new development.yaml /values.yaml uses 200Mb. The upgrade should have never touched the values.yaml, but maybe something similar happened with the ACL?
d
DanielH
11/03/2022, 8:18 PMDanielH
11/03/2022, 8:18 PMshould i increase it ?
j
Jermuk
11/03/2022, 8:19 PMok, so everything stayed the same. does that 4GB match with the other values like redis.master.resources ?
d
DanielH
11/03/2022, 8:21 PMj
Jermuk
11/03/2022, 8:21 PMif yes, I'd suggest sitting together with Ferdinand in a remote call tomorrow. i cant explain what is happening from remote
Jermuk
11/03/2022, 8:21 PMincreasing would be one option
Jermuk
11/03/2022, 8:22 PMlooks all good
d
DanielH
11/03/2022, 8:26 PMlook.
DanielH
11/03/2022, 8:27 PMj
Jermuk
11/03/2022, 8:27 PMwhat
Jermuk
11/03/2022, 8:27 PMjust waited and it fixed itself? 😄
Jermuk
11/03/2022, 8:28 PMkubernetes doing kubernetes stuff. sometimes, it fixes the stuff itself
d
DanielH
11/03/2022, 8:29 PMi increased memory to 6gb and request for 2 then terminated the pods.
DanielH
11/03/2022, 8:29 PMDanielH
11/03/2022, 8:29 PMnow i got a data point
j
Jermuk
11/03/2022, 8:29 PMinteresting! maybe chripstack just needs much more memory of redis than UMH
d
DanielH
11/03/2022, 8:29 PMDanielH
11/03/2022, 8:31 PMit could be that chirpstack need more memory but i do not undestand why since i only using it to bridge the loraWAN messages to mqtt.. but i do however decode them but we are only sending some digital ins so it is not much data.
DanielH
11/03/2022, 8:32 PMWell.. However, you are an Angel and saved my day once again.
DanielH
11/03/2022, 8:36 PMA litte strange though. I increased the memory, deployed the helm chart, terminated the redis nodes but when i look in the new nodes it still says that it has requested 1Gb and have a max of 4Gb
3 Views