https://www.umh.app logo
Join Discord
Powered by
hmm. some interesting problems today. 2024-05-29 1...
# umh-support
d
hmm. some interesting problems today. 2024-05-29 12:36:36 UTC [4990]: [6657214d.137e-1] @,app= [53100] ERROR: could not resize shared memory segment "/PostgreSQL.2112515244" to 8388608 bytes: No space left on device
f
Assuming that you already checked for disk space (
df -h
), can you check what the size of the timescale storage vol is on disk (
Copy code
du -sh /var/lib/rancher/k3s/storage/pvc-<random_stuff>_united-manufacturing-hub_storage-vol
ume-united-manufacturing-hub-timescaledb-0
) ?
Base limit for timescale is set to 2Gi you might need to increase that. You can also try increasing the shared memory limit direcly by mapping an tmpfs to /dev/shm in timescale. Something like this, but adjusted for your timescaledb deployment:
Copy code
apiVersion: v1
kind: Pod
metadata:
  name: timescale-pod
spec:
  containers:
  - name: timescale
    image: timescale/timescaledb:latest
    volumeMounts:
    - mountPath: /dev/shm
      name: dshm
    resources:
      requests:
        memory: "2Gi"
      limits:
        memory: "4Gi"
  volumes:
  - name: dshm
    emptyDir:
      medium: Memory
^ this will increase your memory usage
d
i have not checked diskspace yet. i just found the error when i was doing i query in grafana.
on /dev/sda1 i have 27G availible
let's see if i find right. the timescalde disk is called storage-volume-united-manufacturing-hub-timescaledb-0
disk usage of the folder storage is 25G
f
d
yes. my plan was to look at stuff like that after i upgrade the entire system.. so the quick solution now would be to change the disk size of the pod ? volumeClaimTemplates: - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: storage-volume creationTimestamp: null labels: app: united-manufacturing-hub-timescaledb cluster-name: united-manufacturing-hub heritage: Helm purpose: data-directory release: united-manufacturing-hub spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi volumeMode: Filesystem status: phase: Pending - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: wal-volume creationTimestamp: null labels: app: united-manufacturing-hub-timescaledb cluster-name: united-manufacturing-hub heritage: Helm purpose: wal-directory release: united-manufacturing-hub spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi volumeMode: Filesystem status: phase: Pending so i change the storage volume in the stateful from 2Gi to 35Gi perhaps ?
f
If you have enough storage on the base system
d
i have 27G free on the VM disk but i can ask IT to give me more.
tried to update both disks to 35G but in the stateful but got error: Failed to save resource: {"error":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"StatefulSet.apps \"united-manufacturing-hub-timescaledb\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","reason":"Invalid","details":{"name":"united-manufacturing-hub-timescaledb","group":"apps","kind":"StatefulSet","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","field":"spec"}]},"code":422},"messages":["StatefulSet.apps \"united-manufacturing-hub-timescaledb\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"],"isUsedForNotification":false}
f
1 sec
Ok, iv just checked, local-path does not care about the requested storage size (I just wrote a couple GB into my timescaledb folder from inside kubernetes). So the issue is with shared memory itself. You can edit the deployment of timescaledb and add the 2 entries. By default this will give it unlimited memory (limited only by physical memory) for it's shared memory operations. Otherwise you can also set a limit:
Copy code
- name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 1Gi
You would have to experiment with the amount of mem needed a bit, as i haven't have seen this issue yet
d
hmm. still get the same error: 2024-05-29 13:37:49 UTC [119]: [66572fa7.77-3] factoryinsight@factoryinsight,app=[unknown] [53100] ERROR: could not resize shared memory segment "/PostgreSQL.664900187" to 8388608 bytes: No space left on device i have set a limit to 2Gb for the shared memory.
now. sorry, wait. i forgot to add the path. i will test.
Nice. Now it works, thank you.. I am making some large aggregates of the data so i guess that caused the problem..
i also run in to some other questions. For the timescale data would it be possible to combine more data in to rows? this would save space and when i am analyzing data now i want to look at value1 but only if value2 = 1.. to do this now i first need to combine data with the same timestamp, but in my case i save much of the data with the same timestamp just to have that functionallity..
f
You could save more complex data to a tag_string (json), but that has it's own problems. If you just need to save space checkout https://docs.timescale.com/use-timescale/latest/compression/
d
yes. to save space i have looked at the compression, but when looking at live data i think it will be a problem.. then the compression needs to happen imidiettly to work.
7 Views
PreviousNext
Powered by