https://www.umh.app logo
Join Discord
Powered by
Disk is filling up quickly
# umh-support
d
hmm. i ran into a strange problem. in the old stack, i had disk pressure last week and also my certificates had to be renewed. I increased the VM disk and the root partition and later on rebooted the system for the certificate renewal. However, since then my system keeps growing in size rapidly. about 2Gb per day. If i look in pgadmin the database went from 62.03 to 62.08 since yesterday but the root partition on the server has increased from 125Gb to 127Gb.. I cannot remember changing anything else.. with this rate i will fill the disk soon.. Ideas anyone ? @Ferdinand
j
@Athavan kanapuli
d
looks like the pvc for kafka is 24Gb and prometheus is 27Gb. timescale is 63 so that one seems fine.
also the /var/lib/rancher/k3s/agent is quite large with 9.2Gb
last week when i got disk pressure the root partition was somewhere around 116Gb so it has increased around 10Gb in a week.
looking at the deployment for prometheus it is started with '--storage.tsdb.retention.size=5GiB' so it should not take 27Gb.
a restart of prometheus cleared 27Gb. I will see if it starts to fill upp again.. kafka is still 24Gb which seems alot.
but the total is still around 10 Gb. could it be the scaling that causes this. i see i have alot of restarts 124, and 128 on 2 of the pods.
no. sorry. the restarts is just for the kafkatopostgres. not the kafka itself.
a
Hi @DanielH Thanks for sending a detailed analysis. Let me take a look at this and investigate the potential problem
d
for prometheus it looks like it was a hickup. as of now it seems fine, but for redpanda it is more strange since it looks like it has old chunks left on disk which does not show up in the topics.
a
Quick question about this image. Is this an active topic where still messages are produced continuously ? Looks like a partition skew problem which is a different problem to investigate.
d
yes. I am still using the old system with the raw topics, so all data go first to ia.raw and data i save on db goes to ia.factoryinsight
i am doing a migration to the new stack but i am not finished yet so this should not be a problem soon, but for now it would be good to free some space and find the underlying problem.
a
Thanks , I just started investigating this problem. Will keep you posted
🙂
Can you check the segment.bytes and segment.ms config for this topic. If its same as the image shown above for the ia.raw topic, then the size of these log segments for ia.test.test.test topic makes sense. These log segments will be closed only when it exceeds the configured size in segment.bytes or after the segment.ms duration is crossed.
Added to this , the segment size can be larger than the messages in the topic itself (2.47 kiB in our case) as the segments stores a lot more metadata, headers and index information
d
i deleted it since it is an old topic. it showed up again so i guess i have some test node still running.. the settings are: https://cdn.discordapp.com/attachments/1342027854141067284/1342125238334259241/image.png?ex=67b87f08&is=67b72d88&hm=c6e704fd6274da57035f2a69af07c21ccaf9294de9d66f6456e2403716a54c36&
at least it looks quite stable for now.
j
2 Views
PreviousNext
Powered by