[X] My Assistant

My H@H VPS server has 393.6 GB (412,716,208 KB) of total disk space, not including the Linux distro itself, as measured by "df". As such, I initially set my "Maximum Disk Cache Size" to 394 GB, and received 2017 static ranges (the server is relatively new, so it followed the new 200 MB / static range rule directly, and never received more than 2017 static ranges). I was under the impression that H@H servers don't typically fill up every single static range in the local cache, so slightly going over my total disk space wouldn't cause problems. Well, I couldn't have been more mistaken. The server ran perfectly fine for a couple of months, but a few days ago it shut down for the first time due to low disk space. I guess I'll just reduce my Max Disk Cache Size slightly then, thought I, and I started cutting the value in small decrements. But the server would always run for 6 or 8 hours, and stop again. I have done this cache size reduction for well over 10 times now, most recently reducing it to 389 GB (that's 1996 statics ranges), but it still eventually shutdown. So now I've cut it further down to 388.5 GB (1994 static ranges), but I doubt things will get any better this time.

I basically only use this server for H@H, but in order to be sure that my disk space isn't being consumed by something else, I decided to take some more precise measurements this time. As I said, I have a total disk space of 412,716,208 KB (393.6 GB), and I also took note of the "used" disk space given by "df" both before and after h@h restarted, cleaned up the two removed static ranges, and started normal operation. Before restart, I had about 90 MB of space left, and right after normal operation started, I had 516 MB of free space. That sounds just about right. Next, I went to the cache folder, and ran "du -s". This gave me a total cache size (after H@H restart and cleanup) of 411,143,604 KB (392.10 GB). This is problematic on two different levels: first, it appears that H@H is not respecting my Max Disk Cache Size setting at all (I just reduced that value from 389 GB to 388.5 GB, remember); second, this means that each static range is taking up 201.36 MB on my disk on average, and they'll keep growing until my disk space runs out, without, for example, cleaning up the oldest accessed files to free up space.

For the second point, I guess the 200 MB / static range rule isn't a strict upper limit, so my complaint above can be more or less considered a feature suggestion; but the first point makes me very confused. What am I supposed to do if H@H doesn't respect the Max Disk Cache Size that I set? Is what I'm seeing the expected behavior? I can't stay up all day and keep refreshing my H@H page in order to catch every shutdown immediately, so this has been wasting a lot of my server time. Any help is appreciated, thanks!

(BTW, I'm still running H@H 1.5.4 on the server, because I thought 1.6.0 was a minor update compared to 1.5.4, so haven't updated yet. Would that make any difference?)

Static ranges can indeed go over 200 MB, but the total size of files in the cache should never go above what's set in the config. Keep in mind that this is "apparent size", not size on disk, so you should set the cache size with that in mind rather than trying to set it to 100% of the available disk space.

H@H does not have a way to determine your type of filesystem or its cluster size, so the used cache size calculated does not include filesystem overhead. Usually filesystems use 4 kB cluster size which means the overhead is fairly negligible (avg 2 GB per million files), but if the filesystem is misconfigured with 64 kB clusters the overhead is suddenly significant (avg 32 GB per million files).

(Though I should probably just make it assume 4 kB cluster size for the cache size calculations instead of glossing over it entirely.)


The other possible cause is that the resource tracking has gotten desynced somehow, say if old copies were manually copied over or something screwed up.

The file pruning mechanism has quite a lot of debug output, so if you look in the log_out file, it should be fairly obvious if this is the case. Look for these kinds of entries:

CODE

[debug] CacheHandler: Checked cache space (cacheSize=1610532567384, cacheLimit=1610612736000, cacheFree=80168616)

If "cacheSize" is significantly different from what is reported by du --apparent-size -b -s /path/to/cache, it is desynced; shut down the client, delete the three "pcache" files in the data directory, and start it back up. It will then rescan the cache.

Static ranges can indeed go over 200 MB, but the total size of files in the cache should never go above what's set in the config. Keep in mind that this is "apparent size", not size on disk, so you should set the cache size with that in mind rather than trying to set it to 100% of the available disk space.

H@H does not have a way to determine your type of filesystem or its cluster size, so the used cache size calculated does not include filesystem overhead. Usually filesystems use 4 kB cluster size which means the overhead is fairly negligible (avg 2 GB per million files), but if the filesystem is misconfigured with 64 kB clusters the overhead is suddenly significant (avg 32 GB per million files).

(Though I should probably just make it assume 4 kB cluster size for the cache size calculations instead of glossing over it entirely.)
The other possible cause is that the resource tracking has gotten desynced somehow, say if old copies were manually copied over or something screwed up.

The file pruning mechanism has quite a lot of debug output, so if you look in the log_out file, it should be fairly obvious if this is the case. Look for these kinds of entries:

CODE

[debug] CacheHandler: Checked cache space (cacheSize=1610532567384, cacheLimit=1610612736000, cacheFree=80168616)

If "cacheSize" is significantly different from what is reported by du --apparent-size -b -s /path/to/cache, it is desynced; shut down the client, delete the three "pcache" files in the data directory, and start it back up. It will then rescan the cache.

Ahh, of course I haven't thought of apparent size vs. size on disk... "du -s /path/to/cache" gives 411,150,580 KB (392.10 GB), while "du --apparent-size -b -s /path/to/cache" gives 418,246,369,330 B (389.52 GB). While this is still 1 GB above the limit I set (388.5 GB), now I see why my previous attempts at reducing the limit still ended in shutdowns: I simply didn't have enough disk space.

In fact, H@H has now been running for about eight hours, and the ~500 MB free space as measured by "df" hasn't changed much, in stark contrast to previous runs. Perhaps I've finally hit the sweet spot (that is, if I ignore the 1 GB discrepancy...); I'll continue to monitor the server to see whether the ~500 MB free space persists (it has started going down by a few MB's while I'm writing this...).

As for checking for desyncing, I disabled logging by default, and trying to turn it back on via the web page seemed to have done nothing, so I'll restart H@H and report my findings back later. Thank you so much for helping me solve this issue!

Alright, so the most recent cacheSize in log_out is 418,063,252,562 B (389.35 GB), not too far off from the current output from "du --apparent-size -b -s /path/to/cache", 418,207,651,785 B (389.49 GB). Hopefully this means there isn't any desyncing issue.

BTW, for some reason, shutting down H@H appears to have freed up nearly 50 MB of additional space. Hopefully I won't need to reduce the Max Disk Cache Size again.

This post has been edited by cklodar: Jul 3 2020, 01:41

I wonder if I will have something similar eventually, I currently should have 511 GB free on disk, and H@H indicate around 508 GB free for cache, hopefully I have the right margin.
Well I won't know for sure until next year or later, when I do fill up the space (IMG:[invalid] style_emoticons/default/rolleyes.gif)

Ahh, of course I haven't thought of apparent size vs. size on disk... "du -s /path/to/cache" gives 411,150,580 KB (392.10 GB), while "du --apparent-size -b -s /path/to/cache" gives 418,246,369,330 B (389.52 GB). While this is still 1 GB above the limit I set (388.5 GB), now I see why my previous attempts at reducing the limit still ended in shutdowns: I simply didn't have enough disk space.

...
Alright, so the most recent cacheSize in log_out is 418,063,252,562 B (389.35 GB), not too far off from the current output from "du --apparent-size -b -s /path/to/cache", 418,207,651,785 B (389.49 GB). Hopefully this means there isn't any desyncing issue.

Thanks for confirming. Adding some simple calculations for the overhead is easy enough, so I'll include it with the 1.6.1 release.

Thanks for confirming. Adding some simple calculations for the overhead is easy enough, so I'll include it with the 1.6.1 release.

That would be a great feature to add! Although as you said, H@H doesn't have the ability to determine cluster size, so the calculations will be speculative, based on assuming 4 KB clusters?

Also, final update (hopefully) on the status of my server: it's been running for two days without shutdown now. Available space has been hovering above 500 MB. So I think my case can be considered closed now. Thank you again for helping me diagnose it!

That would be a great feature to add! Although as you said, H@H doesn't have the ability to determine cluster size, so the calculations will be speculative, based on assuming 4 KB clusters?

It will assume 4 kB clusters but it will be adjustable with a command-line argument. I did find a way to determine it within Java automatically, but it requires Java 9, so that won't happen, at least not while a LTS of OpenJDK 8 is being maintained.

Also, final update (hopefully) on the status of my server: it's been running for two days without shutdown now. Available space has been hovering above 500 MB. So I think my case can be considered closed now. Thank you again for helping me diagnose it!

Cool, thanks for the update.