1.3.0 - 2016-10-01Feature Additions- The H@H Downloader has made a surprising yet glorious return. This again allows you to download full galleries with H@H, except it is now fully automatic; the old .hathdl files are a thing of the past.
The new Downloader works in conjunction with the Archiver, and uses the same GP/quota system. These downloads will no longer eat from the image limit nor suffer from the old hitrate-dependent waiting periods. This makes things a lot simpler and more reliable, and allows galleries to be downloaded much faster than before.
Support was added to download any of the six resolution options. It is also not subject to the 25% limit for resampled archives; as long as at least one file will be resampled with a given option, it is available. Like the Archiver, it obeys your Gallery Name Display setting.
You will notice some new buttons at the bottom of the Archive Download popup, one for each resolution option. Clicking any of these buttons will queue the download for your client, which will start downloading it automatically within a couple of minutes.
If you have multiple clients, you will have to select your default downloading client from the H@H page by checking "Use this client for the H@H Downloader" on the desired client on the bottom of the client settings page. If you only have one client, doing this is optional and will have no effect.
By default, the Downloader will limit download speeds to your throttle setting. Speed limits can be disabled globally with the "Disable Client-Side Speed Limit" checkbox or --disable-bwm argument, or separately for downloads with the new --disable-download-bwm argument.
The Downloader will
not count towards nor honor the hourly bandwidth limit setting, so if you have a data traffic cap on the connection, you may want to be careful considering how much faster the new Downloader is.
At the end of each gallery download, the Downloader will report any failed downloads to the H@H servers. This is not currently used for anything, but in the future this will be a factor in trust/quality calculations.
While file requests that can be handled by the downloading client will usually be redirected back to itself by requesting it from localhost, files downloaded with the Downloader will
not in general be added to your cache, as H@H now uses static ranges exclusively.
Therefore, downloading files with the Downloader will have zero effect on the hitrate; the feature is solely a convenience bonus for running H@H, not a way to fill the cache faster than normal.
- Added --cache-dir, --temp-dir, --data-dir, --log-dir and --download-dir arguments that allow you to change the locations of these directories without faffing with symlinks or mount points. Quotes are required if there are spaces in the path. H@H will attempt to create the directory if it does not exist.
linux usage: --cache-dir=/some/cache/location --temp-dir=/dev/shm/hath --log-dir=/dev/shm/hath
windows usage: --download-dir="c:\some download dir with spaces"
Using a ramdisk for the log and/or temp folders will slightly reduce the I/O for the client, at the expense of storing logs and/or temp files that are currently being downloaded in RAM.
You can set the temp and log directories to the same path. However, if you choose to specify the data/cache/download directories, you should use unique paths for these.
- Added the --flush-logs argument that makes the logger flush the log to disk for every written line, which is useful for people who like to analyze their log live. This will increase I/O, and is mostly recommended for when the log is written to a ramdisk.
Memorial Wall of Internal Changes, Improvements and Fixes- The cache handler has been completely rewritten and no longer depends on SQLite. As the client no longer needs to send a list of files to the server, we can now make due with filesystem metadata and an optional in-memory cache to implement the LRU mechanism.
- Most of the socket/file handling stuff has been rewritten to use NIO channels and buffers, reducing buffer copying overhead and CPU usage.
- The bandwidth limiter has been rewritten to use byte buckets with 20ms resolution instead of a fixed delay per packet, which should make it much less prone to excessive sleeping on the job. Note that if the connection is mostly idle, this change can make transfers that only take a small fraction of a second complete faster than the speed limit would suggest.
- Switched to using two levels of redirection for the file cache to avoid problems with very large caches. This means that every static range now has its own separate directory.
- The LRU cache now uses bits 16-39 instead of 0-23 to avoid the grouping effects caused by static ranges.
- Static range and H@H downloader file requests will now get a direct URL to a suitable image server or H@H client rather than having to go through the EH cluster redirection, which is both more efficient and more reliable, and avoids problems caused by factors like routing failures, blacklists and temporary IP blocks.
- The receiving speed tester used to test other clients on behalf of the network no longer stores the received junk data in memory for no particular reason, instead the data is discarded as it comes in.
- The sending speed tester will now create a small buffer of random data and reuse it, rather than burn CPU cycles to generate the up to 25 MB of random data required.
- Validating proxy downloaded files is now done in parallel with downloading them, reducing peak CPU/RAM usage and eliminating some wasted I/O.
- Rather than fully buffering downloading files to RAM before writing them to disk, we now write them to disk as the data arrives, which decreases peak RAM usage. As long as the OS has free RAM available, this will not significantly increase actual disk I/O due to OS-level caching and buffering, but if you want something close to the old behavior, you can point the temp folder to a ramdisk.
- Maximum supported file size is now 4 GB. Which ought to be enough for everyone. Initially this is soft capped to 100 MB, but can be increased server-side without a client update.
- During the initial cache scan, we now check that the filesize for every file is correct, as a form of cheap file verification. For a full verification, you can still use --verify-cache.
- Revalidating files on startup with --verify-cache is now far more efficient memory-wise.
- The log directory is now separate from the data directory, mostly to make it easier to point the logs to non-persistent storage. As we no longer use a backing database, the only current use for the data directory is the client_login file, but unlike the logs we do want that to be persistent.
- When --disable-logging is given as a command-line argument, the log file will no longer be opened or created at startup.
- After startup, H@H now uses the provided list of RPC servers from the initial contact directly instead of relying on the DNS lookup to rpc.hentaiathome.net. This should significantly improve resiliency if any of the RPC servers are unavailable, and also eliminates problems caused by flaky DNS servers.
- Server check-in frequency was increased from 300 seconds to 110 seconds.
- Rewrote all the other things as well.
- (GUI) The logpane was tweaked to avoid gobbling up massive amounts of RAM when the log was scrolling quickly. To enable this, the H@H window is now fixed size. For larger clients it is still recommended to use the CLI version to avoid the overhead.
- (GUI) The graph now shows the 60-second average line only for improved visiblity. It was also tweaked to improve the way it is drawn in general.
- (GUI) Various minor visiblity improvements. The tacky connection/speed bars were also discarded in favor of a plain connection readout and the improved speed graph.
- (Server) Added checks to prevent clients from being speed tested excessively often.
- (Server) Brand new clients now start at 1500 historic/lomark/himark quality instead of 0, meaning that they'll get going faster than before.
- (Server) For new clients, we now require at least 5 Mbit/s of both download and upload bandwidth, with 200 KB/s and 10 GB of disk space reserved for H@H. This is not retroactive for existing clients.
1.3.1 - 2016-10-05- Some very high-traffic clients (3-400+ hitrate) experienced a situation where the file pruning mechanism could fall behind and make the cache grow past its limit. The cache handler will now automatically make the pruning mechanism more aggressive if it is in danger of exceeding the cache limit.
- If the cache is above the limit at startup, the cache handler will now loop the pruning mechanism during initialization until the cache is below the current limit.
- The GUI can again be resized. Note that increasing the window size will somewhat increase how much RAM the GUI consumes. Making the window wider will not reveal full cache lines that were written before being extended, to avoid having to store arbitrarily long log lines.
1.3.2 - 2016-10-08- Rather than blindly iterating over the static range directories to find files to prune, we now cache the timestamp of the oldest remaining file from each static range and scan this data structure to find the one with the oldest file. This should significantly reduce disk I/O when the cache is close to full, and vastly decreases the time required for the startup prune if one is needed.
- The aged static range scan was integrated with the cache initialization, which cuts the startup initialization from three passes to two, making startup a bit faster.
- The cache phase 1 cleanup pass now postpones some parts of the cleanup until the phase 2 init pass, where it can be done far more efficiently.
- The file pruning mechanism will now adjust the actual cutoff time depending on the age of the oldest cached file, which cuts down how often the pruner has to run.
- Added a sanity check during the start of the cache cleanup to warn and delay wiping the cache if it has one and there are no static ranges assigned.
1.3.3 - 2016-10-11- Added persistence to the new cache handler, which means it no longer has to scan the cache on regular startups. The client will now start almost instantly regardless of cache size and CPU/storage performance, and go straight to the speed test.
A cache scan will now only be required if the shutdown was not clean, if the client had a reduction in the number of static ranges, or if a client that previously used --use-less-memory is started without using it. It can also be triggered manually with --rescan-cache
Note that the first startup after upgrading to 1.3.3 will always involve a cache scan, as the necessary data for fast startup is not saved by earlier clients.
- The static range age cache now tracks all static ranges instead of just the "aged" ones. This was necessary for long-term cache persistence, and because of the changes in 1.3.2 it will have no impact on performance.
Notes (READ BEFORE UPGRADING)H@H 1.3 requires Java SE 7 or later to run. This is a version bump from 1.2.6, which could run on Java 6.
The 1.3 branch should be considered experimental/unstable and has a lot of new and rewritten code. If you are not comfortable with this, please stick with 1.2.6 for now.
1.3 and future versions of H@H will not support the
Hentai@Home Proxy functionality. This option will eventually be removed from the site, and using it with a 1.3 client will not work.
The first startup with this client will take longer than normal, as it will have to reorganize the cache and delete the now unused files that are not in the client's list of static ranges.
Because of this reorg, it is not trivial to revert to H@H 1.2.6, as you would have to manually collapse the cache tree from two to one directory levels.
Note that while 1.3 explicity drops files that are outside its static ranges, they are no longer being used with previous versions either, so this does not actually affect the client's hitrate.
Startups past the first one will generally be faster than before, but as it involves a mandatory scan of the cache file structure, it could potentially take longer if you have particularly slow CPU and storage.
Barring any bugs, this release should be far more light-weight than 1.2.6, with reduced CPU usage, 1/4th to 1/2th of the RAM requirements, and a small fraction of the disk I/O for bookkeeping purposes.
After upgrading, you can delete sqlite-jdbc-3.7.2.jar and the hathdl directory from the main directory, as well as hath.db + the log files from the data directory.
To update an existing client: shut it down, download Hentai@Home 1.3.3, extract the archive, copy the jar files over the existing ones, then restart the client.The full source code for H@H is available and licensed under the GNU General Public License v3, and can be downloaded here. Building it from source only requires the free Java SE 7 JDK.For information on how to join Hentai@Home, check out The Hentai@Home Project FAQ.
Hentai@Home 1.3, We will build a wall, and Mexico is going to read it
Oct 1 2016, 12:17
