Welcome Guest ( Log In | Register )

> Hentai@Home 1.3, We will build a wall, and Mexico is going to read it

 
post Oct 1 2016, 12:17
Post #1
Tenboro

Admin




1.3.0 - 2016-10-01

Feature Additions

- The H@H Downloader has made a surprising yet glorious return. This again allows you to download full galleries with H@H, except it is now fully automatic; the old .hathdl files are a thing of the past.

The new Downloader works in conjunction with the Archiver, and uses the same GP/quota system. These downloads will no longer eat from the image limit nor suffer from the old hitrate-dependent waiting periods. This makes things a lot simpler and more reliable, and allows galleries to be downloaded much faster than before.

Support was added to download any of the six resolution options. It is also not subject to the 25% limit for resampled archives; as long as at least one file will be resampled with a given option, it is available. Like the Archiver, it obeys your Gallery Name Display setting.

You will notice some new buttons at the bottom of the Archive Download popup, one for each resolution option. Clicking any of these buttons will queue the download for your client, which will start downloading it automatically within a couple of minutes.

If you have multiple clients, you will have to select your default downloading client from the H@H page by checking "Use this client for the H@H Downloader" on the desired client on the bottom of the client settings page. If you only have one client, doing this is optional and will have no effect.

By default, the Downloader will limit download speeds to your throttle setting. Speed limits can be disabled globally with the "Disable Client-Side Speed Limit" checkbox or --disable-bwm argument, or separately for downloads with the new --disable-download-bwm argument.

The Downloader will not count towards nor honor the hourly bandwidth limit setting, so if you have a data traffic cap on the connection, you may want to be careful considering how much faster the new Downloader is.

At the end of each gallery download, the Downloader will report any failed downloads to the H@H servers. This is not currently used for anything, but in the future this will be a factor in trust/quality calculations.

While file requests that can be handled by the downloading client will usually be redirected back to itself by requesting it from localhost, files downloaded with the Downloader will not in general be added to your cache, as H@H now uses static ranges exclusively.

Therefore, downloading files with the Downloader will have zero effect on the hitrate; the feature is solely a convenience bonus for running H@H, not a way to fill the cache faster than normal.


- Added --cache-dir, --temp-dir, --data-dir, --log-dir and --download-dir arguments that allow you to change the locations of these directories without faffing with symlinks or mount points. Quotes are required if there are spaces in the path. H@H will attempt to create the directory if it does not exist.

linux usage: --cache-dir=/some/cache/location --temp-dir=/dev/shm/hath --log-dir=/dev/shm/hath
windows usage: --download-dir="c:\some download dir with spaces"

Using a ramdisk for the log and/or temp folders will slightly reduce the I/O for the client, at the expense of storing logs and/or temp files that are currently being downloaded in RAM.

You can set the temp and log directories to the same path. However, if you choose to specify the data/cache/download directories, you should use unique paths for these.


- Added the --flush-logs argument that makes the logger flush the log to disk for every written line, which is useful for people who like to analyze their log live. This will increase I/O, and is mostly recommended for when the log is written to a ramdisk.


Memorial Wall of Internal Changes, Improvements and Fixes

- The cache handler has been completely rewritten and no longer depends on SQLite. As the client no longer needs to send a list of files to the server, we can now make due with filesystem metadata and an optional in-memory cache to implement the LRU mechanism.

- Most of the socket/file handling stuff has been rewritten to use NIO channels and buffers, reducing buffer copying overhead and CPU usage.

- The bandwidth limiter has been rewritten to use byte buckets with 20ms resolution instead of a fixed delay per packet, which should make it much less prone to excessive sleeping on the job. Note that if the connection is mostly idle, this change can make transfers that only take a small fraction of a second complete faster than the speed limit would suggest.

- Switched to using two levels of redirection for the file cache to avoid problems with very large caches. This means that every static range now has its own separate directory.

- The LRU cache now uses bits 16-39 instead of 0-23 to avoid the grouping effects caused by static ranges.

- Static range and H@H downloader file requests will now get a direct URL to a suitable image server or H@H client rather than having to go through the EH cluster redirection, which is both more efficient and more reliable, and avoids problems caused by factors like routing failures, blacklists and temporary IP blocks.

- The receiving speed tester used to test other clients on behalf of the network no longer stores the received junk data in memory for no particular reason, instead the data is discarded as it comes in.

- The sending speed tester will now create a small buffer of random data and reuse it, rather than burn CPU cycles to generate the up to 25 MB of random data required.

- Validating proxy downloaded files is now done in parallel with downloading them, reducing peak CPU/RAM usage and eliminating some wasted I/O.

- Rather than fully buffering downloading files to RAM before writing them to disk, we now write them to disk as the data arrives, which decreases peak RAM usage. As long as the OS has free RAM available, this will not significantly increase actual disk I/O due to OS-level caching and buffering, but if you want something close to the old behavior, you can point the temp folder to a ramdisk.

- Maximum supported file size is now 4 GB. Which ought to be enough for everyone. Initially this is soft capped to 100 MB, but can be increased server-side without a client update.

- During the initial cache scan, we now check that the filesize for every file is correct, as a form of cheap file verification. For a full verification, you can still use --verify-cache.

- Revalidating files on startup with --verify-cache is now far more efficient memory-wise.

- The log directory is now separate from the data directory, mostly to make it easier to point the logs to non-persistent storage. As we no longer use a backing database, the only current use for the data directory is the client_login file, but unlike the logs we do want that to be persistent.

- When --disable-logging is given as a command-line argument, the log file will no longer be opened or created at startup.

- After startup, H@H now uses the provided list of RPC servers from the initial contact directly instead of relying on the DNS lookup to rpc.hentaiathome.net. This should significantly improve resiliency if any of the RPC servers are unavailable, and also eliminates problems caused by flaky DNS servers.

- Server check-in frequency was increased from 300 seconds to 110 seconds.

- Rewrote all the other things as well.

- (GUI) The logpane was tweaked to avoid gobbling up massive amounts of RAM when the log was scrolling quickly. To enable this, the H@H window is now fixed size. For larger clients it is still recommended to use the CLI version to avoid the overhead.

- (GUI) The graph now shows the 60-second average line only for improved visiblity. It was also tweaked to improve the way it is drawn in general.

- (GUI) Various minor visiblity improvements. The tacky connection/speed bars were also discarded in favor of a plain connection readout and the improved speed graph.

- (Server) Added checks to prevent clients from being speed tested excessively often.

- (Server) Brand new clients now start at 1500 historic/lomark/himark quality instead of 0, meaning that they'll get going faster than before.

- (Server) For new clients, we now require at least 5 Mbit/s of both download and upload bandwidth, with 200 KB/s and 10 GB of disk space reserved for H@H. This is not retroactive for existing clients.


1.3.1 - 2016-10-05

- Some very high-traffic clients (3-400+ hitrate) experienced a situation where the file pruning mechanism could fall behind and make the cache grow past its limit. The cache handler will now automatically make the pruning mechanism more aggressive if it is in danger of exceeding the cache limit.

- If the cache is above the limit at startup, the cache handler will now loop the pruning mechanism during initialization until the cache is below the current limit.

- The GUI can again be resized. Note that increasing the window size will somewhat increase how much RAM the GUI consumes. Making the window wider will not reveal full cache lines that were written before being extended, to avoid having to store arbitrarily long log lines.


1.3.2 - 2016-10-08

- Rather than blindly iterating over the static range directories to find files to prune, we now cache the timestamp of the oldest remaining file from each static range and scan this data structure to find the one with the oldest file. This should significantly reduce disk I/O when the cache is close to full, and vastly decreases the time required for the startup prune if one is needed.

- The aged static range scan was integrated with the cache initialization, which cuts the startup initialization from three passes to two, making startup a bit faster.

- The cache phase 1 cleanup pass now postpones some parts of the cleanup until the phase 2 init pass, where it can be done far more efficiently.

- The file pruning mechanism will now adjust the actual cutoff time depending on the age of the oldest cached file, which cuts down how often the pruner has to run.

- Added a sanity check during the start of the cache cleanup to warn and delay wiping the cache if it has one and there are no static ranges assigned.


1.3.3 - 2016-10-11

- Added persistence to the new cache handler, which means it no longer has to scan the cache on regular startups. The client will now start almost instantly regardless of cache size and CPU/storage performance, and go straight to the speed test.

A cache scan will now only be required if the shutdown was not clean, if the client had a reduction in the number of static ranges, or if a client that previously used --use-less-memory is started without using it. It can also be triggered manually with --rescan-cache

Note that the first startup after upgrading to 1.3.3 will always involve a cache scan, as the necessary data for fast startup is not saved by earlier clients.

- The static range age cache now tracks all static ranges instead of just the "aged" ones. This was necessary for long-term cache persistence, and because of the changes in 1.3.2 it will have no impact on performance.


Notes (READ BEFORE UPGRADING)

H@H 1.3 requires Java SE 7 or later to run. This is a version bump from 1.2.6, which could run on Java 6.

The 1.3 branch should be considered experimental/unstable and has a lot of new and rewritten code. If you are not comfortable with this, please stick with 1.2.6 for now.

1.3 and future versions of H@H will not support the Hentai@Home Proxy functionality. This option will eventually be removed from the site, and using it with a 1.3 client will not work.

The first startup with this client will take longer than normal, as it will have to reorganize the cache and delete the now unused files that are not in the client's list of static ranges.

Because of this reorg, it is not trivial to revert to H@H 1.2.6, as you would have to manually collapse the cache tree from two to one directory levels.

Note that while 1.3 explicity drops files that are outside its static ranges, they are no longer being used with previous versions either, so this does not actually affect the client's hitrate.

Startups past the first one will generally be faster than before, but as it involves a mandatory scan of the cache file structure, it could potentially take longer if you have particularly slow CPU and storage.

Barring any bugs, this release should be far more light-weight than 1.2.6, with reduced CPU usage, 1/4th to 1/2th of the RAM requirements, and a small fraction of the disk I/O for bookkeeping purposes.

After upgrading, you can delete sqlite-jdbc-3.7.2.jar and the hathdl directory from the main directory, as well as hath.db + the log files from the data directory.


To update an existing client: shut it down, download Hentai@Home 1.3.3, extract the archive, copy the jar files over the existing ones, then restart the client.

The full source code for H@H is available and licensed under the GNU General Public License v3, and can be downloaded here. Building it from source only requires the free Java SE 7 JDK.

For information on how to join Hentai@Home, check out The Hentai@Home Project FAQ.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post


Posts in this topic
Tenboro   Hentai@Home 1.3   Oct 1 2016, 12:17
Simpleton8   [quote name='Tenboro' post='4618991' date='Oct 1 2...   Oct 1 2016, 13:13
sc01a   Applying the update on some of my clients rn.   Oct 1 2016, 17:43
Maximum_Joe   What did you use as the fixed GUI size? Seems odd....   Oct 1 2016, 20:30
blue penguin   [quote name='Tenboro' post='4618991' date='Oct 1 2...   Oct 1 2016, 21:46
Simpleton8   [quote name='blue penguin' post='4619388' date='Oc...   Oct 1 2016, 21:59
Tenboro   [quote name='Maximum_Joe' post='4619355' date='Oct...   Oct 1 2016, 22:29
Simpleton8   [quote name='Tenboro' post='4619420' date='Oct 1 2...   Oct 1 2016, 23:26
Infinal   Updating appears to have broken images being loade...   Oct 1 2016, 22:43
blue penguin   [quote name='Tenboro' post='4619420' date='Oct 1 2...   Oct 1 2016, 22:43
Dideet1936   There are a few new (or at least I don't recog...   Oct 1 2016, 22:44
Maximum_Joe   [quote name='Dideet1936' post='4619428' date='Oct ...   Oct 1 2016, 23:17
Tenboro   [quote name='Infinal' post='4619426' date='Oct 1 2...   Oct 1 2016, 23:49
Infinal   [quote name='Tenboro' post='4619475' date='Oct 1 2...   Oct 2 2016, 00:52
Soul--Reaver   Client works nice Kinda missing the proxy but you...   Oct 2 2016, 07:13
riesza   Is this kind of multi resolution option planned fo...   Oct 2 2016, 14:39
Tenboro   [quote name='riesza' post='4620165' date='Oct 2 20...   Oct 2 2016, 15:05
Maximum_Joe   Since the logpane is harder to copy and paste and ...   Oct 2 2016, 15:17
Tenboro   [quote name='Maximum_Joe' post='4620202' date='Oct...   Oct 2 2016, 16:56
Maximum_Joe   [quote name='Tenboro' post='4620294' date='Oct 2 2...   Oct 2 2016, 17:11
blue penguin   [quote name='Tenboro' post='4620294' date='Oct 2 2...   Oct 2 2016, 18:48
dlactionn   Sounds awesome, though I'll wait with update i...   Oct 2 2016, 20:54
Tenboro   [quote name='dlactionn' post='4620459' date='Oct 2...   Oct 3 2016, 19:17
Jay Low   Awesome improvement Tenboro! galleryinfo.txt i...   Oct 2 2016, 23:25
zxwecrt   Well, that's certainly a welcome change for ma...   Oct 3 2016, 17:27
Jay Low   2 Things for now: Is there a way to select which ...   Oct 3 2016, 20:13
unicomt   [quote name='Tenboro' post='4618991' date='Oct 1 2...   Oct 3 2016, 22:23
Tenboro   [quote name='Jay Low' post='4621361' date='Oct 3 2...   Oct 3 2016, 23:08
unicomt   [quote name='Tenboro' post='4621436' date='Oct 3 2...   Oct 4 2016, 02:14
Jay Low   [quote name='Tenboro' post='4621436' date='Oct 3 2...   Oct 4 2016, 08:34
TacticalBigBoss   Could the website be updated so that this could be...   Oct 3 2016, 23:15
Maximum_Joe   [quote name='TacticalBigBoss' post='4621440' date=...   Oct 3 2016, 23:24
Tenboro   [quote name='TacticalBigBoss' post='4621440' date=...   Oct 4 2016, 00:04
Phreeman   Upgraded, and my disk cache just keep growing, the...   Oct 4 2016, 06:07
Tenboro   [quote name='Phreeman' post='4621704' date='Oct 4 ...   Oct 4 2016, 15:08
Phreeman   [quote name='Tenboro' post='4622038' date='Oct 4 2...   Oct 5 2016, 06:50
Daodaodaidai   Hi, thanks for the update. But it seems i can...   Oct 4 2016, 23:00
Maximum_Joe   [quote name='Tenboro' post='4618991' date='Oct 1 2...   Oct 4 2016, 23:10
Chunga   What are the chances to get an auto-updater?   Oct 5 2016, 00:12
Tenboro   1.3.1 was released, which fixes the following issu...   Oct 5 2016, 16:36
Tenboro   [quote name='Chunga' post='4622358' date='Oct 5 20...   Oct 5 2016, 17:02
Alt1523   I updated to 1.3 today and it resolved the startup...   Oct 5 2016, 18:54
mike23   How long should the cache cleanup take? It's b...   Oct 6 2016, 02:17
Maximum_Joe   I'd say just under an hour for your case.   Oct 6 2016, 02:19
mike23   [quote name='Maximum_Joe' post='4623449' date='Oct...   Oct 6 2016, 02:25
foobar20324   Experimental indeed. 1.3 works somewhat decent on...   Oct 6 2016, 02:55
mrpops   no mention of >5k static ranges :(   Oct 6 2016, 10:47
Tenboro   [quote name='mike23' post='4623461' date='Oct 6 20...   Oct 6 2016, 13:34
mike23   [quote name='Tenboro' post='4623892' date='Oct 6 2...   Oct 7 2016, 05:42
Tenboro   [quote name='mike23' post='4624604' date='Oct 7 20...   Oct 7 2016, 13:28
Tenboro   1.3.2 was released, which primarily improves the f...   Oct 8 2016, 18:00
Tenboro   1.3.3 was released, which primarily makes startups...   Oct 11 2016, 15:50
Maximum_Joe   [quote name='Tenboro' post='4629665' date='Oct 11 ...   Oct 11 2016, 23:43
Tenboro   [quote name='Maximum_Joe' post='4629974' date='Oct...   Oct 11 2016, 23:48
tes143   hah 1.3.3 openjdk version "1.8.0_102" O...   Oct 12 2016, 06:55
tes143   Sorry my bad. Few days ago i install btsync for ba...   Oct 12 2016, 07:55
foobar20324   [quote name='tes143' post='4630373' date='Oct 12 2...   Oct 12 2016, 09:07
Tenboro   [quote name='tes143' post='4630373' date='Oct 12 2...   Oct 12 2016, 13:29
yulisunny   This is great!   Oct 13 2016, 03:30
Sapo84   Updated one of my client to 1.3.3, went pretty smo...   Oct 13 2016, 12:20
Tenboro   [quote name='Sapo84' post='4631587' date='Oct 13 2...   Oct 13 2016, 12:37
Sapo84   [quote name='Tenboro' post='4631604' date='Oct 13 ...   Oct 13 2016, 12:57
Tenboro   [quote name='Sapo84' post='4631624' date='Oct 13 2...   Oct 13 2016, 13:12
Mantra64   - (Server) Brand new clients now start at 1500 his...   Oct 13 2016, 23:39
Tenboro   [quote name='Mantra64' post='4632009' date='Oct 13...   Oct 13 2016, 23:54
TennoujiSaburou   I know I'm not a techie, but can I ask for a m...   Oct 14 2016, 07:04
zbxehzqn   sorry im new here when i see how you do the speed ...   Oct 14 2016, 09:34
Tenboro   [quote name='zbxehzqn' post='4632456' date='Oct 14...   Oct 14 2016, 17:09
zbxehzqn   this? private class walkFile extends SimpleFileVi...   Oct 14 2016, 19:41
Tenboro   [quote name='zbxehzqn' post='4632823' date='Oct 14...   Oct 15 2016, 00:18
zbxehzqn   i can make that based off 1.3.3. you will use it? ...   Oct 15 2016, 09:52
foobar20324   new HashMap<>(5, 1); You certainly...   Oct 15 2016, 11:55
zbxehzqn   ok. i will try something else.   Oct 15 2016, 12:27
zbxehzqn   foobar20324 this? private class walkFile extends ...   Oct 15 2016, 18:55
foobar20324   Not what I meant (just that if you are using a Has...   Oct 16 2016, 16:50
tdms   zbxehzqn I think these should be in the else block...   Oct 16 2016, 21:39
Maximum_Joe   Tenboro, maybe make a GitHub page or at least a de...   Oct 16 2016, 21:44
Tenboro   [quote name='Maximum_Joe' post='4634944' date='Oct...   Oct 18 2016, 14:22
Void Domain   10 hours in, initial setup is only on cleanup pass...   Oct 18 2016, 08:52
Majala   Hi, so sorry for being a noob but even though I...   Oct 18 2016, 16:58
Tenboro   [quote name='Majala' post='4636786' date='Oct 18 2...   Oct 18 2016, 18:13
Majala   [quote name='Tenboro' post='4636850' date='Oct 18 ...   Oct 18 2016, 20:04
Void Domain   Finally got it to run. I miss the speed meter in t...   Oct 19 2016, 14:01
Tenboro   [quote name='Void Domain' post='4637689' date='Oct...   Oct 19 2016, 17:45
Synitare   My client seems to have timed out sometime last ni...   Oct 21 2016, 17:09
lishde1   iv never been able to get H@H to work it allways s...   Oct 24 2016, 17:32
blue penguin   [quote name='lishde1' post='4643037' date='Oct 24 ...   Oct 24 2016, 21:00
jenga201   [quote name='blue penguin' post='4643138' date='Oc...   Oct 26 2016, 01:53
mewsf   I've seen that the client now is fetching file...   Nov 3 2016, 02:29
zbxehzqn   sorry for how long i taked. i had to make a server...   Nov 6 2016, 17:22
whzzz28   Hi, Not sure if this was overlooked but the H@H s...   Nov 10 2016, 08:29
Sapo84   [quote name='whzzz28' post='4659572' date='Nov 10 ...   Nov 10 2016, 10:21
haijiawei   sound good,thank you!   Nov 11 2016, 05:43
aas889   We will gonna learn lots of things after 4 years.   Nov 11 2016, 23:03
Vulpix   Ever since I switched to 1.3.3, my "quality...   Nov 12 2016, 23:18
Tenboro   [quote name='Vulpix' post='4662345' date='Nov 12 2...   Nov 13 2016, 12:33
perfut   Helpful, thanks.   Nov 13 2016, 09:23
zbxehzqn   tenboro: there is anything i had miss in the code ...   Nov 13 2016, 14:42
Tenboro   [quote name='zbxehzqn' post='4663024' date='Nov 13...   Nov 13 2016, 16:57
zbxehzqn   you can remove the new files to make less complex?...   Nov 13 2016, 17:33
2 Pages V  1 2 >


Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 


Lo-Fi Version Time is now: 17th July 2025 - 04:55