Welcome Guest ( Log In | Register )

6 Pages V  1 2 3 > »   
Closed TopicStart new topic
> Hentai@Home 1.3, We will build a wall, and Mexico is going to read it

 
post Oct 1 2016, 12:17
Post #1
Tenboro

Admin




1.3.0 - 2016-10-01

Feature Additions

- The H@H Downloader has made a surprising yet glorious return. This again allows you to download full galleries with H@H, except it is now fully automatic; the old .hathdl files are a thing of the past.

The new Downloader works in conjunction with the Archiver, and uses the same GP/quota system. These downloads will no longer eat from the image limit nor suffer from the old hitrate-dependent waiting periods. This makes things a lot simpler and more reliable, and allows galleries to be downloaded much faster than before.

Support was added to download any of the six resolution options. It is also not subject to the 25% limit for resampled archives; as long as at least one file will be resampled with a given option, it is available. Like the Archiver, it obeys your Gallery Name Display setting.

You will notice some new buttons at the bottom of the Archive Download popup, one for each resolution option. Clicking any of these buttons will queue the download for your client, which will start downloading it automatically within a couple of minutes.

If you have multiple clients, you will have to select your default downloading client from the H@H page by checking "Use this client for the H@H Downloader" on the desired client on the bottom of the client settings page. If you only have one client, doing this is optional and will have no effect.

By default, the Downloader will limit download speeds to your throttle setting. Speed limits can be disabled globally with the "Disable Client-Side Speed Limit" checkbox or --disable-bwm argument, or separately for downloads with the new --disable-download-bwm argument.

The Downloader will not count towards nor honor the hourly bandwidth limit setting, so if you have a data traffic cap on the connection, you may want to be careful considering how much faster the new Downloader is.

At the end of each gallery download, the Downloader will report any failed downloads to the H@H servers. This is not currently used for anything, but in the future this will be a factor in trust/quality calculations.

While file requests that can be handled by the downloading client will usually be redirected back to itself by requesting it from localhost, files downloaded with the Downloader will not in general be added to your cache, as H@H now uses static ranges exclusively.

Therefore, downloading files with the Downloader will have zero effect on the hitrate; the feature is solely a convenience bonus for running H@H, not a way to fill the cache faster than normal.


- Added --cache-dir, --temp-dir, --data-dir, --log-dir and --download-dir arguments that allow you to change the locations of these directories without faffing with symlinks or mount points. Quotes are required if there are spaces in the path. H@H will attempt to create the directory if it does not exist.

linux usage: --cache-dir=/some/cache/location --temp-dir=/dev/shm/hath --log-dir=/dev/shm/hath
windows usage: --download-dir="c:\some download dir with spaces"

Using a ramdisk for the log and/or temp folders will slightly reduce the I/O for the client, at the expense of storing logs and/or temp files that are currently being downloaded in RAM.

You can set the temp and log directories to the same path. However, if you choose to specify the data/cache/download directories, you should use unique paths for these.


- Added the --flush-logs argument that makes the logger flush the log to disk for every written line, which is useful for people who like to analyze their log live. This will increase I/O, and is mostly recommended for when the log is written to a ramdisk.


Memorial Wall of Internal Changes, Improvements and Fixes

- The cache handler has been completely rewritten and no longer depends on SQLite. As the client no longer needs to send a list of files to the server, we can now make due with filesystem metadata and an optional in-memory cache to implement the LRU mechanism.

- Most of the socket/file handling stuff has been rewritten to use NIO channels and buffers, reducing buffer copying overhead and CPU usage.

- The bandwidth limiter has been rewritten to use byte buckets with 20ms resolution instead of a fixed delay per packet, which should make it much less prone to excessive sleeping on the job. Note that if the connection is mostly idle, this change can make transfers that only take a small fraction of a second complete faster than the speed limit would suggest.

- Switched to using two levels of redirection for the file cache to avoid problems with very large caches. This means that every static range now has its own separate directory.

- The LRU cache now uses bits 16-39 instead of 0-23 to avoid the grouping effects caused by static ranges.

- Static range and H@H downloader file requests will now get a direct URL to a suitable image server or H@H client rather than having to go through the EH cluster redirection, which is both more efficient and more reliable, and avoids problems caused by factors like routing failures, blacklists and temporary IP blocks.

- The receiving speed tester used to test other clients on behalf of the network no longer stores the received junk data in memory for no particular reason, instead the data is discarded as it comes in.

- The sending speed tester will now create a small buffer of random data and reuse it, rather than burn CPU cycles to generate the up to 25 MB of random data required.

- Validating proxy downloaded files is now done in parallel with downloading them, reducing peak CPU/RAM usage and eliminating some wasted I/O.

- Rather than fully buffering downloading files to RAM before writing them to disk, we now write them to disk as the data arrives, which decreases peak RAM usage. As long as the OS has free RAM available, this will not significantly increase actual disk I/O due to OS-level caching and buffering, but if you want something close to the old behavior, you can point the temp folder to a ramdisk.

- Maximum supported file size is now 4 GB. Which ought to be enough for everyone. Initially this is soft capped to 100 MB, but can be increased server-side without a client update.

- During the initial cache scan, we now check that the filesize for every file is correct, as a form of cheap file verification. For a full verification, you can still use --verify-cache.

- Revalidating files on startup with --verify-cache is now far more efficient memory-wise.

- The log directory is now separate from the data directory, mostly to make it easier to point the logs to non-persistent storage. As we no longer use a backing database, the only current use for the data directory is the client_login file, but unlike the logs we do want that to be persistent.

- When --disable-logging is given as a command-line argument, the log file will no longer be opened or created at startup.

- After startup, H@H now uses the provided list of RPC servers from the initial contact directly instead of relying on the DNS lookup to rpc.hentaiathome.net. This should significantly improve resiliency if any of the RPC servers are unavailable, and also eliminates problems caused by flaky DNS servers.

- Server check-in frequency was increased from 300 seconds to 110 seconds.

- Rewrote all the other things as well.

- (GUI) The logpane was tweaked to avoid gobbling up massive amounts of RAM when the log was scrolling quickly. To enable this, the H@H window is now fixed size. For larger clients it is still recommended to use the CLI version to avoid the overhead.

- (GUI) The graph now shows the 60-second average line only for improved visiblity. It was also tweaked to improve the way it is drawn in general.

- (GUI) Various minor visiblity improvements. The tacky connection/speed bars were also discarded in favor of a plain connection readout and the improved speed graph.

- (Server) Added checks to prevent clients from being speed tested excessively often.

- (Server) Brand new clients now start at 1500 historic/lomark/himark quality instead of 0, meaning that they'll get going faster than before.

- (Server) For new clients, we now require at least 5 Mbit/s of both download and upload bandwidth, with 200 KB/s and 10 GB of disk space reserved for H@H. This is not retroactive for existing clients.


1.3.1 - 2016-10-05

- Some very high-traffic clients (3-400+ hitrate) experienced a situation where the file pruning mechanism could fall behind and make the cache grow past its limit. The cache handler will now automatically make the pruning mechanism more aggressive if it is in danger of exceeding the cache limit.

- If the cache is above the limit at startup, the cache handler will now loop the pruning mechanism during initialization until the cache is below the current limit.

- The GUI can again be resized. Note that increasing the window size will somewhat increase how much RAM the GUI consumes. Making the window wider will not reveal full cache lines that were written before being extended, to avoid having to store arbitrarily long log lines.


1.3.2 - 2016-10-08

- Rather than blindly iterating over the static range directories to find files to prune, we now cache the timestamp of the oldest remaining file from each static range and scan this data structure to find the one with the oldest file. This should significantly reduce disk I/O when the cache is close to full, and vastly decreases the time required for the startup prune if one is needed.

- The aged static range scan was integrated with the cache initialization, which cuts the startup initialization from three passes to two, making startup a bit faster.

- The cache phase 1 cleanup pass now postpones some parts of the cleanup until the phase 2 init pass, where it can be done far more efficiently.

- The file pruning mechanism will now adjust the actual cutoff time depending on the age of the oldest cached file, which cuts down how often the pruner has to run.

- Added a sanity check during the start of the cache cleanup to warn and delay wiping the cache if it has one and there are no static ranges assigned.


1.3.3 - 2016-10-11

- Added persistence to the new cache handler, which means it no longer has to scan the cache on regular startups. The client will now start almost instantly regardless of cache size and CPU/storage performance, and go straight to the speed test.

A cache scan will now only be required if the shutdown was not clean, if the client had a reduction in the number of static ranges, or if a client that previously used --use-less-memory is started without using it. It can also be triggered manually with --rescan-cache

Note that the first startup after upgrading to 1.3.3 will always involve a cache scan, as the necessary data for fast startup is not saved by earlier clients.

- The static range age cache now tracks all static ranges instead of just the "aged" ones. This was necessary for long-term cache persistence, and because of the changes in 1.3.2 it will have no impact on performance.


Notes (READ BEFORE UPGRADING)

H@H 1.3 requires Java SE 7 or later to run. This is a version bump from 1.2.6, which could run on Java 6.

The 1.3 branch should be considered experimental/unstable and has a lot of new and rewritten code. If you are not comfortable with this, please stick with 1.2.6 for now.

1.3 and future versions of H@H will not support the Hentai@Home Proxy functionality. This option will eventually be removed from the site, and using it with a 1.3 client will not work.

The first startup with this client will take longer than normal, as it will have to reorganize the cache and delete the now unused files that are not in the client's list of static ranges.

Because of this reorg, it is not trivial to revert to H@H 1.2.6, as you would have to manually collapse the cache tree from two to one directory levels.

Note that while 1.3 explicity drops files that are outside its static ranges, they are no longer being used with previous versions either, so this does not actually affect the client's hitrate.

Startups past the first one will generally be faster than before, but as it involves a mandatory scan of the cache file structure, it could potentially take longer if you have particularly slow CPU and storage.

Barring any bugs, this release should be far more light-weight than 1.2.6, with reduced CPU usage, 1/4th to 1/2th of the RAM requirements, and a small fraction of the disk I/O for bookkeeping purposes.

After upgrading, you can delete sqlite-jdbc-3.7.2.jar and the hathdl directory from the main directory, as well as hath.db + the log files from the data directory.


To update an existing client: shut it down, download Hentai@Home 1.3.3, extract the archive, copy the jar files over the existing ones, then restart the client.

The full source code for H@H is available and licensed under the GNU General Public License v3, and can be downloaded here. Building it from source only requires the free Java SE 7 JDK.

For information on how to join Hentai@Home, check out The Hentai@Home Project FAQ.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 13:13
Post #2
Simpleton8



Casual Poster
****
Group: Members
Posts: 314
Joined: 2-November 15
Level 258 (Godslayer)


QUOTE(Tenboro @ Oct 1 2016, 12:17) *


Barring any bugs, this release should be far more light-weight than 1.2.6, with reduced CPU usage, 1/4th to 1/2th of the RAM requirements, and a small fraction of the disk I/O for bookkeeping purposes.



This sounds nice. Going to try it out from scratch. I'll ignore all the new arguments, assuming the default setting is good.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 17:43
Post #3
sc01a



Lurker
Group: Recruits
Posts: 5
Joined: 15-March 16
Level 15 (Novice)


Applying the update on some of my clients rn.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 20:30
Post #4
Maximum_Joe



Smug Mod
***********
Group: Members
Posts: 19,623
Joined: 17-April 11
Level 489 (Godslayer)


What did you use as the fixed GUI size? Seems odd. Also not the best idea for the log portion to have no scroll bars. Too many newbies will need to be walked through opening the log files...

QUOTE(Tenboro @ Oct 1 2016, 06:17) *

we now require at least 5 Mbit/s of both download and upload bandwidth, with 200 KB/s and 10 GB of disk space

This worldwide or region specific?

This post has been edited by Maximum_Joe: Oct 1 2016, 21:26
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 21:46
Post #5
blue penguin



in umbra, igitur, pugnabimus
*********
Group: Global Mods
Posts: 6,686
Joined: 24-March 12
Level 435 (Godslayer)


QUOTE(Tenboro @ Oct 1 2016, 11:17) *
You will notice some new buttons at the bottom of the Archive Download popup, one for each resolution option. Clicking any of these buttons will queue the download for your client, which will start downloading it automatically within a couple of minutes.
I know that this is pedantry, but the buttons still appear although I do not have a running client.

The buttons fail alright, but wouldn't be easier (from a user's perspective) to not show them at all?
User is online!Profile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 21:59
Post #6
Simpleton8



Casual Poster
****
Group: Members
Posts: 314
Joined: 2-November 15
Level 258 (Godslayer)


QUOTE(blue penguin @ Oct 1 2016, 21:46) *

I know that this is pedantry, but the buttons still appear although I do not have a running client.

The buttons fail alright, but wouldn't be easier (from a user's perspective) to not show them at all?


Buttons also don't appear if Archiver Settings is set to "Auto Select Original, Auto Start." In that case, it auto downloads the gallery as zip. This is probably intentional.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 22:29
Post #7
Tenboro

Admin




QUOTE(Maximum_Joe @ Oct 1 2016, 20:30) *

What did you use as the fixed GUI size? Seems odd. Also not the best idea for the log portion to have no scroll bars. Too many newbies will need to be walked through opening the log files...


It's the old default size. And the lack of scroll bars is caused by how I optimized the log pane. During testing I observed it gobble up as much as 250 MB RAM during fast scrolling, having a hard limit on the size means we can prune it aggressively.

QUOTE(Maximum_Joe @ Oct 1 2016, 20:30) *

This worldwide or region specific?


I found that having region-specific requirements just lead to confusion, and we can afford to be just a little picky.

QUOTE(blue penguin @ Oct 1 2016, 21:46) *

I know that this is pedantry, but the buttons still appear although I do not have a running client.


That's bait.

QUOTE(Simpleton8 @ Oct 1 2016, 21:59) *

Buttons also don't appear if Archiver Settings is set to "Auto Select Original, Auto Start." In that case, it auto downloads the gallery as zip. This is probably intentional.


It is intentional, you'll have to turn that off to use the downloader.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 22:43
Post #8
Infinal



Newcomer
**
Group: Members
Posts: 79
Joined: 7-April 13
Level 411 (Godslayer)


Updating appears to have broken images being loaded through H@H while browsing the galleries, both on e-hentai.org and super secret website that forum censoring doesn’t want me to mention. What do?

This post has been edited by Infinal: Oct 1 2016, 22:46
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 22:43
Post #9
blue penguin



in umbra, igitur, pugnabimus
*********
Group: Global Mods
Posts: 6,686
Joined: 24-March 12
Level 435 (Godslayer)


QUOTE(Tenboro @ Oct 1 2016, 21:29) *
That's bait.
Ha!

But you're not catching me that easily (and I think I lost my old H@H cache btw).
User is online!Profile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 22:44
Post #10
Dideet1936



Newcomer
*
Group: Recruits
Posts: 11
Joined: 27-March 14
Level 24 (Apprentice)


There are a few new (or at least I don't recognise them) directories in the options now. Could you elaborate on them a little more?

Cache dir I'm guessing is where the static ranges live, data dir I assume is where the database about state and such goes, logs are logs of course and the download dir is where manual downloads must go.

What's the temp directory for? And am I correct about the others?
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 23:17
Post #11
Maximum_Joe



Smug Mod
***********
Group: Members
Posts: 19,623
Joined: 17-April 11
Level 489 (Godslayer)


QUOTE(Dideet1936 @ Oct 1 2016, 16:44) *

What's the temp directory for?

Some proxy files that come and go very quickly. You can reassign it to be the same directory as your log files if you want 1 less folder.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 23:26
Post #12
Simpleton8



Casual Poster
****
Group: Members
Posts: 314
Joined: 2-November 15
Level 258 (Godslayer)


QUOTE(Tenboro @ Oct 1 2016, 22:29) *

It is intentional, you'll have to turn that off to use the downloader.

Cool. I also noticed that the downloads are queue and can't be cancelled (w/o stopping H@H).
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 1 2016, 23:49
Post #13
Tenboro

Admin




QUOTE(Infinal @ Oct 1 2016, 22:43) *

Updating appears to have broken images being loaded through H@H while browsing the galleries, both on e-hentai.org and super secret website that forum censoring doesn’t want me to mention. What do?


In the READ BEFORE UPDATING section, it says: "1.3 and future versions of H@H will not support the proxy functionality. While this option will eventually be removed from the site, trying to use this on a 1.3 client will not work."

QUOTE(Simpleton8 @ Oct 1 2016, 23:26) *

Cool. I also noticed that the downloads are queue and can't be cancelled (w/o stopping H@H).


Currently there's no way to cancel downloads, stopping H@H will just make it pick it back up when you restart it. I'll probably make an actual queue interface at some point.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 00:52
Post #14
Infinal



Newcomer
**
Group: Members
Posts: 79
Joined: 7-April 13
Level 411 (Godslayer)


QUOTE(Tenboro @ Oct 1 2016, 23:49) *

In the READ BEFORE UPDATING section, it says: "1.3 and future versions of H@H will not support the proxy functionality. While this option will eventually be removed from the site, trying to use this on a 1.3 client will not work."

Oh, I thought this referred to the in-H@H proxy settings. My bad. :v
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 07:13
Post #15
Soul--Reaver



Lurker
Group: Lurkers
Posts: 4
Joined: 8-October 08
Level 184 (Lord)


Client works nice

Kinda missing the proxy but you probably have a reason for removing it

What is really annoying though is that the window size cannot be changed anymore. I would prefer maximizing the window so that i can see more log entries
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 14:39
Post #16
riesza



Newcomer
*
Group: Members
Posts: 23
Joined: 24-February 12
Level 51 (Expert)


Is this kind of multi resolution option planned for the normal archive downloader? The regular resizer is too small for me.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 15:05
Post #17
Tenboro

Admin




QUOTE(riesza @ Oct 2 2016, 14:39) *

Is this kind of multi resolution option planned for the normal archive downloader? The regular resizer is too small for me.


Creating and caching archives for all six resolution options isn't really feasible, unfortunately. The H@H Downloader works by downloading them more or less as a normal person browsing the site would, which is why we can support that.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 15:17
Post #18
Maximum_Joe



Smug Mod
***********
Group: Members
Posts: 19,623
Joined: 17-April 11
Level 489 (Godslayer)


Since the logpane is harder to copy and paste and newbies will inevitably want to do so... Just put an "Open Log File" in the Program dropdown menu.

Also maybe include a README file with the various error messages and what to do to resolve them. Basically an offline copy of the Technical Issues - H@H page.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 16:56
Post #19
Tenboro

Admin




QUOTE(Maximum_Joe @ Oct 2 2016, 15:17) *

Since the logpane is harder to copy and paste and newbies will inevitably want to do so... Just put an "Open Log File" in the Program dropdown menu.


Well, you'd need to open it in some kind of external program, which is fine for Windows since Notepad has been around since '95. Linux is tricky since there's no such thing as a "standard" GUI text editor, and I have no idea what they use in OS X.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Oct 2 2016, 17:11
Post #20
Maximum_Joe



Smug Mod
***********
Group: Members
Posts: 19,623
Joined: 17-April 11
Level 489 (Godslayer)


QUOTE(Tenboro @ Oct 2 2016, 10:56) *

I have no idea what they use in OS X.

[en.wikipedia.org] https://en.wikipedia.org/wiki/TextEdit
User is offlineProfile CardPM
Go to the top of the page
+Quote Post


6 Pages V  1 2 3 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 


Lo-Fi Version Time is now: 1st May 2017 - 00:24