Welcome Guest ( Log In | Register )

5 Pages V  1 2 3 > »   
Closed TopicStart new topic
> Hentai@Home 1.2.0, Beta Wrap-Up

 
post Dec 20 2013, 22:19
Post #1
Tenboro

Admin




Since the changes introduced in the 1.1 branch seem to be stable, we're pushing 1.1.3 BETA as 1.2.0 Stable as is. The only real change between these builds is that 1.2.0 does not print debug output to the console per default.

As part of this, the minimum usable version was increased to r69 (1.0.10). The 14 people with older clients will at least have to update to this.

To sum up the changes from 1.0.10 to 1.2.0:

***

The major new feature for 1.2 is static ranges. If a client is assigned a static range, the server will always assume that the client is able to serve files from that range without tracking the files individually. Instead of being pre-distributed, if a file isn't currently cached, the client will proxy-request it from the image servers on-demand, then store it for later use.

Files in a static range will not be tracked by the server, nor will the client bug the server about it when it adds the file or in the unlikely case it decides to delete it. This should greatly reduce the server-side load of tracking clients. It will also allow us to greatly increase the file coverage per region.

Static ranges will only be assigned to faster clients that have a proven track record, and only if it has enough disk space set aside to store all the files for the static range - in other words, 1/65536th of all hosted images. Currently, the minimum requirement is 250 MB disk space and 3 KB/s effective speed per assigned range. A client can be assigned an arbitrary number of ranges depending on its capabilities. Note that static ranges will not be assigned unless the client has been running for 24 hours, and it will only be assigned one range every two hours at most.

A completely new client-to-client speed tester has also been added. This one will be able to detect higher speeds much more reliably than before.

This version of H@H will be able to make advantage of a lot more disk space than before. So if you have loads of it just laying around, it doesn't hurt to bump it up a bit.

- Core: Support for static ranges was added.

- Core: The client will no longer terminate if it fails to set socket buffer sizes for any reason.

- Core: Some platforms didn't like the argument order we used for ping, so this was switched around. (Only relevant for testing against pre-1.1.1 clients.)

- Core: Added user friendly interpretations for a few missing startup errors.

- Core: The stale file pruning mechanism will no longer run if the server hasn't been reachable lately.

- Core: HTTPServer should now recognize IPv6 localhost and private network ranges.

- Core: Removed support for pre-1.0.4 URL format.

- Core: The --use-more-memory mode is now the default. If this causes issues, you can revert to the old behavior with the new --use-less-memory flag.
-- Unless you have physical memory limitations, if you run into Out of Memory errors, you should try using -Xmx1g flag first, which will increase the maximum Java heap space to 1GB.

- Core: Fixed logging could throw a NullPointerException if a null somehow ended up in the final print.

- Core: Added support for multi-threaded client-to-client testing.

- Core: The standard file download mechanism has been rewritten, primarily to support the new testing system. As a bonus, the new one uses significantly less CPU than the old downloader.

- Core: The database schema was altered to remove the unused strlen field. Note that the existing database will automatically be updated on first startup, after which you cannot downgrade without deleting the database.

- Core: The client should now be able to understand requests that use encoded equal signs (=) in the URL.

- Core: The client will now enforce available cache size, and will therefore no longer start if the setting for the cache size is larger than the available disk space minus the total size of the files in the cache. (You can adjust this from the web interface.)

- Core: Added a safety check to prevent the client from starting if it has static ranges assigned but an empty cache, which would indicate some sort of error. (To start a client that has lost its cache, you have to manually reset the static ranges from the web interface.)

- GUI: Workaround for Java 7 randomly failing to return the underlying text of the logging pane when using RDP.

- Downloader: Fixed cutoff for overly long filenames not being applied correctly. (Some poorly written operating systems fail the final file copy operation when total path length exceeds 255 characters.)

- Downloader: HathDL files with gallery names containing tab characters will now be filtered and handled gracefully, no longer causing the downloader to terminate.

- Web Interface: A checkbox for disabling the client-side speed limit was added to the on-site Hentai@Home settings page. This has the same effect as starting the client with the --disable-bwm flag.

- Web Interface: The limit on how much disk space you can assign has been removed.

- Web Interface: Added an option to reset static ranges, for those cases where the cache has been lost for some reason or another.

- Web Interface: Instead of having it as a warning, the interface screen will now simply refuse to change a client's port or key while it is running.

- Dispatcher: The trust mechanics was tweaked to take static ranges into account. Depending on the number of assigned ranges and frequency of requests, a request for a static range file can now cause a slight reduction in trust. The effect should be very minor for well-behaving clients, and is solely to prevent clients with frequent cache wipes from having ranges assigned.


Download from the usual place.

For information on how to join Hentai@Home, check out The Hentai@Home Project FAQ.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 20 2013, 22:25
Post #2
Maximum_Joe



Legendary Poster
***********
Group: Gold Star Club
Posts: 24,074
Joined: 17-April 11
Level 500 (Dovahkiin)


Multi-transfer support @ 1.3.0 then?

Oh and the copyright still says 2008-2012.

This post has been edited by Maximum_Joe: Dec 20 2013, 22:30
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 20 2013, 22:55
Post #3
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,046
Joined: 24-March 12
Level 500 (Godslayer)


QUOTE
- Core: HTTPServer should now recognize IPv6 localhost and private network ranges.
yay
QUOTE
- Web Interface: A checkbox for disabling the client-side speed limit was added to the on-site Hentai@Home settings page. This has the same effect as starting the client with the --disable-bwm flag.
yay

These make my life easier.

Btw Tenboro, this is not related to the new release as it happened on H@H 1.0.10, yet I think it's worth taking note. Some time ago I helped a guy renting a seedbox (Linux) where he runs 2 H@H clients on 2 different IPs and he managed to make them work in an interesting way: he setup 2 different users, say, "hath1" and "hath2" and one group "hath" which both users are members. He used iptables to assign the different IPs to each user. Then:

1. as user "hath1" he copied h@h files to a home directory of hath1, say, /home/hath1/client and started h@h.
2. he chmod that directory ( /home/hath1/client ) and all files within to 770
3. then he started h@h from the same directory ( /home/hath1/client ) as user "hath2"

Amazingly this works pretty well! Both clients manage to boot and use the same cache. Unfortunatelly this defeats half of the purpose of having two clients, which is having more cache. It also corrupts the database after some time, yet it can run weeks without problems.

I am worried that with static ranges such setup would be devstating to the files within the range. It might be a good idea to prevent such setup: something like a lock file with a timestamp inside while accessing the DB.

(We changed the setup, with this guy in question, to the standard way of dealing with two clients: each user maintaining it's own cache. Therefore his clients are doing well. But someone might try this setup by mistake in the future)

This post has been edited by blue penguin: Dec 20 2013, 23:00
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 20 2013, 23:31
Post #4
Tenboro

Admin




QUOTE(blue penguin @ Dec 20 2013, 21:55) *
Amazingly this works pretty well! Both clients manage to boot and use the same cache. Unfortunatelly this defeats half of the purpose of having two clients, which is having more cache. It also corrupts the database after some time, yet it can run weeks without problems.

I am worried that with static ranges such setup would be devstating to the files within the range. It might be a good idea to prevent such setup: something like a lock file with a timestamp inside while accessing the DB.


You *really* don't want to run with something like that, since Client 1 won't know when Client 2 decides to nuke a file and vice versa, and if they happen to fetch the same file, one will be prevented from doing so since the file exists and is owned by the other client. But I'm not really sure if it would accomplish anything to put in some kind of lock against it, as if you're trying to do that, you're probably capable of editing out that check from the code.

Since 1.2, if you're running a client with static ranges and it doesn't actually have as much disk space as the system is told, it will take a massive trust hit pretty soon.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 21 2013, 00:36
Post #5
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,046
Joined: 24-March 12
Level 500 (Godslayer)


QUOTE(Tenboro @ Dec 20 2013, 21:31) *
But I'm not really sure if it would accomplish anything to put in some kind of lock against it, as if you're trying to do that, you're probably capable of editing out that check from the code.

I guess you're right. Although the guy I was helping (I promised to not mention his username when talking about this after he bribed me with some HV gifts (IMG:[invalid] style_emoticons/default/smile.gif) ) did this just by googling around for unix shell commads and by reading old threads, i.e. he really ended with this setup by mere chance. Then again, that's a one in a million case to end with this setup by chance/mistake, so not worth changing code.

Note to self: I need to find time to upgrade my H@H this weekend I'm still running on 1.1.1.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 21 2013, 15:12
Post #6
Amaduyu Mitsumi



Active Poster
*******
Group: Gold Star Club
Posts: 1,609
Joined: 2-October 11
Level 500 (Godslayer)


I don't really understand an aspect of static ranges.

Let's say I'm setting aside 5GB for my home client's cache, all of which has been used up to store to be served images. If this client gets assigned a static range, will some of the older images be actively purged to make space for the range, or will it be denied of static range altogether?


QUOTE(Tenboro @ Dec 21 2013, 03:19) *

Currently, the minimum requirement is 250 MB disk space and 3 KB/s effective speed per assigned range.

Statistically, how long will it take for a slow client (75KB/s) to get a range? I kept my H@H up for more than 4 days, still not a single range assigned.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 21 2013, 15:19
Post #7
Tenboro

Admin




QUOTE(Kagoromo @ Dec 21 2013, 14:12) *
Let's say I'm setting aside 5GB for my home client's cache, all of which has been used up to store to be served images. If this client gets assigned a static range, will some of the older images be actively purged to make space for the range, or will it be denied of static range altogether?


Space is shared between static ranges and dynamically assigned files, but the former is prioritized. The space required per range does take this into account, so it won't be stuffed to the brink with static range files.

QUOTE(Kagoromo @ Dec 21 2013, 14:12) *
Statistically, how long will it take for a slow client (75KB/s) to get a range? I kept my H@H up for more than 4 days, still not a single range assigned.


Statistically never, since only faster clients are assigned static ranges. The absolute minimum right now is 100 KB/s, and will probably increase to 200 KB/s.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 21 2013, 15:25
Post #8
Amaduyu Mitsumi



Active Poster
*******
Group: Gold Star Club
Posts: 1,609
Joined: 2-October 11
Level 500 (Godslayer)


QUOTE(Tenboro @ Dec 21 2013, 20:19) *

Statistically never, since only faster clients are assigned static ranges. The absolute minimum right now is 100 KB/s.

Got it. (IMG:[invalid] style_emoticons/default/tongue.gif)
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 22 2013, 03:52
Post #9
brand00



Lurker
Group: Lurkers
Posts: 1
Joined: 3-August 11


Thank for remind~~~
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 22 2013, 13:58
Post #10
Owyn



Casual Fapper
*****
Group: Members
Posts: 692
Joined: 12-May 10
Level 354 (Godslayer)


A quick question:
- What happens when cache reaches maximum? Will I still be getting new files? Will old barely used or uploaded files be erased from cache sometime?

P.S. - I like how I'm getting max speed 9764 KB/s with new H@H now (IMG:[invalid] style_emoticons/default/happy.gif)
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 22 2013, 14:01
Post #11
Tenboro

Admin




QUOTE(Owyn @ Dec 22 2013, 12:58) *
- What happens when cache reaches maximum? Will I still be getting new files? Will old barely used or uploaded files be erased from cache sometime?


When the cache is full, it uses a Least Recently Used mechanism to remove older files in order to make room for new ones.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 26 2013, 15:22
Post #12
pisarenkoSA



Newcomer
*
Group: Members
Posts: 31
Joined: 17-December 13


after downloading can not connect ... though all the settings are correct ..
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 26 2013, 22:28
Post #13
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,046
Joined: 24-March 12
Level 500 (Godslayer)


QUOTE(pisarenkoSA @ Dec 26 2013, 13:22) *
after downloading can not connect ... though all the settings are correct ..
digress please...
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 27 2013, 05:19
Post #14
Vulpix



Newcomer
*
Group: Recruits
Posts: 15
Joined: 4-September 08
Level 12 (Novice)


Heya!

I have a question.

I've recently experienced storage malfunction (I have 23 static ranges assigned to me).

I started the game with verify cache and I could see that some files got corrupted:

QUOTE

2013-12-27T03:04:28Z [info] CacheHandler: Database initialized
2013-12-27T03:04:28Z [info] CacheHandler: Initializing the cache system...
2013-12-27T03:04:28Z [info] CacheHandler: A full cache verification has been requested. This can take quite some time.
2013-12-27T03:04:28Z [info] CacheHandler: Loading cache.. (this could take a while)
2013-12-27T03:04:56Z [info] CacheHandler: Loaded 1000 files so far...
2013-12-27T03:04:58Z [WARN] CacheHandler: The file cache\0b\0b1bcd509b3054982b50d7cefa6b2c5a4c637ded-825571-1191-1703-jpg was corrupt. It is now deleted.
2013-12-27T03:05:15Z [WARN] CacheHandler: The file cache\10\10cf2a5c59c5442132018bc5f20d970255f2b78c-442902-1200-1565-jpg was corrupt. It is now deleted.
2013-12-27T03:05:15Z [WARN] CacheHandler: The file cache\10\10cfa9e08783b0b861cb21aa421efa7fb5ea91a4-91285-600-800-jpg was corrupt. It is now deleted.
2013-12-27T03:05:23Z [info] CacheHandler: Loaded 2000 files so far...
2013-12-27T03:05:48Z [info] CacheHandler: Loaded 3000 files so far...
2013-12-27T03:06:14Z [info] CacheHandler: Loaded 4000 files so far...
2013-12-27T03:06:18Z [WARN] CacheHandler: The file cache\27\27af2487f4abe067d899b200b7920057e564a290-654881-1280-960-jpg was corrupt. It is now deleted.
2013-12-27T03:06:26Z [WARN] CacheHandler: The file cache\29\29dc9b1122398a97baee77b1b5c9f61f238abb52-718983-1200-1721-jpg was corrupt. It is now deleted.
2013-12-27T03:06:40Z [info] CacheHandler: Loaded 5000 files so far...
2013-12-27T03:07:15Z [info] CacheHandler: Loaded 6000 files so far...
2013-12-27T03:07:34Z [WARN] CacheHandler: The file cache\39\39bd58231cbae6ee790128310742a47f3408350f-38495-400-328-jpg was corrupt. It is now deleted.
2013-12-27T03:07:42Z [info] CacheHandler: Loaded 7000 files so far...
2013-12-27T03:08:11Z [info] CacheHandler: Loaded 8000 files so far...
2013-12-27T03:08:36Z [info] CacheHandler: Loaded 9000 files so far...
2013-12-27T03:08:38Z [WARN] CacheHandler: The file cache\49\499e7443b84bce8797d6d9d5c0bfe27913c17732-396643-974-1400-jpg was corrupt. It is now deleted.
2013-12-27T03:08:38Z [WARN] CacheHandler: The file cache\49\499ef97541735a9c183c7b9d48dfecd10f7cd736-582750-1200-1777-jpg was corrupt. It is now deleted.
2013-12-27T03:09:08Z [info] CacheHandler: Loaded 10000 files so far...
2013-12-27T03:09:35Z [info] CacheHandler: Loaded 11000 files so far...
2013-12-27T03:10:00Z [info] CacheHandler: Loaded 12000 files so far...
2013-12-27T03:10:14Z [WARN] CacheHandler: The file cache\5f\5f7f232026c46794b31f9bbee358df4e89116bbb-86327-512-384-jpg was corrupt. It is now deleted.
2013-12-27T03:10:15Z [WARN] CacheHandler: The file cache\5f\5f7f548464de6dc448277a12f29e1fd64b479ab4-53463-600-499-jpg was corrupt. It is now deleted.
2013-12-27T03:10:26Z [info] CacheHandler: Loaded 13000 files so far...
2013-12-27T03:10:54Z [info] CacheHandler: Loaded 14000 files so far...
2013-12-27T03:11:20Z [info] CacheHandler: Loaded 15000 files so far...
2013-12-27T03:11:45Z [info] CacheHandler: Loaded 16000 files so far...
2013-12-27T03:12:05Z [WARN] CacheHandler: The file cache\79\798c5fb7dc809fb88c631ff9b3f78815975548cc-66050-800-640-jpg was corrupt. It is now deleted.
2013-12-27T03:12:10Z [info] CacheHandler: Loaded 17000 files so far...
2013-12-27T03:12:34Z [info] CacheHandler: Loaded 18000 files so far...
2013-12-27T03:13:01Z [info] CacheHandler: Loaded 19000 files so far...
2013-12-27T03:13:27Z [info] CacheHandler: Loaded 20000 files so far...
2013-12-27T03:13:52Z [info] CacheHandler: Loaded 21000 files so far...
2013-12-27T03:14:10Z [WARN] CacheHandler: The file cache\99\994e967a28bfef64804955d575a1657ffced2631-68886-800-600-jpg was corrupt. It is now deleted.
2013-12-27T03:14:10Z [WARN] CacheHandler: The file cache\99\994ea1eed22e3bad41aa02699b405ce34b635f91-67849-660-800-jpg was corrupt. It is now deleted.
2013-12-27T03:14:16Z [info] CacheHandler: Loaded 22000 files so far...
2013-12-27T03:14:45Z [info] CacheHandler: Loaded 23000 files so far...
2013-12-27T03:15:14Z [info] CacheHandler: Loaded 24000 files so far...
2013-12-27T03:15:26Z [WARN] CacheHandler: The file cache\aa\aa2f01ba46a735bbc17241441060d23bc8d7de81-70773-637-478-jpg was corrupt. It is now deleted.
2013-12-27T03:15:27Z [WARN] CacheHandler: The file cache\aa\aa2fa77af28f7712f2acd0c702d8a07695ed09f9-141477-1200-847-jpg was corrupt. It is now deleted.
2013-12-27T03:15:41Z [info] CacheHandler: Loaded 25000 files so far...
2013-12-27T03:16:01Z [WARN] CacheHandler: The file cache\b1\b1f5b7589b29c9689216453854803a53684978b1-682830-1269-1800-jpg was corrupt. It is now deleted.


How do I know if I should reset my static ranges? I didn't lose all of my cache. In fact, only bits and pieces here and there. Is there a way to check what got deleted so that I can decide whether to reset the ranges (or not) ?

Thanks!
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 27 2013, 11:10
Post #15
Tenboro

Admin




QUOTE(Vulpix @ Dec 27 2013, 04:19) *
How do I know if I should reset my static ranges? I didn't lose all of my cache. In fact, only bits and pieces here and there. Is there a way to check what got deleted so that I can decide whether to reset the ranges (or not) ?


A few files lost doesn't matter.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 29 2013, 01:28
Post #16
mike23



Casual Poster
***
Group: Gold Star Club
Posts: 132
Joined: 23-August 07
Level 377 (Godslayer)


Can files cached through hathdl downloads push static range cache files out of the cache? I'm wondering if my trust is hurting because of how many hathdl files I'm downloading.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 29 2013, 14:26
Post #17
Tenboro

Admin




QUOTE(mike23 @ Dec 29 2013, 00:28) *
Can files cached through hathdl downloads push static range cache files out of the cache? I'm wondering if my trust is hurting because of how many hathdl files I'm downloading.


Well, in theory they could, but in practice it won't happen. When the client picks the least recently used files to evict, static range files get a bonus corresponding to ten years in the future.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 31 2013, 05:07
Post #18
Makaijin



Casual Poster
***
Group: Members
Posts: 104
Joined: 22-April 08
Level 406 (Godslayer)


A question on the use less memory flag.
the old "--use_more_memory" flag had underscores for spaces, but the patch notes have "--use-less-memory" as hyphens. So, should I use hyphens or underscores?
Reason I'm asking is my VPS only has 512 MB available (sucks but cheap) so the less memory flag is useful in my case.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Dec 31 2013, 08:28
Post #19
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,046
Joined: 24-March 12
Level 500 (Godslayer)


From the H@H code:
CODE
public static boolean updateSetting(String setting, String value) {
                setting = setting.replace("-", "_");

...

                        else if(setting.equals("use_less_memory")) {
                                useLessMemory = value.equals("true");
                        }
So it does not matter if you write the option with underscores or with hyphens (or with a combination of them), they will be understood the same.

(This is true for all H@H command line options)
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Jan 3 2014, 02:51
Post #20
Vulpix



Newcomer
*
Group: Recruits
Posts: 15
Joined: 4-September 08
Level 12 (Novice)


Out of curiosity - is it somehow possible to enable debug output for the quality system? I assume it's probably not possible seeing as it's tracked server-side; but it could help diagnosing issues not apparent at first sight.

If I have a stable / fast connection and yet lately my quality/trust hovers around 7000 / 700; it'd be great to know what is my client taking trust/quality hits for - so as to be able to remedy the situation.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post


5 Pages V  1 2 3 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 


Lo-Fi Version Time is now: 4th September 2025 - 19:36