[X] My Assistant

1.1.0 Release Notes

The major new feature of the 1.1 experimental branch is static ranges. If a client is assigned a static range, the server will always assume that the client is able to serve files from that range without tracking the files individually. Instead of being pre-distributed, if a file isn't currently cached, the client will proxy-request it from the image servers on-demand, then store it for later use.

Static ranges will only be assigned to faster clients that have a proven track record, and only if it has enough disk space set aside to store all the files for the static range - in other words, 1/65536th of all hosted images. Currently, the minimum requirement is 500 MB disk space per assigned range. A client can be assigned an arbitrary number of ranges depending on its capabilities.

Files in a static range will not be tracked by the server, nor will the client bug the server about it when it adds the file or in the unlikely case it decides to delete it. This should greatly reduce the server-side load of tracking clients. It will also allow us to greatly increase the file coverage per region.

This version of H@H will eventually be able to make advantage of a lot more disk space than before. So if you have loads of it just laying around, it doesn't hurt to bump it up a bit.

- Core: Support for static ranges was added.

- Core: The client will no longer terminate if it fails to set socket buffer sizes for any reason.

- Core: Some platforms didn't like the argument order we used for ping, so this was switched around.

- Core: Added user friendly interpretations for a few missing startup errors.

- Core: The stale file pruning mechanism will no longer run if the server hasn't been reachable lately.

- Core: HTTPServer should now recognize IPv6 localhost and private network ranges.

- Core: Removed support for pre-1.0.4 URL format.

- Core: The --use-more-memory mode is now the default. If this causes issues, you can revert to the old behavior with the new --use-less-memory flag.
-- Unless you have physical memory limitations, if you run into Out of Memory errors, you should try using -Xmx1g flag first, which will increase the maximum Java heap space to 1GB.

- Core: Fixed logging could throw a NullPointerException if a null somehow ended up in the final print.

- GUI: Workaround for Java 7 randomly failing to return the underlying text of the logging pane when using RDP.

- Downloader: Fixed cutoff for overly long filenames not being applied correctly. (Some poorly written operating systems fail the final file copy operation when total path length exceeds 255 characters.)


As the 1.1 branch is considered Beta, it's not added to the H@H client page. Download it here:

Runnable
Source

New patch for H@H?

I would like to know if this 'static ranges' is compulsory for clients? Cause AFAIK some of those H@H clients would like to keep more popular galleries, and I'm not sure if this would interfere with this 'static range' plan.

Seems this needs JRE 7 to run.

CODE

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/hath/base/HentaiAtHomeClient : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(Unknown Source)
        at java.lang.ClassLoader.defineClass(Unknown Source)
        at java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.access$000(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
Could not find the main class: org.hath.base.HentaiAtHomeClient. Program will exit.

Java 6 is EOL, but you should still be able to compile with it.

Edit: I rebuilt it with Java 6 compatibility, you should be able to run it with JRE 6 now if you redownload.

Cool. Updated mine.
Everything seems fine so far.

Edit:
Noticed there is quite a bit of flushing:
[i.imgur.com] http://i.imgur.com/P83ikf6.png

This post has been edited by Inferius: Oct 29 2013, 14:54

- Core: The --use-more-memory mode is now the default. If this causes issues, you can revert to the old behavior with the new --use-less-memory flag. -- Unless you have physical memory limitations, if you run into Out of Memory errors, you should try using -Xmx1g flag first, which will increase the maximum Java heap space to 1GB.

This will be worrying for little boxes, mine have 256 memory. Currently testing and --use-less-memory is doing alright...

About free space... i have lots of it on HDD's but i prefer to keep H@H running on my SSD. And i use Symbolic link on "downloads" folder to link it to folder on HDD, so my downloads not eat up free space on SSD.
Currently all cache files are put into "cache" folder with lot of sub-folders it in.
Can you make separate caches with varying cache sizes and use different folders for them so we can link different HDD?

For example:
Hot files cache for most requested files. Can be main one and on SSD.
Range cache
Archive cache #1-#infinity for everything else.

This post has been edited by Kahm: Oct 29 2013, 16:40

Noticed there is quite a bit of flushing:
[i.imgur.com] http://i.imgur.com/P83ikf6.png

imgur seems to be down as usual, but if you're talking about the "written bit" debug messages, those are normal.

This will be worrying for little boxes, mine have 256 memory. Currently testing and --use-less-memory is doing alright...

--use-less-memory reverts to the exact same behavior as before, so if it worked then, it should still work now.

Can you make separate caches with varying cache sizes and use different folders for them so we can link different HDD?

Multiple caches is on my maybe-list, but it probably wouldn't actively move files between slower and faster storage. Then again, a platter drive should be able to handle the cache directory easily even for fast clients, and you could manually symlink the subdirectories if you wanted to split it.

What's this?
2013-10-29T16:19:16Z [debug] Response Written:
2013-10-29T16:19:16Z [debug] An error has occurred. (400)

I notice that this happens an awful lot when running H@H.

2013-10-29T16:19:16Z [debug] Response Written:
2013-10-29T16:19:16Z [debug] An error has occurred. (400)

Isn't it betwwen a lot of 200s? H@H gets 400s when the person connecting closes the tab before the image finishes loading.

Isn't it betwwen a lot of 200s? H@H gets 400s when the person connecting closes the tab before the image finishes loading.

Well it is, but I've never seen it before this patch. There were some disconnects, but that was a different message.

Well it is, but I've never seen it before this patch. There were some disconnects, but that was a different message.

1.1.0 is BETA, and logging outputs therefore include debugging statements as well as a dump of "shorter" messages passed to the server and other clients. Which makes it quite a bit noisier than usual, but would help to track down any bugs in case of a fault.

Looks good at first glance.

Will the flush mechanism make room for static ranges when the maximum cache size is sufficient?

Will the flush mechanism make room for static ranges when the maximum cache size is sufficient?

Yup. They enter into the same mechanism, but they will essentially never be evicted.

- Core: Fixed logging could throw a NullPointerException if a null somehow ended up in the final print.

AFAIK, the only path that could produce this behavior could be fixed this way ("e" cannot be null, but "e.getMessage()" can):

CODE

--- src/org/hath/base/HTTPSession.java  2013-10-30 01:57:58.000000000 +0100
+++ src/org/hath/base/HTTPSession.fix.java      2013-10-30 02:25:16.000000000 +0100
@@ -231,7 +231,7 @@
                        }
                } catch(Exception e) {
                        Out.info(info + "The connection was interrupted or closed by the remote host.");
-                       Out.debug(e == null ? "(no exception)" : e.getMessage());
+                       Out.debug(e.getMessage() == null ? e.toString() : e.getMessage());
                } finally {
                        if(hpc != null) {
                                hpc.cleanup();

- Downloader: Fixed cutoff for overly long filenames not being applied correctly. (Some poorly written operating systems fail the final file copy operation when total path length exceeds 255 characters.)

Doesn't seem to work

CODE

2013-10-30T03:27:45Z [ERR] {org.hath.base.FileTools.copy(FileTools.java:115)} java.io.FileNotFoundException: downloaded\[Crimson Comics    (Carmine)] Rancor Enkon (BLEACH) [15912]\bleach_enkon_002.jpg (The filename, directory name, or volume label syntax is incorrect)
2013-10-30T03:27:45Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)} java.lang.NullPointerException
2013-10-30T03:27:45Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.FileTools.copy(FileTools.java:122)
2013-10-30T03:27:45Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.GalleryFile.attemptDownload(GalleryFile.java:135)
2013-10-30T03:27:45Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.Gallery.galleryPass(Gallery.java:76)
2013-10-30T03:27:45Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.GalleryDownloadManager.run(GalleryDownloadManager.java:188)
2013-10-30T03:27:46Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at java.lang.Thread.run(Unknown Source)

CODE

2013-10-30T01:43:55Z [ERR] {org.hath.base.FileTools.copy(FileTools.java:115)} java.io.FileNotFoundException: downloaded\(C73) [Big Boss (Hontai Bai)] Negi-sensei to Himitsu no School Mizugi (Mahou Sensei Negima!    ) [En... [142]\001.jpg (The filename, directory name, or volume label syntax is incorrect)
2013-10-30T01:43:55Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)} java.lang.NullPointerException
2013-10-30T01:43:55Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.FileTools.copy(FileTools.java:122)
2013-10-30T01:43:55Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.GalleryFile.attemptDownload(GalleryFile.java:135)
2013-10-30T01:43:55Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.Gallery.galleryPass(Gallery.java:76)
2013-10-30T01:43:55Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at org.hath.base.GalleryDownloadManager.run(GalleryDownloadManager.java:188)
2013-10-30T01:43:55Z [ERR] {java.lang.Throwable$WrappedPrintStream.println(Unknown Source)}     at java.lang.Thread.run(Unknown Source)

Doesn't seem to work

Where are you running H@H from? I'd recommend just making it a directory on root (as in c:\hath), as it doesn't dynamically adjust the cutoff, it just assumes there are at least 185 characters available.

There are times like this where I miss running H@H.

Where are you running H@H from? I'd recommend just making it a directory on root (as in c:\hath), as it doesn't dynamically adjust the cutoff, it just assumes there are at least 185 characters available.

Java supports long paths (~32727 characters) on Windows NT since Java5. The problem is different here: there's a TAB character in the title of both galleries, and since this is a control character, it's forbidden in Windows file names. Thus, "FileTools.checkAndCreateDir()" fails (silently, as the result of "File.mkdirs()" isn't checked).

This post has been edited by Hairs Fan: Oct 30 2013, 21:57

Java supports long paths (~32727 characters) on Windows NT since Java5.

The long answer is, "it's more complex than that". If you were running H@H off FAT32, there's a hard limit in the file system itself. Nor does Explorer work properly beyond 255 characters in any case. I haven't properly tested to make it's not an issue in NTFS, even assuming that you don't give a crap about actually *viewing* the files.

The problem is different here: there's a TAB character in the title of both galleries, and since this is a control character, it's forbidden in Windows file names. Thus, "FileTools.checkAndCreateDir()" fails (silently, as the result of "File.mkdirs()" isn't checked).

Those are supposed to be filtered, but I guess I'll make a note to check it.