[X] My Assistant

Updated one of my client to 1.3.3, went pretty smoothly.
Is it normal that it's now using 2-3GB more than before?

Of disk space, or bandwidth? Sure, as long as it's below the limit.

Of RAM? Depends if you're talking about virtual memory space or actual resident memory usage. If the former, it doesn't matter. If the latter, definitely no.

Grab a screenshot of the thing that's showing you the usage, and you'll get a more concise answer.

Of disk space, or bandwidth? Sure, as long as it's below the limit.

Of RAM? Depends if you're talking about virtual memory space or actual resident memory usage. If the former, it doesn't matter. If the latter, definitely no.

Grab a screenshot of the thing that's showing you the usage, and you'll get a more concise answer.

You're right.
I was talking about disk space.

I had 3.5GB free before, removed 15GB worth of static ranges during the update and I'm now with 15GB free.
Just wanted to check if that is normal because of the way it handles the cache or if it shouldn't have happened.

I had 3.5GB free before, removed 15GB worth of static ranges during the update and I'm now with 15GB free.
Just wanted to check if that is normal because of the way it handles the cache or if it shouldn't have happened.

Deleting files outside of the assigned static ranges is intended behavior, yes. They wouldn't have been used, so they would just be taking up space, bloat the directory tree, and clog up the pruning mechanism.

- (Server) Brand new clients now start at 1500 historic/lomark/himark quality instead of 0, meaning that they'll get going faster than before.

What does that mean?



My client has been created 2016-09-21. I began with 1.2.6. and changed to 1.3 when it came out.

When running 1.2.6. my trust and quality was almost at max the whole time. But with 1.3 those stats had severely dropped. Trust now rarely reach 700 and quality varies now between 2000 - 5000. Static range at 201.

My client is constantely running and my hitrate stays at 2.2 and my hathrate stays at 3.3.

Is that normal?

- (Server) Brand new clients now start at 1500 historic/lomark/himark quality instead of 0, meaning that they'll get going faster than before.

What does that mean?

Only affects the initial value when the client is first created.

My client has been created 2016-09-21. I began with 1.2.6. and changed to 1.3 when it came out.

When running 1.2.6. my trust and quality was almost at max the whole time. But with 1.3 those stats had severely dropped. Trust now rarely reach 700 and quality varies now between 2000 - 5000. Static range at 201.

My client is constantely running and my hitrate stays at 2.2 and my hathrate stays at 3.3.

Is that normal?

Possibly. There's currently no significant difference in trust/quality for active pre-1.3 and 1.3+ clients, but the quality/trust of yours seems to be fluctuating quite a bit. That indicates your connection or ISP might be having some intermittent issue that degrades your client.

I know I'm not a techie, but can I ask for a more specific guide to upgrade to H@H v1.3? I'm running it on a Windows RDP and Java SDK 8.

I tried replacing the files with those inside the 1.3.x archive and it seems that it doesn't work. I tried opening the GUI file and the version number in the window title is still the same.

I'm sure I'm doing something wrong so I ask to be enlightened about this.

EDIT: My bad. It worked in second try after a reboot. Could just have been some bug.

This post has been edited by TennoujiSaburou: Oct 14 2016, 07:18

sorry im new here when i see how you do the speed improvement i can't stand,did you try


CacheHandler.java
startupInitCache()
delete this and all inside this

CODE

// cache register pass
        for(File l1dir : Tools.listSortedFiles(cachedir)) {

use this in the same place of before code

CODE

    
        walkFile dw = new walkFile();
        dw.recentlyAccessedCutoff = recentlyAccessedCutoff;
        dw.validator = validator;
        dw.oldestLastModified = oldestLastModified;
        dw.printFreq = printFreq;
        
        Files.walkFileTree(cachedir.toPath(), EnumSet.of(FileVisitOption.FOLLOW_LINKS), 3, dw);
        
        oldestStaticRangeElement = (String) dw.oldestStaticRangeElement;

CacheHandler.java

CODE

    private class walkFile extends SimpleFileVisitor<Path> {
        long oldestLastModified = 0;
        long fileLastModified = 0;
        int printFreq = 0;
        Path cfile = null;
        private FileValidator validator;
        private long recentlyAccessedCutoff;
        private Object oldestStaticRangeElement;
        
        public FileVisitResult visitFile(Path cfile, BasicFileAttributes attr) throws IOException {
            HVFile hvFile = HVFile.getHVFileExistsFile(cfile, attr, validator);

            if (hvFile == null) {
                Out.debug("CacheHandler: The file " + cfile + " was corrupt.");
                Files.delete(cfile);
            } else if (!Settings.isStaticRange(hvFile.getFileid())) {
                Out.debug("CacheHandler: The file " + cfile + " was not in an active static range.");
                Files.delete(cfile);
            } else {
                addFileToActiveCache(hvFile);
                fileLastModified = attr.lastModifiedTime().toMillis();

                if (fileLastModified > recentlyAccessedCutoff) {
                    // if lastModified is from the last week, mark this
                    // as recently accessed in the LRU cache. (this does
                    // not update the metadata)
                    markRecentlyAccessed(hvFile, true);
                }

            }

            if (fileLastModified < oldestLastModified) {
                // cache this as the static range directory with the
                // currently oldest file. this is used whenever we need
                // to free up some space, to avoid having to start over
                // from the first range after every restart.
                oldestLastModified = fileLastModified;
                oldestStaticRangeElement = cfile.getFileName().toString().substring(0, 4);
            }

            if (cacheCount % printFreq == 0) {
                Out.info("CacheHandler: Loaded " + cacheCount + " files so far...");
            }

            return FileVisitResult.CONTINUE;
        }
    }

HVFile.java

CODE

public static HVFile getHVFileExistsFile(Path file, BasicFileAttributes attr, FileValidator validator) {
            String fileid = file.getFileName().toString();

            try {
                HVFile hvFile = getHVFileFromFileid(fileid);
                
                if(attr.size() != hvFile.getSize()) {
                    return null;
                }
                
                if(validator != null) {
                    if(!validator.validateFile(file, fileid.substring(0, 40))) {
                        return null;
                    }
                }
                
                return hvFile;
            }
            catch(java.io.IOException e) {
                e.printStackTrace();
                Out.warning("Warning: Encountered IO error computing the hash value of " + file);
            }
        
        return null;
    }
    

h@h 3.0.1 start in 2second with that code. It has taking 8 minute before.

This post has been edited by zbxehzqn: Oct 14 2016, 10:17

sorry im new here when i see how you do the speed improvement i can't stand,did you try

h@h 3.0.1 start in 2second with that code. It has taking 8 minute before.

That's quite similar to what someone else suggested, and has the same problem: you're not pruning empty directories. And if you try to do that, it'll likely be much slower than today's version.

It's not a big deal anymore since that code will only be run after abnormal shutdowns, so I probably won't spend any more time improving it.

this?

CODE

    private class walkFile extends SimpleFileVisitor<Path> {
        long oldestLastModified = 0;
        long fileLastModified = 0;
        int printFreq = 0;
        Path cfile = null;
        private FileValidator validator;
        private long recentlyAccessedCutoff;
        private Object oldestStaticRangeElement;
        
        private LinkedList<String> dirs = new LinkedList<>();
        private HashMap<String, Boolean> count = new HashMap<>(5, 1);
        
        @Override
        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
            String dirName = dir.getFileName().toString();
            count.put(dirName, false);
            dirs.addLast(dirName);
            
            return FileVisitResult.CONTINUE;
        }
        
        public FileVisitResult visitFile(Path cfile, BasicFileAttributes attr) throws IOException {
            
            for(String dirname : dirs){
                count.put(dirname, true);
            }
            
            HVFile hvFile = HVFile.getHVFileExistsFile(cfile, attr, validator);

            if (hvFile == null) {
                Out.debug("CacheHandler: The file " + cfile + " was corrupt.");
                Files.delete(cfile);
            } else if (!Settings.isStaticRange(hvFile.getFileid())) {
                Out.debug("CacheHandler: The file " + cfile + " was not in an active static range.");
                Files.delete(cfile);
            } else {
                addFileToActiveCache(hvFile);
                fileLastModified = attr.lastModifiedTime().toMillis();

                if (fileLastModified > recentlyAccessedCutoff) {
                    // if lastModified is from the last week, mark this
                    // as recently accessed in the LRU cache. (this does
                    // not update the metadata)
                    markRecentlyAccessed(hvFile, true);
                }

            }

            if (fileLastModified < oldestLastModified) {
                // cache this as the static range directory with the
                // currently oldest file. this is used whenever we need
                // to free up some space, to avoid having to start over
                // from the first range after every restart.
                oldestLastModified = fileLastModified;
                oldestStaticRangeElement = cfile.getFileName().toString().substring(0, 4);
            }

            if (cacheCount % printFreq == 0) {
                Out.info("CacheHandler: Loaded " + cacheCount + " files so far...");
            }

            return FileVisitResult.CONTINUE;
        }
        
        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException attr) throws IOException {
            
            String visited = dirs.removeLast();
            boolean hasFiles = count.get(visited);
            if(!hasFiles){
                Files.delete(dir);
            }
            
            return FileVisitResult.CONTINUE;
        }
    }

This post has been edited by zbxehzqn: Oct 14 2016, 19:47

this?

Hmm. Yes, that's definitely a lot more clever than the last attempt I saw at it, I just might use a variant of that. But you seem to have based off an earlier build than the current (1.3.3), so it would need some adaptation.

i can make that based off 1.3.3. you will use it?
i am doing without test. i cant make it start.
-
"pcache_" can not be there with thiss changes?

This post has been edited by zbxehzqn: Oct 15 2016, 11:44

CODE

new HashMap<>(5, 1);

You certainly NEVER want to set the load factor to 1.0. Otherwise you could just use a vector instead and iterate. Just keep it at the default 0.75, so that hash collisions are properly mitigated.

Not that more than 3k hash table entries are to be expected, but still.

ok. i will try something else.

foobar20324 this?

CODE

    private class walkFile extends SimpleFileVisitor<Path> {
        long oldestLastModified = 0;
        long fileLastModified = 0;
        int printFreq = 0;
        Path cfile = null;
        private FileValidator validator;
        private long recentlyAccessedCutoff;
        private Object oldestStaticRangeElement;
        
        private LinkedList<DirPath> dirs = new LinkedList<>();
        
        class DirPath{
            String name;
            boolean empty = true;
        }
        
        @Override
        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
            DirPath dirPath = new DirPath();
            dirPath.name = dir.getFileName().toString();
            dirs.addLast(dirPath);
            
            return FileVisitResult.CONTINUE;
        }
        
        public FileVisitResult visitFile(Path cfile, BasicFileAttributes attr) throws IOException {
            
            
            HVFile hvFile = HVFile.getHVFileExistsFile(cfile, attr, validator);

            if (hvFile == null) {
                Out.debug("CacheHandler: The file " + cfile + " was corrupt.");
                Files.delete(cfile);
            } else if (!Settings.isStaticRange(hvFile.getFileid())) {
                Out.debug("CacheHandler: The file " + cfile + " was not in an active static range.");
                Files.delete(cfile);
            } else {
                addFileToActiveCache(hvFile);
                fileLastModified = attr.lastModifiedTime().toMillis();

                if (fileLastModified > recentlyAccessedCutoff) {
                    // if lastModified is from the last week, mark this
                    // as recently accessed in the LRU cache. (this does
                    // not update the metadata)
                    markRecentlyAccessed(hvFile, true);
                }
                // dir has files
                for(DirPath dir : dirs){
                    dir.empty = false;
                }
            }

            if (fileLastModified < oldestLastModified) {
                // cache this as the static range directory with the
                // currently oldest file. this is used whenever we need
                // to free up some space, to avoid having to start over
                // from the first range after every restart.
                oldestLastModified = fileLastModified;
                oldestStaticRangeElement = cfile.getFileName().toString().substring(0, 4);
            }

            if (cacheCount % printFreq == 0) {
                Out.info("CacheHandler: Loaded " + cacheCount + " files so far...");
            }

            return FileVisitResult.CONTINUE;
        }
        
        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException attr) throws IOException {
            
            DirPath visited = dirs.removeLast();
            if(visited.empty){
                Files.delete(dir);
            }
            
            return FileVisitResult.CONTINUE;
        }
    }

Not what I meant (just that if you are using a HashMap, don't try to be "smart" by overriding the defaults), but it sure works.

Except that now, you don't even need that list either. A plain boolean is enough. Reset in pre-visit, set on file visit, check in post-visit. Done.

zbxehzqn I think these should be in the else block above

CODE

if (fileLastModified < oldestLastModified) {
...
}
if (cacheCount % printFreq == 0) {
...
}

foobar20324 you mean like this?

in startupInitCache()

CODE

            final WalkFile dw = new WalkFile(validator, printFreq, recentlyAccessedCutoff, oldestLastModified);
            Files.walkFileTree(cachedir.toPath(), dw);
            oldestStaticRangeElement = dw.getOldestStaticRangeElement();

CacheHandler.java

CODE

    private class WalkFile extends SimpleFileVisitor<Path>{
        private final FileValidator validator;
        private final int printFreq;
        private final long recentlyAccessedCutoff;

        private long oldestLastModified;
        private boolean deleteDir;
        private String oldestStaticRangeElement;

        private WalkFile(FileValidator validator, int printFreq, long recentlyAccessedCutoff, long oldestLastModified){
            this.validator = validator;
            this.printFreq = printFreq;
            this.recentlyAccessedCutoff = recentlyAccessedCutoff;
            this.oldestLastModified = oldestLastModified;
        }

        @Override
        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException{
            deleteDir = true;
            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult visitFile(Path cFile, BasicFileAttributes attr) throws IOException{
            final HVFile hvFile = HVFile.getHVFileExistsFile(cFile, attr, validator);

            if(hvFile == null){
                Out.debug("CacheHandler: The file " + cFile + " was corrupt.");
                Files.delete(cFile);
            }else if(!Settings.isStaticRange(hvFile.getFileid())){
                Out.debug("CacheHandler: The file " + cFile + " was not in an active static range.");
                Files.delete(cFile);
            }else{
                addFileToActiveCache(hvFile);
                final long fileLastModified = attr.lastModifiedTime().toMillis();

                if(recentlyAccessedCutoff < fileLastModified){
                    // if lastModified is from the last week, mark this
                    // as recently accessed in the LRU cache. (this does
                    // not update the metadata)
                    markRecentlyAccessed(hvFile, true);
                }
                deleteDir = false;

                if(fileLastModified < oldestLastModified){
                    // cache this as the static range directory with the
                    // currently oldest file. this is used whenever we need
                    // to free up some space, to avoid having to start over
                    // from the first range after every restart.
                    oldestLastModified = fileLastModified;
                    oldestStaticRangeElement = hvFile.getStaticRange();
                }

                if(cacheCount % printFreq == 0){
                    Out.info("CacheHandler: Loaded " + cacheCount + " files so far...");
                }
            }

            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException attr) throws IOException{
            if(deleteDir)
                Files.delete(dir);
            return FileVisitResult.CONTINUE;
        }

        private String getOldestStaticRangeElement(){
            return oldestStaticRangeElement;
        }
    }

Tenboro, maybe make a GitHub page or at least a dedicated thread for code submission?

10 hours in, initial setup is only on cleanup pass 40%. I know it will be slow because the client is on an external HDD but thats even slower than a full cache verify (IMG:[invalid] style_emoticons/default/sad.gif)

Tenboro, maybe make a GitHub page or at least a dedicated thread for code submission?

If H@H were like most open source projects, it would have had a Github long ago. It is however not something I want to spend time managing, since H@H normally only gets updated when I need it to support new site functionality or to fix bugs. Posting code here, or in PMs, works just fine.

10 hours in, initial setup is only on cleanup pass 40%. I know it will be slow because the client is on an external HDD but thats even slower than a full cache verify (IMG:[invalid] style_emoticons/default/sad.gif)

For a large client, it has to move a lot of files around, which will indeed take some time for high-latency storage. That is just for the very first startup though, and it will continue where it ended if it were to be interrupted.

Hi,

so sorry for being a noob but even though I'm used to bit-torrent I never get to under them completely. So with bit-torrent, it's a direct peer2peer right? So basically, I need to to have files sitting on the PC running the torrent with tracker right? Not sure if I'm right.

Now with Hentai@Home, I don't need to have any files sitting in an Hentai@Home folders right? Basically with the client, E-hentai distribute loads of the server to the users with clients as a form of cache correct? Now, when people browse E-hentai Galleries, the images loaded on the webpage will be browse from the cache in Hentai@Home clients. So just like bit-torrent the server will randomly choose which client to cache from right?

Now to lets say to uploads JPEG files to E-Hentai Galleries, you still need to go to the gallries-website and upload directly to the web server correct? It's not like people can get those JPEG files directly from Hentai@Home << Now that would need a bit-torrent correct?

Sorry for all these questions, just want to clarify my understanding on the Hentai@Home client. I've been using the client for 2 days, it seems to be doing what it's supposed to do I guess since I've seen .jpg files going around on the client terminal and connect to the server blah blah. Total Files Sent and Received kept increasing. << This is correct right?

Also one last note, My Trust number kept increasing (Now it's like 1000), is it supposed to go down and be maintained at "0" ?

Thank You for your answer in advance