Welcome Guest ( Log In | Register )

6 Pages V  1 2 3 > »   
Closed TopicStart new topic
> New Feature: Resample Archives, Itty bitty

 
post Aug 5 2016, 18:09
Post #1
Tenboro

Admin


清く正しく


Resample Archives

The archiver can now also be used to download the 1280 horizontal resolution resamples that are used by default on the site. Initially, this is limited to galleries that have 25% or more resampled files at this resolution.

Because of these changes, clicking on the Archive Download button for an archive you have previously unlocked will now take you to the archive selector instead of straight to the archive link, unless you have selected Auto Select Original or Auto Select Resample under My Settings.


Retired H@H Downloader

Since the archiver is now capable of generating a download with resampled files, the H@H Downloader has been retired. The addition of static ranges has made that suboptimal, and with resampled archives it also no longer has a unique function. Therefore, it has instead been replaced with a free archiver quota. (See below)


Donation/H@H Archiver Quotas

The old free archiver downloads for donators was changed from a fixed number of archives to use a quota based on archiver size. This also allows us to use this system for people who run H@H, to replace the retired H@H Downloader system.

For donators, each free archive from the old system now instead adds +50MB/day to the archiver quota. Tiers above gold were changed from +5 (250 MB/day) to +10 (500 MB/day) per tier. (See; Donation Archiver Quota in the MFP section below)

For people who run H@H, the archiver quota is now increased by +1000 MB/day for running at least one qualifying client, in addition to +10 MB/day per total adjusted avghit. H@H clients will only qualify if they have been running for more than 24 hours and have a hitrate of at least 1. (See; H@H Archiver Quota in the MFP section below)

Like the old system, the "daily" quota is measured over a 7-day window. You can spend up to 7 days worth of quota in a day, but you would obviously then have to wait a week for it to cool down.


Math For Pedants

Whenever we change from one system to another, there is always a heated discussion about the exact numbers used, so I pasted in the math and reasoning used below.

Donation Archiver Quota:

Average archive size calculated from 77215 files for a total of 3775 GB cached on one archiver:
3775 * 1024 / 77215 = 50.06 MB

H@H Archiver Quota:

Average size per resampled 1280x file sampled from a million files = 199156 bytes
Old H@H downloader rate was limited by each individual client's average hitrate, at floor(1 + avghits/25) files per minute, capped to 5
This gives 199156 bytes per minute per avghits/25 for 10.94 MB/day per avghit, plus 273.5 MB/day per client
Hard cap per client was removed. We instead calculate a global adjusted avghits from the sum of avghits for all clients on the account, set as min(avghits, log(1.003 + (avghits / 200), 1.003))
This is effectively a soft cap, linear up to about 300 avghits, increasing to ~600 at 1000 avghits and ~800 at 2000 avghits.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 18:13
Post #2
rawrpies



Casual Poster
****
Group: Catgirl Camarilla
Posts: 341
Joined: 25-April 11
Level 460 (Godslayer)


Explains why I couldn't find the H@H download button anymore.

Any reason why the resampled archive isn't available for alot of galleries? (I'm assuming the servers are still updating or something)

And lastly, is there any way to mass download galleries without having to unzip every single one of them? I'm lazy, I know
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 18:19
Post #3
Tenboro

Admin


清く正しく


QUOTE(rawrpies @ Aug 5 2016, 18:13) *

Any reason why the resampled archive isn't available for alot of galleries? (I'm assuming the servers are still updating or something)


Assuming it does have >=25% resampled files at 1280x, it may not be available on a given gallery until it has refreshed its cache record, which should happen soon enough.

QUOTE(rawrpies @ Aug 5 2016, 18:13) *

And lastly, is there any way to mass download galleries without having to unzip every single one of them? I'm lazy, I know


I'm sure there are tools that let you unzip multiple files at a time.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 18:59
Post #4
preaction



Lurker
Group: Lurkers
Posts: 1
Joined: 30-April 12
Level 30 (Journeyman)


This is great, but I run the H@H on my home computer, and copy the hathdl files from all my other devices to that computer to have the archive downloaded when I get home. Is there a way to bring that back? Or are you planning on removing the feature from the H@H downloader?

The H@H downloader also seemed to not re-download individual images it already had, which saves bandwidth. But that's impossible with the archives, which increases the amount needed to download (and also the amount the archiver needs to upload).

Also, could the Archive download turn into a single button (one for resampled, one for original)? It might make the workflow easier for users.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 19:05
Post #5
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,043
Joined: 24-March 12
Level 500 (Godslayer)


QUOTE(Tenboro @ Aug 5 2016, 17:09) *
Resample Archives
YES, Yes, yes, YEs, YES!

Just a question, since
QUOTE
resampled archive isn't available for alot of galleries
I am not sure if we get the size of the resample archive. On the archiver page we have "Original Gallery Size: XXX MB", will we have an extra "Resampled Gallery Size: XXX MB" as well?
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 19:23
Post #6
Tenboro

Admin


清く正しく


QUOTE(preaction @ Aug 5 2016, 18:59) *

This is great, but I run the H@H on my home computer, and copy the hathdl files from all my other devices to that computer to have the archive downloaded when I get home. Is there a way to bring that back? Or are you planning on removing the feature from the H@H downloader?


You can copy the download link for the archive instead.

QUOTE(preaction @ Aug 5 2016, 18:59) *
The H@H downloader also seemed to not re-download individual images it already had, which saves bandwidth. But that's impossible with the archives, which increases the amount needed to download (and also the amount the archiver needs to upload).


Generally this would have a very low hit rate unless you had previously browsed the gallery with the proxy enabled, but neither of those functions work very well since H@H changed to primarily use static ranges.

QUOTE(preaction @ Aug 5 2016, 18:59) *
Also, could the Archive download turn into a single button (one for resampled, one for original)? It might make the workflow easier for users.


You can customize the behavior of the download button here. Starting either original or resample downloads automatically is one option, or it can auto-select it and give you the download link if you want to download it elsewhere.

QUOTE(blue penguin @ Aug 5 2016, 19:05) *

Just a question, sinceI am not sure if we get the size of the resample archive. On the archiver page we have "Original Gallery Size: XXX MB", will we have an extra "Resampled Gallery Size: XXX MB" as well?


I didn't include that line with the "new feature" post, but that's noted in the update log.

QUOTE
... the archiver is not currently able to estimate (or even guess roughly) how large the resampled archive will be. This may be changed in the future.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 19:46
Post #7
rodrick



Newcomer
*
Group: Recruits
Posts: 14
Joined: 17-April 11
Level 34 (Journeyman)


QUOTE(preaction @ Aug 5 2016, 18:59) *

This is great, but I run the H@H on my home computer, and copy the hathdl files from all my other devices to that computer to have the archive downloaded when I get home. Is there a way to bring that back? Or are you planning on removing the feature from the H@H downloader?

I also use it like that, as a download and forget simple and practical feature, so that everything is downloaded when I go back to home (IMG:[invalid] style_emoticons/default/sad.gif)

QUOTE(Tenboro @ Aug 5 2016, 19:23) *

You can copy the download link for the archive instead.

What do you mean? Where to put the download link? In a text file inside hathdl folder? Recreating a manual hathdl file?
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 20:03
Post #8
Tenboro

Admin


清く正しく


QUOTE(rodrick @ Aug 5 2016, 19:46) *

What do you mean? Where to put the download link? In a text file inside hathdl folder? Recreating a manual hathdl file?


In a browser, or wget, or a download manager (like [www.freedownloadmanager.org] Free Download Manager). Unless I'm misunderstanding something? It shouldn't be harder to copy a link than to copy a .hathdl file.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 20:17
Post #9
rodrick



Newcomer
*
Group: Recruits
Posts: 14
Joined: 17-April 11
Level 34 (Journeyman)


QUOTE(Tenboro @ Aug 5 2016, 20:03) *

In a browser, or wget, or a download manager (like [www.freedownloadmanager.org] Free Download Manager). Unless I'm misunderstanding something? It shouldn't be harder to copy a link than to copy a .hathdl file.

Oh OK, that's what you meant.
The thing is that, the hathdl was really practical, as my computer isn't running 24/7, contrary to my little server, in which I dumped directly my hathdl files and forgot about them, letting h@h download.
Kind of like the same use as a torrent file: you download it in a shared folder, and forget about it until downloaded...
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 20:22
Post #10
Tenboro

Admin


清く正しく


QUOTE(rodrick @ Aug 5 2016, 20:17) *

The thing is that, the hathdl was really practical, as my computer isn't running 24/7, contrary to my little server, in which I dumped directly my hathdl files and forgot about them, letting h@h download.
Kind of like the same use as a torrent file: you download it in a shared folder, and forget about it until downloaded...


I still don't get how that's better than dumping the archive link into a download manager on the same server.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 20:36
Post #11
rodrick



Newcomer
*
Group: Recruits
Posts: 14
Joined: 17-April 11
Level 34 (Journeyman)


QUOTE(Tenboro @ Aug 5 2016, 20:22) *

I still don't get how that's better than dumping the archive link into a download manager on the same server.

The difference is the practicality of doing it. Basically:
hathdl :
- click on the link
- download on the shared folder
- rinse and repeat.

link:
- click on the download option
- copy the link
- open the download web server or bash if lazy
- authenticate to the download web server or command line
- paste the link in opened web server or paste the right command
- rinse and repeat

This copy and paste thing, when done for a lot of file, is a real hassle, way simpler to just download a file somewhere and let it be downloaded...

Anyway, I'll adapt I guess, but that was a really neat feature that disappeared here.

This post has been edited by rodrick: Aug 5 2016, 20:37
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 21:01
Post #12
Tenboro

Admin


清く正しく


QUOTE(rodrick @ Aug 5 2016, 20:36) *

The difference is the practicality of doing it. Basically:
hathdl :
- click on the link
- download on the shared folder
- rinse and repeat.

link:
- click on the download option
- copy the link
- open the download web server or bash if lazy
- authenticate to the download web server or command line
- paste the link in opened web server or paste the right command
- rinse and repeat


I guess I can see how it would be better if you have very slow internet, but it's a quite specific use case when you actually want a small file that triggers a slow download of a larger file rather than just as easily downloading the file in the first place.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 21:27
Post #13
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,043
Joined: 24-March 12
Level 500 (Godslayer)


QUOTE(rodrick @ Aug 5 2016, 19:36) *
- authenticate to the download web server or command line

Once you get the link there is no extra authentication step. That's why there is a limited number of times you can use a link.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 22:15
Post #14
hzqr



Savagely Still
********
Group: Gold Star Club
Posts: 4,672
Joined: 13-May 09
Level 462 (Dovahkiin)


Can you add the size (or an approximation thereof) of the resampled archive on the download popup? Something like [i.imgur.com] this
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 22:33
Post #15
blue penguin



in umbra, igitur, pugnabimus
***********
Group: Gold Star Club
Posts: 10,043
Joined: 24-March 12
Level 500 (Godslayer)


QUOTE(tiap @ Aug 5 2016, 21:15) *
Can you add the size (or an approximation thereof) of the resampled archive on the download popup? Something like [i.imgur.com] this
Heh, I asked the same thing (IMG:[invalid] style_emoticons/default/smile.gif)
QUOTE
... the archiver is not currently able to estimate (or even guess roughly) how large the resampled archive will be. This may be changed in the future.


But that left me thinking. How do you estimate the GP price of the resampled archive?
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 5 2016, 22:47
Post #16
Tenboro

Admin


清く正しく


QUOTE(blue penguin @ Aug 5 2016, 22:33) *

But that left me thinking. How do you estimate the GP price of the resampled archive?


That's the rest of the sentence in question.

QUOTE
- Resample archives currently "cost" a fixed 75% of the original archive, since the archiver is not currently able to estimate (or even guess roughly) how large the resampled archive will be. This may be changed in the future.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 6 2016, 00:12
Post #17
Fudo Masamune



Passive Poster
*******
Group: Gold Star Club
Posts: 1,635
Joined: 2-February 10
Level 500 (Ponyslayer)


So.. Is there any way to see the remaining quota I have at a moment?
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 6 2016, 02:02
Post #18
Scremaz



A certain pervert. OT expert. Just dancing around in the game.
***********
Group: Gold Star Club
Posts: 24,287
Joined: 18-January 07
Level 500 (Ponyslayer)


QUOTE(rawrpies @ Aug 5 2016, 18:13) *

And lastly, is there any way to mass download galleries without having to unzip every single one of them? I'm lazy, I know

i'm sure WinRAR allows you to click on a series of archives and unzip all of them in a folder each. iirc Universal Extractor lets you do the same as well, so i'm pretty positive that pretty much all the most known archive softwares will work.
you can even use a [en.wikipedia.org] comic book reader like CDisplayEX to look at them without actually needing to unzip them.

if that's the case, note that changing their extensions from zip, rar, 7z, tar to respectively cbz, cbr, cb7, cbt will make them open by default on the comic viewer, yet it will keep the chance to open them in the archive software - and modify them, whenever possible.
ie: WinRAR will allow you to open both a cbr and a cb7, but you will be able to modify only a cbr.
7zip will still allow you to open both cbr and cb7, but you will be able to modify only a cb7
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 6 2016, 02:37
Post #19
WuvviDuvvi



Newcomer
**
Group: Members
Posts: 99
Joined: 21-February 15
Level 203 (Ascended)


QUOTE(blue penguin @ Aug 5 2016, 12:27) *
Once you get the link there is no extra authentication step. That's why there is a limited number of times you can use a link.

He's not talking about authing on E-Hentai, he's talking about authing to his own server running H@H.

I get what rodrick and others are complaining about, but it'll probably be better for everyone to create some clever & convenient tool that bridges the archiver and the user's server rather than bring H@H Downloader back.

This post has been edited by WuvviDuvvi: Aug 6 2016, 02:42
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

 
post Aug 6 2016, 02:42
Post #20
zxwecrt



Newcomer
**
Group: Members
Posts: 66
Joined: 7-December 12
Level 222 (Lord)


The old system was a lot more fire and forget. What a shame. Probably not just me but right now I'm waiting for a 50 MB gallery to download at the wonderful rate of 12 kb/s (no it's not my internet). Pretty terrible for a DDL, but it woudn't even matter if it were torrented via H@H.

This post has been edited by zxwecrt: Aug 6 2016, 02:43
User is offlineProfile CardPM
Go to the top of the page
+Quote Post


6 Pages V  1 2 3 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 


Lo-Fi Version Time is now: 29th March 2024 - 00:25