[X] My Assistant

all of my JP H@H clients lost most of the traffic in the same hour

(IMG:[s2.loli.net] https://s2.loli.net/2023/08/28/uOMXLochmJPjtD1.jpg)
(IMG:[s2.loli.net] https://s2.loli.net/2023/08/28/zQnSFBJiN9cD8TE.jpg)
(IMG:[s2.loli.net] https://s2.loli.net/2023/08/28/LuCXBbjHkhQ1SzF.jpg)

but their trust and quality didn't decrease
(IMG:[s2.loli.net] https://s2.loli.net/2023/08/28/5ua6HnJSFirZGTz.jpg)
and my HK client started running at five times the usual hitrate, I can confirm I can't find any error logs, but the trust still keeps decreasing...
I have no idea why

(IMG:[s2.loli.net] https://s2.loli.net/2023/08/28/hg2NtmbWL9CTi3H.jpg)
how...............
my HK client now increased to 8 times higher than usual......

All of my clients have same Trust issue this time

2023-08-28

- The algorithm to determine which H@H client to use for a particular image request has been replaced with a new range-focused priority system that will strongly prefer to use the same clients for the same range over time. This should strongly improve cache locality, and therefore significantly decrease the number of cache misses that require the client grabbing the file from the image servers.

- The requirement for having a client count towards the free archive quota has changed, and no longer requires a hitrate of 1. It now only requires that the client is healthy (i.e. usable, meeting minimum speed, trust and quality requirements), and that its current or previous session lasted at least 24 hours (i.e., it's been more than 24 hours since a restart, or the previous run was more than 24 hours since the previous restart.)

- Free archives from Adopt-a-Server slots was roughly doubled, and is now fixed at 7 GB/week per slot.

https://forums.e-hentai.org/index.php?showtopic=244935

all of my JP H@H clients lost most of the traffic in the same hour
and my HK client started running at five times the usual hitrate, I can confirm I can't find any error logs, but the trust still keeps decreasing...
I have no idea why

With the new priority system, clients need to fulfill certain criteria to be assigned what I call P1 ranges (for Priority-1), which are picked first whenever traffic is distributed. Clients 42358 and 44234 aren't being assigned any yet because they experienced a bunch of overloading recently, and 45451 hasn't been assigned any yet because its current and previous uptime are too short; clients need to have been running continuously for 24 hours to qualify.

If their situation improves, they should get some priority ranges assigned, which would make the traffic return.

As for client 40556, the "problem" is that the system really likes it and assigned it a bunch of P1 ranges, which caused a sudden increase of traffic, which is also increasing the number of backend fetches. There's nothing wrong with the client, and it seems stable for now, but I'll take a closer look at the trust formula tomorrow, it probably needs a tweak.

the "problem" is that the system really likes it and assigned it a bunch of P1 ranges

Cool, I love this server too. If the trust factor doesn't play an important part in assigning ranges in the new system, I will ignore it for now.

I've tweaked the trust formula for backend requests a bit; it's now based on the average hitrate instead of the number of ranges assigned, which makes a lot more sense after priority ranges was introduced. Let me know if you see any clients developing trust issues after this.

Cool, I love this server too. If the trust factor doesn't play an important part in assigning ranges in the new system, I will ignore it for now.

Trust doesn't really matter unless it's in danger of dropping below 0.

I figured I'd post here rather than starting a new thread, since it's related.

All eight of my H@H clients (edit: which are in France) have seen major drops in traffic (it used to be closer to 2 requests/s, and now is more like 2 requests/min on each client).

I'm curious what's up with that. They all have +1000 trust and 10000 quality right now (previously they were +1000 trust, but more variable quality) and have been up for a while (days or weeks at minimum).

This post has been edited by kamio11: Aug 30 2023, 02:54

Hey @Tenboro, just curious as I've also seen a huge drop in traffic. I'm guessing that this is because I had some downtime this weekend due to a power outage? Just seems a little odd because the traffic was flowing pretty well and actually steadily increasing when I started it back up Sunday and then suddenly dropped off today while my trust/quality is pegged at 1k/10k. How long would you expect before these "P1 Ranges" start being assigned and traffic returns?

All of my 8 clients also have traffic drop issue these days under normal operation. I could reach yesterday top 10 in toplist, but it dropped to behind 100+ today. @Tenboro please help to check the client assigning algorithms.

All of my 8 clients also have traffic drop issue these days under normal operation. I could reach yesterday top 10 in toplist, but it dropped to behind 100+ today. @Tenboro please help to check the client assigning algorithms.

OK, so you are the one who using 8 IPs from "same" provider and pointing to "same" machine, the algo change is for lowering such activity since the cache hit rate is so low and the response is so slow. You might be the one of slowing the access of whole country.

With one IP and one machine you can get at least 100Mbps outbound traffic when you have high connectivity.

I figured I'd post here rather than starting a new thread, since it's related.

All eight of my H@H clients (edit: which are in France) have seen major drops in traffic (it used to be closer to 2 requests/s, and now is more like 2 requests/min on each client).

I'm curious what's up with that. They all have +1000 trust and 10000 quality right now (previously they were +1000 trust, but more variable quality) and have been up for a while (days or weeks at minimum).

Your clients are all in Europe, which has a lot of spare capacity. The way the dispatcher used to work is that your traffic was basically linear to the number of ranges assigned, but this is no longer the case, since priority is assigned primarily based on the client's speed, which is far better for the overall health of the system. This means a client with 10 ranges assigned and a client with 1000 ranges assigned might both be assigned 10 P1 ranges, if they have the same speed.

Your clients all seem to have 100 Mbit connections and have 6000 ranges assigned, which means a large drop of traffic would be expected. I do have some other plans to make better use of clients with lots of space and relatively low speed, but it'll probably take some time before that happens.

I'm probably going to make some small tweaks to the assignment algorithm to give somewhat higher weight to the disk space available to make better use of it, but in your case, while it would help some it won't make the traffic go back to what it was.

Hey @Tenboro, just curious as I've also seen a huge drop in traffic. I'm guessing that this is because I had some downtime this weekend due to a power outage? Just seems a little odd because the traffic was flowing pretty well and actually steadily increasing when I started it back up Sunday and then suddenly dropped off today while my trust/quality is pegged at 1k/10k. How long would you expect before these "P1 Ranges" start being assigned and traffic returns?

The initial batch of priority assignments ran this weekend (on Sunday specifically), so if your client was offline or had just been restarted at that time, it would not have been assigned any initially. This is only temporary however - the way the priority assignment works is if one client is underutilized, it'll keep assigning more P1 ranges to it as soon as some are available, until it's at the average level.

All of my 8 clients also have traffic drop issue these days under normal operation. I could reach yesterday top 10 in toplist, but it dropped to behind 100+ today. @Tenboro please help to check the client assigning algorithms.

Most of your clients seem fine, and they'll likely get more P1 ranges assigned over time. The one with no traffic is probably because it was offline when ranges were assigned. See above for that answer.

(Asia traffic has been a bit unstable overall due to a combination of GFW issues and CF aggroing on some third party viewers whose traffic look like "HTTP requests trying to impersonate browsers".)

After looking at this a bit, I decided to tweak the client scoring algorithm a bit by adding a separate factor for number of total ranges assigned - this does make sense from a system perspective as well, seeing as these clients are a lot less likely to be pruning files from the cache and also will have a more older files in their cache simply because they have been running for so long, which makes them "worth" more despite the lower speed.

Specifically, this means +1 will be added to a score multiplier for every 2000 ranges assigned, so a client with 6000 ranges will have a 4x multiplier to their speed score, and therefore, over time, will get ~4x the number of P1 ranges than a client with only a handful of ranges and the same speed.

Note that this won't have an immediate effect on traffic, but these clients will be prioritized for new P1 assignments.

Your clients are all in Europe, which has a lot of spare capacity. The way the dispatcher used to work is that your traffic was basically linear to the number of ranges assigned, but this is no longer the case, since priority is assigned primarily based on the client's speed, which is far better for the overall health of the system. This means a client with 10 ranges assigned and a client with 1000 ranges assigned might both be assigned 10 P1 ranges, if they have the same speed.

Your clients all seem to have 100 Mbit connections and have 6000 ranges assigned, which means a large drop of traffic would be expected. I do have some other plans to make better use of clients with lots of space and relatively low speed, but it'll probably take some time before that happens.

I'm probably going to make some small tweaks to the assignment algorithm to give somewhat higher weight to the disk space available to make better use of it, but in your case, while it would help some it won't make the traffic go back to what it was.

After looking at this a bit, I decided to tweak the client scoring algorithm a bit by adding a separate factor for number of total ranges assigned - this does make sense from a system perspective as well, seeing as these clients are a lot less likely to be pruning files from the cache and also will have a more older files in their cache simply because they have been running for so long, which makes them "worth" more despite the lower speed.

Specifically, this means +1 will be added to a score multiplier for every 2000 ranges assigned, so a client with 6000 ranges will have a 4x multiplier to their speed score, and therefore, over time, will get ~4x the number of P1 ranges than a client with only a handful of ranges and the same speed.

Note that this won't have an immediate effect on traffic, but these clients will be prioritized for new P1 assignments.

Thanks, I'll keep an eye on them and see what develops. The clients are indeed all on (separate) 100 Mbit connections and have been running for long enough to have 6000 ranges and 1-2 TB of filled cache per server. How frequently are new P1 ranges assigned (so I have an idea of how long I should watch)?

I'm curious what the "preferred" speed for a client is now, if 100 Mbit is relatively slow. If I'm only getting a couple hits per minute on each of my clients, I would guess the marginal benefit of them is quite low, which makes me question whether it's worth it to keep them running in the first place (either for my own gains or for the site's benefit) or whether I should consolidate them into fewer servers with greater bandwidth and storage. At first blush it seems to me that while the new system may better utilize higher-speed servers, it seems to dramatically underutilize servers like mine (which is not necessarily a criticism, since it may be better for the site overall).

A related thing: I mostly just check the H@H client page to monitor my clients unless something seems odd. I'm guessing the hitrate and hathrate calculations use some kind of moving average. Maybe it should be trimmed or reset? Because it seems to be reporting wildly different numbers than what I'm observing (e.g., I've gained almost no hath or GP from them in the last couple days).

Thanks, I'll keep an eye on them and see what develops. The clients are indeed all on (separate) 100 Mbit connections and have been running for long enough to have 6000 ranges and 1-2 TB of filled cache per server. How frequently are new P1 ranges assigned (so I have an idea of how long I should watch)?

It could potentially assign 8 P1 ranges per region per minute, but how many are actually assigned depends on whether there are any P1 ranges to assign, so it's not really possible to estimate.

I'm curious what the "preferred" speed for a client is now, if 100 Mbit is relatively slow. If I'm only getting a couple hits per minute on each of my clients, I would guess the marginal benefit of them is quite low, which makes me question whether it's worth it to keep them running in the first place (either for my own gains or for the site's benefit) or whether I should consolidate them into fewer servers with greater bandwidth and storage. At first blush it seems to me that while the new system may better utilize higher-speed servers, it seems to dramatically underutilize servers like mine (which is not necessarily a criticism, since it may be better for the site overall).

100 Mbit/s is the minimum we currently accept for new applications, and you'd probably see better performance with a single gigabit client with 6000 ranges than six 100 Mbit clients with 6000 ranges each.

The original dispatcher was designed for the situation where the same range only existed on maybe two or three nodes per region, but in Europe right now the same range exists on average on 19.14 nodes, which was causing poor cache hitrates and correspondingly a high backend server load. The new system fixes that, but inevitably it did cause low-speed high-disk clients in high-saturation regions like yours to take a large hit in traffic.

A related thing: I mostly just check the H@H client page to monitor my clients unless something seems odd. I'm guessing the hitrate and hathrate calculations use some kind of moving average. Maybe it should be trimmed or reset? Because it seems to be reporting wildly different numbers than what I'm observing (e.g., I've gained almost no hath or GP from them in the last couple days).

It uses the same historic hitrate to calculate the hath rate as the one displayed on the H@H info screen, but keep in mind that it only credits it every four hours or so. GP on the other hand are credited from actual hits, so that one would be reflected more or less instantly.

Since the new algorithm will be affected by the network speed, I think the size of the speed test should be increased, the current size is a bit too small for high-speed networks.

Note that there were some further adjustments to the allocation algorithm where it will now somewhat prefer promoting a P2 or P3 range to P1 rather than picking a candidate by speed+disk score alone. This will further favor long-lived clients with an aged cache.

Since the new algorithm will be affected by the network speed, I think the size of the speed test should be increased, the current size is a bit too small for high-speed networks.

It's probably fine, since it specifically measures the time from first byte received to last byte received, so the time to establish the connection is excluded.

I'm in a similar situation. Most of my clients are in Japan and I used to crack the top 10 but now I'm in the top 30-50. I just increased the bandwidth limit because I'm not entirely sure what my provider is capable of. Most of my machines are about near their limit hd wise, I can expand storage but I thought with their ranges at 6k that didn't really matter.

Can you tell me why my hitrate has gone down from 850-900 to 500ish? Notably my Europe based machine's hath generation has gone up a bit I think. It has uncapped upload on a bigger pipe and a ton of free space.

With the new priority system, clients need to fulfill certain criteria to be assigned what I call P1 ranges (for Priority-1), which are picked first whenever traffic is distributed. Clients 42358 and 44234 aren't being assigned any yet because they experienced a bunch of overloading recently, and 45451 hasn't been assigned any yet because its current and previous uptime are too short; clients need to have been running continuously for 24 hours to qualify.

If their situation improves, they should get some priority ranges assigned, which would make the traffic return.

As for client 40556, the "problem" is that the system really likes it and assigned it a bunch of P1 ranges, which caused a sudden increase of traffic, which is also increasing the number of backend fetches. There's nothing wrong with the client, and it seems stable for now, but I'll take a closer look at the trust formula tomorrow, it probably needs a tweak.

nice, my Gigabit network sees some real usage now(nowhere near full of course).

I'm in a similar situation. Most of my clients are in Japan and I used to crack the top 10 but now I'm in the top 30-50. I just increased the bandwidth limit because I'm not entirely sure what my provider is capable of. Most of my machines are about near their limit hd wise, I can expand storage but I thought with their ranges at 6k that didn't really matter.

Can you tell me why my hitrate has gone down from 850-900 to 500ish? Notably my Europe based machine's hath generation has gone up a bit I think. It has uncapped upload on a bigger pipe and a ton of free space.

If your clients have 6K ranges and your hitrate "only" dropped by half, the clients will probably get more traffic than they used to when the priority assignments have been fully rebalanced, though that will take some time.