[X] My Assistant

I just finished switching the site over to use our new primary web cluster. This replaces the old setup that had six web servers and two database replicas with a new setup that consists of 8x Intel Xeon E3-1270V2s with 32GB RAM and a 120GB Intel 520 SSD.

Unlike the old setup, each node in the new cluster runs its own separate copy of the database, which means that overloading either of the old replicas - which has been a big problem in the past - should no longer be an issue. The raw processing capacity is also several times higher than the old cluster, which leaves us both with some badly needed redundancy, as well as plenty of room for future traffic increases.

With the new cluster, we have also acquired our own private rack, with an actual dedicated private switch that should prevent some other issues we have had in the past - namely where internal network problems could disrupt private communications between our servers. While the switch is obviously a single point of failure, it's still a significant improvement over the old situation.

Note that several of the existing servers will be moved to the new rack some time in January, but until this happens, the site is in somewhat of a transitional state, where parts of the internal traffic has to traverse a public network. I don't expect this to be a major problem, but just so you know, we're not entirely done yet.

Either way, if you notice any weirdness with the site after the switch, let me know.

Sounds cool (IMG:[invalid] style_emoticons/default/happy.gif)

I don't notice anything unusual, so there should be a thanks to Tenb (IMG:[invalid] style_emoticons/default/smile.gif)

Hmm, single point of failure...
How fast can the guys at the datacenter run if or when the switch fails? :P

Though in all seriousness it's nice to see everything is in one place... or going to be.

Hmm, single point of failure...
How fast can the guys at the datacenter run if or when the switch fails? (IMG:[invalid] style_emoticons/default/tongue.gif)

Heh well, everything can run off the public switch if the private switch fails. It's still a closed VLAN.

Though in all seriousness it's nice to see everything is in one place... or going to be.

Not everything, we have more than a rack worth of servers.. but most of the stuff that depends on reliable interconnectivity between the servers.

Either way, if you notice any weirdness with the site after the switch, let me know.

Does this include HV?
I've been having some noticeable lagsplkes/delays between actions since yesterday, despite my usual 5-15ms to hentaiverse.org. It's not always the case, just happens irregularly. Or it might be because of the recent holidays and completely unrelated.

As far as galleries go: all fine. So thanks as always for all your work. (IMG:[invalid] style_emoticons/default/biggrin.gif)

Did not notice and difference for HV.

didn't find any difference. ...

Just curious what kind of interconnect are you using between the servers? I do enclosures, interconnects, and blades for a living so it all interests me. Also what kind of enclosure/blades did you go with? Something with a lights out setup?

I want to ask a ton of more questions but I'll try and contain myself.

I've been having some noticeable lagsplkes/delays between actions since yesterday, despite my usual 5-15ms to hentaiverse.org. It's not always the case, just happens irregularly. Or it might be because of the recent holidays and completely unrelated.

Chances are that's related to the transitional bits I mentioned. The old state tracking server has a bit of a flaky public connection, which is causing some issues. It will be about ten days or so before we can start migrating the existing servers, so it'll just have to be like that until then.

Just curious what kind of interconnect are you using between the servers? I do enclosures, interconnects, and blades for a living so it all interests me. Also what kind of enclosure/blades did you go with? Something with a lights out setup?

Run of the mill gigabit ethernet. And run of the mill 1U rack servers.

Nifty. Now all you need is a big red button

I had a period of about an hour today where my H@H client timed out and ceased activity for some reason. I'm not sure if that could be connected to the changes, but in any case it seems to have fixed itself. I haven't noticed any differences with the site or HV.

No it's related to this issue. Still unsolved.
https://forums.e-hentai.org/index.php?showtopic=91503

All fine now (IMG:[invalid] style_emoticons/default/smile.gif)

Oh it would be so awesome to have a pic of the rack (IMG:[invalid] style_emoticons/default/smile.gif) Network porn!

Oh it would be so awesome to have a pic of the rack (IMG:[invalid] style_emoticons/default/smile.gif) Network porn!

I was going to ask when all the servers have been moved. Though it'll only be about half full, for now. The rest of the rack will be reserved for future expansion.

An own rack sounds awesome. Great news.

I was going to ask when all the servers have been moved. Though it'll only be about half full, for now. The rest of the rack will be reserved for future expansion.

Please do. I also wanna see a picture of it.

Looking back, I'm kind of shocked we didn't have our own rack already! (IMG:[invalid] style_emoticons/default/smile.gif)

Hard to predict what your growth is going to be - and be economical about it.

I can only imagine what the H@H initiative has done to help reduce load on servers. That was probably the biggest method to help cope with the growth.

Its a shame I retired from H@H but I like to think I helped improve it and that makes me feel good. (IMG:[invalid] style_emoticons/default/smile.gif) Not needing a major update for over a year means we (Hairsfan, and a bunch of others) did a mass amount of testing and its paid off. (IMG:[invalid] style_emoticons/default/smile.gif)

Looking back, I'm kind of shocked we didn't have our own rack already! (IMG:[invalid] style_emoticons/default/smile.gif)

It's a bit more of a commitment than picking up servers here and there as they were needed. But the existing servers that are now being racked together were connected over a private VLAN already, since ~June 2010 anyway.

I can only imagine what the H@H initiative has done to help reduce load on servers. That was probably the biggest method to help cope with the growth.

It's saving us in the ballpark of $5k to $10k per month in pure bandwidth costs. Or if you compare it to a manestream CDN, $25k to $150k per month.