[X] My Assistant

A few people have asked me what kind of infrastructure it takes to run Galleries and the rest of E-Hentai. My usual answer to this question is "it's too complex to explain" or "it would probably bore you, and it kinda is, and it probably will.

Buut, in order to satisfy those curious souls, I decided to put together a somewhat simplified overview of the system as it is right now. It wasn't exactly like this a month ago, and it probably won't be exactly like this a month from now, since stuff changes literally all the time, but at least you'll get some indication of what we have running.

Since a picture says more than a thousand words - I made a chart!

[g.ehgt.org] *clicky*

This is largely a logical view - some servers have more than one task. Not shown are the various offline backup systems that are there in case of total system failure, as well as some administrative stuff.

Soo, basically. Your request for a page enters through one of the four the reverse proxies that provide load balancing and automatic failover, and gets passed to one of the four primary web servers. They process your request, usually by drawing on a shared memory cache and the three database replicas. If you request an image, they look up an available H@H client, and if none is available at the moment, punt your request over to one of the six image servers.

The image servers are arranged in three sets of two servers each, where each set is located at different hosts. Every pair of servers in a set hold a full copy of the entire Galleries repository, arranged in a way so that if any one server goes down, its requests are divided between every other server in all other sets. So losing one server in Set 1 would divide its load in equal 25% shares to each server in Set 2 and Set 3.

Thumbnails are similarly arranged, but since they could be regenerated in case more than one set was lost, there are only two of them - one with two servers and one with four. These use simple DNS round-robin for load balancing and failover.

I could probably write fifty pages on the minute details of how this hangs together, but I'll skip that for now.


As for the actual server hardware that runs it - this is most of it, anyway. All the servers except four or five have some flavor of Intel Xeon CPU, but I can't be arsed to look up the specific model for each. A couple of servers don't directly participate in hosting the site, so they aren't listed.

1x: 16GB RAM, 4x 73GB 10k RPM SAS (HW RAID-10), 10 TB @ 100 Mbit
1x: 16GB RAM, 2x 73GB 15k RPM SAS (HW RAID-1), 4 TB @ 100 Mbit
2x: 4GB RAM, 4x 2TB disk, unmetered 100 Mbit
2x: 4GB RAM, 1x 1.5 TB disk (OS), 3x 3TB disk (SW RAID-5), unmetered 100Mbit
2x: 4GB RAM, 2x 500GB disk, unmetered 100 Mbit
2x: 16GB RAM, 12x 1TB disk (2x HW RAID-1 OS, two 5x HW RAID-5), unmetered 100 Mbit
1x: 4GB RAM, 2x 1TB disk (HW RAID-1, 10x 2TB disk (two 5x HW RAID-5), unmetered 100 Mbit
1x: 4GB RAM, 1x 1TB disk, 5x 2TB disk (SW RAID-5), 30 TB @ 1 Gbit
1x: 2GB RAM, 1x 250GB disk, 2 TB @ 100 Mbit
2x: 8GB RAM, 2x 32GB Intel X-25E SSD (RAID-1), 2x 160GB Intel X-25M SSD, unmetered 100 Mbit
1x: 16GB RAM, 1x 80GB Intel X-25M SSD, 5TB @ 100 Mbit
4x: 16GB RAM, 1x 300GB Intel 320 Series SSD, unmetered 100 Mbit

All those goodies, put together, cost about $56k USD per year to lease and host. And no, it's very unlikely that you could do it cheaper.

Finally, I grabbed a screenshot of my status overview page, which I use to get a quick look at the system - just in case you are curious. A few identifiers and stuff are redacted, to protect the innocent.

[g.ehgt.org] *clicky*

The two "downed" servers are actually the two image servers I pulled offline earlier today. The dots and X'es on the top left are the heartbeat monitors for the image servers. Then, there are live bandwidth graphs for all the servers that come with it - the gaps aren't actual downtime, just times where the bandwidth measurements are unavailable. When you are familiar with the system, you can tell a whole lot about whether something is wrong or not just by looking at that.

I guess that's all for this time.


TL;DR: We have a bunch of servers. That's pretty much it.

Charts are awesome. Everybody loves charts.

/nerdgasm

I really appreciate you replacing the old image servers and getting better ones and the sort but I also wouldn't mind a new HV server. Specifically, one that's lag-free 24/7 would be nice. Staying up at the wee hours of the night just to play lag-free on weekdays isn't such a good idea. (IMG:[invalid] style_emoticons/default/smile.gif)

Actually, one of the things I've been working on lately is eliminating the "HentaiVerse" box and moving that whole thing onto the Primary Web Cluster. Hopefully, that should take care both of the lag and of a bunch of random connectivity issues between the PWC and the HV server.

It's somewhat complex to make work, but I've solved most of it. Unless something else turns up, I'll probably start the deployment testing this weekend.

Gonna turn the old HV box into anything specific?

Actually, one of the things I've been working on lately is eliminating the "HentaiVerse" box and moving that whole thing onto the Primary Web Cluster. Hopefully, that should take care both of the lag and of a bunch of random connectivity issues between the PWC and the HV server.

It's somewhat complex to make work, but I've solved most of it. Unless something else turns up, I'll probably start the deployment testing this weekend.

Best news since the invention of the wheel. Hopefully that will take care of the lag problem.

This post has been edited by sushi0: Sep 19 2011, 22:35

Gonna turn the old HV box into anything specific?

The server in question does other stuff as well. Probably won't put anything more on it.

I don't understand anything you wrote there. (IMG:[invalid] style_emoticons/default/biggrin.gif)

This is probably one of the coolest things I've seen on the internet in a long, long time.

56K? Do you get enough in donations (etc...) to at least break even from that cost?? (IMG:[invalid] style_emoticons/default/sad.gif)

It's incredibly impressive how you've put everything together.


That post is also really freaking neat.

TL;DR: We have a bunch of servers. That's pretty much it.

Teehee.

That really was pretty neat. I never wondered about how the site worked, but I assumed it was something in this general form. Its actually quote marvelous -- it makes me happy in a nerdy sort of way. As a tech-oriented sort of person, I can really appreciate the amount of thought and effort put into keeping the place running smoothly.

Thanks for the description, that was interesting. I understood most of the relationship between servers except for the forum: why is it connected as a requester to the HV and the web cluster ?

Also, there are more In traffic than Out traffic in most of the bandwidth graphs. Shouldn't it be the opposite ? Moreover, I'm not sure which graph belongs to which server, but it seems the network load isn't accurately balanced between servers that have the same purpose.

And I wonder about the same thing as Tigenzero.

Its actually quote marvelous -- it makes me happy in a nerdy sort of way.

I believe the word you're looking for is nerdgasm.

Thanks for the description, that was interesting. I understood most of the relationship between servers except for the forum: why is it connected as a requester to the HV and the web cluster ?

Some functions of HV are directly connected to the forums. An example would be player level. If they weren't connected directly it would probably take too long to transfer information back and forth. It's most likely set up this way to facilitate faster updates between the two.


I've said it before and I'll say it again. I'm constantly amazed at how he's able to maintain this entire site by himself. It's a one man job and it's not easy. Some people have said that I play 3+ people's worth in HV, but Tenboro probably does 10+ people's work on this site. On top of all this he still has time for Ponies? Well damn. Scratch what I said, this man is worth at least 20 people.

I believe the word you're looking for is nerdgasm.
Some functions of HV are directly connected to the forums. An example would be player level. If they weren't connected directly it would probably take too long to transfer information back and forth. It's most likely set up this way to facilitate faster updates between the two.
I've said it before and I'll say it again. I'm constantly amazed at how he's able to maintain this entire site by himself. It's a one man job and it's not easy. Some people have said that I play 3+ people's worth in HV, but Tenboro probably does 10+ people's work on this site. On top of all this he still has time for Ponies? Well damn. Scratch what I said, this man is worth at least 20 people.

He has time for ponies? No no.... Ponies have time for him.

Heh, and some folks think its just 1 or 2 servers that run everything!

Well, in theory you COULD do that...but you'd need a few servers like a Dell PowerEdge R910 with max configuration and 16x SSD drives to handle everything but the actual image gallery.

Much appreciation on the backend info (IMG:[invalid] style_emoticons/default/smile.gif) Maybe now folks will up their donations after realizing how much hardware it really takes to run this place!

This post has been edited by -Enforcer-: Sep 20 2011, 06:26

All those goodies, put together, cost about $56k USD per year to lease and host. And no, it's very unlikely that you could do it cheaper.

That's a little scary to read. If you don't mind my asking, how much of that is covered by donations, ads, Adopt-a-Server slots, etc? How do you make up the difference, if there is one?

That's a little scary to read. If you don't mind my asking, how much of that is covered by donations, ads, Adopt-a-Server slots, etc? How do you make up the difference, if there is one?

Tenboro is actually a reclusive billionaire who made his fortune in the beaver trade. His parents were accosted by an errant naughty book peddler during his childhood while they were exiting a theater, so that day he vowed he would make the world a better place.

I'm kinda curious about this, but I kinda don't wanna know.

This post has been edited by cmal: Sep 20 2011, 07:03

wow...six years of progress basing on the join date could even be seven plus and it's amassed to such a juggernaut of indescribable proportions by the thought, dedication, and perseverance of one man this...this is the work of a genius. All I could give is a meager thanks from all lurkers, leechers, lechers, noobs, and I whom congregate this site for our own personal gain.

Sincerely,
An '07 Lurker

This post has been edited by GandrakeIII: Sep 20 2011, 07:58

Woah I have no idea what any of that means but judging by all the graphs and the amount of money it costs I'm gonna assume its pretty impressive.

Seriously though I for one am very greatful for all the work and time you put into this site making it the best hentai image sharing site on the web, you are a legend (IMG:[invalid] style_emoticons/default/biggrin.gif)

Impressive. When the website first started out, did you expect it to grow to the size that it has today? When a new doujin comes out, this is the website where it first becomes available. I've often wondered what is it that attracts people from all over the world to first come and post their images onto the galleries on this site and not other websites.