Server load problems Idea.

[^7_of_9]

Been thinking about this and here's what I've been thinking about.

The Daemon (Which I currently use myself) is a great idea for us Farm people.

Is it possible to make a "Special" edition of this.

The Special edition would be for people who were willing to share some bandwidth come update time and they would run this program for say starting a couple hours before the client actually gets released and probably for about 24 hours after it gets released to make sure any last minute changes are picked up.

This Special Client would log onto a different location than the current one thereby reducing bandwidth on the main server. It would pick the updates up off of there. From there people who would wish to do so can then redirect there clients one of those Daemon's to download the updates from there.

I'd recommend anyone who wishes access to the special client have to agree to open their bandwidth up for anyone who wishes to use it
They also must agree that if they stop participating in the "Free Bandwidth Program" then they must surrender the software. This could be controlled by utilization of passwords and login names that the client would use.

This could help take some load off the DF servers during Update Day.

I'd be willing to participate and help out in Beta testing any kind of Software to help get this rolling as I've got the bandwidth to handle quite a few simul. connections

For those of you who are familiar with SETI and have run SetiQueue or used a Public Queue before (Which I have also run before at one time) then this would be somewhat similar in function

[PinHead]

Couldn't you use the current daemon on a public IP.

Then possibly each TEAM could setup their own update source if they choose to do so.

Each member of the team would only need to add a foldtraj_fileserver.cfg file to their client directory. It would contain an entry like:

http://www.teamproxy.org/dfp

The only problem I see is a corrective action for events similar to what happened the previous update. Wrong clients were set on the main page and the foldtrajupdateproxy grabbed them before they were replaced. When they were replaced, the version numbers were not changed so the proxy didn't grab the newest ones before the update was triggered.

Just a thought!!

[rsbriggs]

bitorrent aka "download swarming"?

http://bitconjurer.org/BitTorrent/index.html

[Brian the Fist]

We already do this actually! Of course we only have one mirror (beefdart) and we seem to have lost contact with him. People with cable/ADSL modems cannot be much help but people with T1's, T3's, OCxx, etc are welcome to become official mirrors, just send an e-mail to trades@mshri.on.ca and we'll set you up.

Just for the record, the updates were NOT incorrect, only the complete packages. Also, the updateproxy is based on file lengths, so as long as when a file is updated the package size changes, the proxy DOES grab the new version. However if your client has already downloaded the wrong one, IT won't get the next update if the version doesn't change. Make sense?

[PinHead]

Wasn't talking about this update. It happened on the update before that.

Either way, if a second entry is placed in the .cfg file will it try that server next?

[Brian the Fist]

Originally posted by PinHead
Wasn't talking about this update. It happened on the update before that.

Either way, if a second entry is placed in the .cfg file will it try that server next?

No. However, if I enter multiple servers in my 'master server list', your clients will randomly choose one, and if that one fails, try the next on the list, and so on.

[^7_of_9]

Originally posted by PinHead
Couldn't you use the current daemon on a public IP.

Then possibly each TEAM could setup their own update source if they choose to do so.

Each member of the team would only need to add a foldtraj_fileserver.cfg file to their client directory. It would contain an entry like:

http://www.teamproxy.org/dfp

The only problem I see is a corrective action for events similar to what happened the previous update. Wrong clients were set on the main page and the foldtrajupdateproxy grabbed them before they were replaced. When they were replaced, the version numbers were not changed so the proxy didn't grab the newest ones before the update was triggered.

Just a thought!!

I currently do this for my Team Members. I run my Daemon before the Update process starts (Usually the night before so I don't forget) and then stop it after everything is up and running ok. My Team Members have the info for the ip address and directory to use for their clients. The problems is that this really doesn't reduce server load now as it's a PITA for the Daemon to try and grab stuff.

There's been times where the update isn't grabbed by the daemon in a decent amount of time cause it keeps timing out cause of the current server load. It's gottan to the point where in Phase II the Daemon hasn't done anything for me except make really really large log files (Which Hopefully Brian fixed with this new version).


My main point really is that nout enough people are running the daemon in the first place. I mean I'm sure some people have some decent connections they can run the daemon on for 24 hours once a changeover for their team. It doesn't even have to be publicizede outside of your team. I'm just sick and tired of having a B***H of a time getting through cause of the load. It's not Howards fault or anything.

Overall I'd like to see some of these "Team Leaders" Get their heads together and help out in alleviating Server overload.

Hell if it would help at all I'll publish my Hostname and directory here for my Daemon. I rarley complain about anything. My biggest beef with anything regarding where I work and what I do on the internet is Network Congestion/Response Time.

[pointwood]

I have suggested this before and rsbriggs now suggests it again - Bittorrent is the perfect answer to this.

ADDED:
It is already being used for lots of stuff like Linux ISO's and other such big files that people download. It works amazingly well.

It has helped avoid the /. effect several times.

I'm willing to bet a box of beers that if this was implemented in the client, the BW problems would be gone!

[Darkness Productions]

As I've said before, I agree with Pointwood on this subject. The bandwidth problem is just that. A problem. If the clients can't connect to the server, then no work ever gets done. The update proxy is a great idea, and staggered proxies would be even better (being able to point my proxy to a Free-DC proxy, which would point to some other proxy, which would point somewhere else, etc, until one (or two of them) pointed to the official DF servers...), but I don't see either of these happening any time soon.

The bittorrent idea would require Howard to bundle in a copy of bittorrent with the client, making the initial client download, while already massively large for a DC client, even larger. Not only that, but then there would be any issues with packaging them together, since bittorrent is GPL and the client code is not.

The staggered proxy idea would probably be better and easier to develop in the long run, but the chances of everyone using them would be smaller. However, if several initial servers (say, an Ars server, a Free-DC server, an OCAU server, etc etc etc) pulled the updates from DF, then that's a single set of transfers for each of the team servers. Then, the users (for those with several boxes, bguint01, etc) have their own proxy setup, pointing to their teams proxy. This allows them to pull one time from the team proxy, instead of 10-100 (or even more) from the DF server. All their clients update from their local proxy, and the amount of bandwidth used by the DF servers shrinks from probably 1mbit/second to 10kbit/second (very rough guesstimations, but the idea is still the same).

This would also allow for more client connections (for those single machine people that aren't on a team) and also keep upload connections available as well.


Any thoughts from the project administrators?

[pointwood]

From the bittorrent site:

What license is BitTorrent released under?

MIT, which basically lets you do anything you want with it so long as you leave the license notification in the source.

[Darkness Productions]

Ok, so I was wrong about that. There still might arise some other issues with the bittorrent client being packaged alongside it. Besides, that adds another layer of complexity to the client. If the bittorrent stuff doesn't work (it crashes, etc) then Howard would have to fix it. I don't think he wants to do that

[rsbriggs]

This kind of thing is exactly what bitorrent was developed for, infrequent, massive downloads. For those of you that don't know what it is - think of it being something like this:

The first two people connect to the master site to download the client. They use a small part of their bandwidth to serve up the client to two people each who do the same, and you have 4 servers. The next 4 people to connect do the same, and you have 8 servers, then 16, then 32, then 64, then 128 servers, possibly within seconds of the first connections to the main server.

This is horrible oversimplification of what takes actually place, and actually doesn't describe the real process taking place, but should give you the general idea - the more people that download at once, the more efficient the process becomes. It's why places like linuxiso.org can serve up tens-of-thousands of downloads of a 600 meg ISO simultaneously when something new and popular gets released.

EDIT:

I'm not even saying that it has to be bundled. In my case, I'd be more than happy to manually download the new client, and manually upgrade my clients, if I could do it efficiently. It was nearly two days before things settled down enough to get a clean enough connection for the clients to start folding, the last time.

Letting the clients auto-connect and download proved to be rather a disaster last time - I ended up with 2 successful downloads, and 12 crashes due to the client getting disconnected, hung up, whatever, during the last switch. (And I couldn't restart the clients, because they couldn't contact the server...)

I'd rather do one download, and spend the 10 minutes it takes me to manually upgrade my 14 clients, and be done with it, rather than having it drag out for two days...

[rsbriggs]

Ok, so I was wrong about that. There still might arise some other issues with the bittorrent client being packaged alongside it. Besides, that adds another layer of complexity to the client. If the bittorrent stuff doesn't work (it crashes, etc) then Howard would have to fix it. I don't think he wants to do that

Then you'll understand me when I say that if there is one even one screw up during THIS upgrade - I'm likely to leave the project *I* don't want to have to deal with "another layer of complexity" trying to keep the client running on 14 machines, or to have to keep fixing it.

[pointwood]

Originally posted by Darkness Productions
Ok, so I was wrong about that. There still might arise some other issues with the bittorrent client being packaged alongside it.

FUD - it can only get better IMHO

Besides, that adds another layer of complexity to the client. If the bittorrent stuff doesn't work (it crashes, etc) then Howard would have to fix it. I don't think he wants to do that

True, it would add more complexity to the client and it would mean more work to Howard.

But it is, IMHO, very important to have a reliable update function. If the update continues to be a problem, people will eventually get tired of the project and leave...that will of course solve the problem too, but I don't think that's a better solution

UPDATE: Just check rsbriggs' post for proof...

[Brian the Fist]

As I said before, I think our network was bogged down with SoBig worms last time, which has thankfully died down now, so there shouldn't be much network congestion this time (I hope). Again, mirrors are always welcome. Anyone got a spare T1 or three out there?

On the BitTorrent thingy, I havent looked at it but sounds like Kazaa kind of. If it would involve adding more code to the client though, thats a big negative, especially if its GPL (which the client isnt).

[PCZ]

What is required to set up the mirror ?

[IronBits]

T1 or better upload speed, and about 4GB of drive space
I have 30KB upload speed and Howard won't let me play mirror

Folks like you and me should be running the Auto-update Daemon
to feed our numerous boxen anyways

[pointwood]

Originally posted by Brian the Fist
As I said before, I think our network was bogged down with SoBig worms last time, which has thankfully died down now, so there shouldn't be much network congestion this time (I hope). Again, mirrors are always welcome. Anyone got a spare T1 or three out there?

I've posted your request on our forum, maybe someone will bite

On the BitTorrent thingy, I havent looked at it but sounds like Kazaa kind of. If it would involve adding more code to the client though, thats a big negative, especially if its GPL (which the client isnt).

1. The client is NOT using the GPL license, it uses the MIT license which basically means you use have to include the copyright statement, that's all. You're free to do whatever you want with the code

2. This is NOT like Kazaa! Bittorrent has been created with the goal of being able to spread huge files to lots of people in a secure, fast and efficient way. It scales very well with demand!

You can see/read a very short introduction to it on the website right here: http://bitconjurer.org/BitTorrent/

It'll take you all of 30 secs. to read it

And their FAQ right here: http://bitconjurer.org/BitTorrent/FAQ.html

Here is a quote in regards to file integrity:

How do I know the download isn't corrupted?
BitTorrent does cryptographic hashing (SHA1) of all data. When you see "Download Succeeded" you can be sure that BitTorrent has already verified the integrity of the data. The integrity and authenticity of a BitTorrent download is as good as the original request to the tracker. Checking the MD5 of a file downloaded via BitTorrent is redundant.

3. Yes, it would likely involve adding more code to the client (or at least have bittorrent installed on client machines), but as I said in a previous post - I belive that is a better solution than loosing a lot of participants because the update function is unreliable and you *will* loose a lot of participants - it will spread quickly that this project isn't working reliably and people will avoid it. To a lot of people, this is "install and forget" if the update doesn't work, you lose those people in the next update...

[pointwood]

My request has already given results:
http://arstechnica.infopop.net/OpenT...1&m=2860985285

[PCZ]

IB
I was considering setting up a mirror at work not at home.