Current server loads

[Stardragon]

You will be glad to know that the webservers are now running at ~1/2 - 2/3 of their maximum capacity, and the database machine is coping as well, although it is still mostly running close to max capacity.

I understand things have been running at a reasonable pace throughout the weekend from. Again, some people may have been buffering over the weekend so slightly higher load levels on Monday are to be expected.

I realize that many people are running the client on work machines, or have dial-up, but please try to minimize offline buffering to the best of your ability. That will allow everyone to upload faster at all times.

[gistech1978]

ive left my 2 comps at work run without buffering two nights earlier this week
and i come into to work in the morning to find them both sitting idle.

i thought the client was changed to keep crunching even when it cannot connect to the server? if it has been changed, it hasnt been working very well. at least in my experience.
perhaps im in the minority here.

[Darkness Productions]

Once again, we ask if there's anything that the community can do. I'm sure that there are those of us who'd be willing to donate hardware/money/time to the cause (on top of what we already do), we just need something to tell us what you need.

If you don't need anything hardware/money/time related from us further than crunching, please say so.

[tpdooley]

I've seen numerous 8+ hour stretches in the error.log files at work where the router acted up, or the cable modem needed to be shut off and turned back on to get a working connection again. I've even seen it once or twice here at home. They all seem to still be running.

Mine were all running prior to losing the internet connection (they weren't started with the internet connection off). So 6-8 of my machines losing the internet connection overnight - until the router and cable modem get reset in the morning - don't seem to be worried about loss of the internet - as long as they started with access to the internet.

[Stardragon]

Originally posted by gistech1978
ive left my 2 comps at work run without buffering two nights earlier this week
and i come into to work in the morning to find them both sitting idle.

i thought the client was changed to keep crunching even when it cannot connect to the server? if it has been changed, it hasnt been working very well. at least in my experience.
perhaps im in the minority here.

This has indeed been changed and tested, as well as reported to be functioning as required. Is anyone else running into this problem? And did you check your error log for possible errors?

[gopher_yarrowzoo]

Minor Problems with uploading here, machines just keep loosing server connection...
Im on dial-up still, Im hoping to get ADSL within the next week or so (just hope my phone line is usable)

3 Machines - Networked, WinRoute Pro on 1 to connect all 3 to the net via 56k modem - worked fine until this new protien came along

[Mikus]

I'm a dial-up user, meaning that every time I connect to the DF server, I have multiple filesets to upload. I had gotten spoiled when for several days each upload I started would continue until there were no more filesets to be transmitted.

But right now the DF server has again become HORRIBLE. I start an upload, and that upload is timing out when fewer than ten filesets have been transmitted. Meaning: to get any quantity of work sent to DF, I have to start uploads again and again and again - I'm tired of that. The fact that it worked adequately end of last week shows that DF is sometmes usable by dial-up participants. But today's server behavior makes the DF project feel repellent, not welcoming.

mikus

[HansArne]

It's the same thing here.
I haven't been able to upload in two days.

[QIbHom]

I'm having the same problem as Mikus is on my one dial up box. Anywhere between 7 and 40 generations upload, then I get a Server Cannot Be Reached error. It takes some time to get stuff uploaded.

[Fozzie]

at the moment.is an absolute nightmare.

As the guys before me have stated the connection continually times out.

There is only one way to do it and that is to set a batch process to restart the upload everytime it times out.

As they say it was fine with the previous protein it is next to impossible now.

I could upload about 1 million points worth of gens over a 2 hour period before with the server as it is I'm "lucky" to get half that due to timeouts.

So is the database server the bottleneck?

DF =

[TazAmdmb]

Pretty much the same here in IL.
Cable connection (usually 256 upload speed)

I've got roughly 12,000 generations to upload and at best, it uploads 1 generation per minute and times out before 10.

I tried again this morning at 6am CST and again at 11am CST with the same results.

[Darkness Productions]

How often are peoples clients connecting? AKA, how long does it take to run through the generations?

[Beyond]

I'm in the same boat, 17000+ buffered and have been unable to connect since Sunday night, that is until an hour or 2 ago. However the connections seem to time out before more than 5-10 generations are sent. In the time it takes to upload the 5 or 10 generations most of my machines can do 10 or more new generations, so I am falling behind the curve.

[rel279]

DP you are probably pretty close......fast machines are probably repetedly trying to upload causing a slowdown......I wonder if client had an upload setting for 10or 20 gens at a time it would provide some relief

[Darkness Productions]

I honestly think it's mostly a configuration issue on the server at this point. An 8x CPU box should handle nearly millions of simultaneous connections. So it's either that the server isn't configured well enough, or the bandwidth is becoming the limiting factor... And I don't think the bandwidth is limiting it, because I can fetch the client download at approx 349.12K/s from the server...

It'd be interesting to see how the box is configured.

[Beyond]

Originally posted by rel279
DP you are probably pretty close......fast machines are probably repetedly trying to upload causing a slowdown......I wonder if client had an upload setting for 10or 20 gens at a time it would provide some relief

The problem with that is, when you are at a point of total saturation any scheme to game the connections, be it caching to some point before uploading, or as has been suggested "no offline buffering" will not resolve the problem. The cacheing to some point only creates a situation were the connections last longer, dening connections for those not cacheing, so you end up with more of the same. This holds true for the no offline cacheing scheme as well, if all thos ethat are cacheing right now were to suddenly stop thier cacheing the flood of new connection requests after every generation would only serve to compound the problem. At this point (at least to me) it looks as if we are at or very near total saturation.

[Fozzie]

The project is saturated.

And some bright spark has changed the goal posts to 30 billion for this nightmare protein.

That is gonna switch off a lot of people.

I have switched off 50% of my boxen cos I know I can't upload the work they are producing.

Makes you wonder whether the guys at the server end have their finger on the pulse.

I had 17000 WUs and have spent 24 hours + trying to upload them, I've still got 4500 left to do.

Probably got 10k waiting at work to great me too.

If they are serious at the Uni and want us to do the work then they have to have a look at the config of the servers.

[gistech1978]

Originally posted by Stardragon
This has indeed been changed and tested, as well as reported to be functioning as required. Is anyone else running into this problem? And did you check your error log for possible errors?

yes ma'am i have checked the error logs.
nothing out of the ordinary except the cannot reach server type errors.
it happened again on my laptop last night. i had tried it again folding online and at some point during the night, it went idle again with 370 some odd gens buffered. i ran it online to hopefully upload the buffered gens over night. it must have ran for a while because i checked it before bed and had 410 or so.

my error logs consist of hundreds of this same error

========================[ Mar 22, 2004 4:03 PM ]========================
Starting foldtrajlite built Jan 12 2004
Mon Mar 22 16:07:31 2004 ERROR: [010.003] {taskapi.c, line 1218} [ReadServerResponse] Timeout waiting for response, got 0 chars.
Mon Mar 22 16:07:31 2004 ERROR: [000.000] {foldtrajlite2.c, line 4933} Error during upload: NO RESPONSE FROM SERVER - WILL TRY AGAIN LATER

[willy1]

25% of my always-connected clients (multiple T3 pipe) are now buffering results because they can't reach the servers.

[pfb]

5 hours to upload 140 workunits over broadband...not good

Seems to be a vicious circle with this protein/server problem - slow uploads due to lots of uploads = buffering to avoid the slow upload = lots more workunits being dumped at once = slow uploads...

Oh well, only another 4-5 weeks or so to go...

[goldie22]

Same boat here. I'm on DSL and and have several boxen w/ units in their belly I can't upload. Doesn't make a lot of sense to keep crunching units you can't send in!
goldie

[deranged128[OCAU]]

Originally posted by rel279
DP you are probably pretty close......fast machines are probably repetedly trying to upload causing a slowdown......I wonder if client had an upload setting for 10or 20 gens at a time it would provide some relief

The concept of batched uploads has been mentioned. The client could put together say 20 gens in a batch, with checksum and upload them in one transaction. Decreases the number of times the client has to negotiate with the server by a considerable factor for the price of a longer upload.

[N.V.M.]

this is just nuts. i CANNOT upload my files!!! i've been trying for three days now, at various times of the day. it just times out after two or three gens of uploading. i am getting seriously pissed off .

[rel279]

Checkout this thread. It might work for you.

http://www.free-dc.org/forum/showthr...&threadid=5937

[Mikus]

Looking at the lights on my modem (while an upload of multiple filesets is going on):

After the period when the 'send' light is on (presumably this is when a single fileset is being transmitted to the server), BOTH the 'send' and the 'receive' lights go dark for a period lasting from seconds to MULTIPLE MINUTES.

My guess is that the DF network server *did* receive the fileset correctly, but is now waiting for that fileset to be recorded in the DF database server - i.e., that the handshake to receive the *next* fileset is not initiated until the dialogue between the network server and the database server is completed for the *current* fileset.

If my guess is correct, then it is the DATABASE server that is holding up the uploading of completed generations by DF participants. People have commented that the hardware for the database server seems adequate. Could the dialogue involved in storing each fileset be speeded up, so that the user's client does not time out while waiting for the network server to tell it to transmit the next fileset ? [The worst wait - multiple minutes - seems like a LONG time to perform the storing of a record in a database.]

mikus

[IronBits]

Originally posted by Mikus
Could the dialogue involved in storing each fileset be speeded up, so that the user's client does not time out while waiting for the network server to tell it to transmit the next fileset ?

There could be a directory for every handle so we could be uploading our work into our own directory.
Then they could have a daemon running that runs thru the 1st directory until it hits the last one, then starts again, this way, the database server can work at it's best pace and let us work at our best pace

[N.V.M.]

thats that. my disk ran out of space, and it won't even load now.(Libranet OS)

CRASH!

[nailbomb]

I can't upload any buffered results either. I run online as much as I can with my dialup, but buffer when I sleep. And now I can't get anything up.

[2fast4u]

Same thing here nailbomb. I have several thousand gens to upload but I am lucky to get 2 or 3 uploaded before I lose the server. In the time I upload 3 I have buffered 15 more! Somethings gotta give or this will never end. We are gonna end up with a billion buffered gens and nowhere to send them! Sitting on about 18 million points and waiting for a nice power outage or disk faliure to make my day complete!!!

Email me about whats going on at XS jimking1@adelphia.net

[N.V.M.]

ok, new plan. everybody with the names starting with "A" through "M" , disconnect for 4 hours so "N" through "Z" can upload. tomorrow,we'll reverse that.

[pfb]

looks like the stats have died - currently ' as of: Wed Mar 24 02:19:11 2004 EST' which is ~4 hours old...

[TazAmdmb]

Originally posted by 2fast4u
Same thing here nailbomb. I have several thousand gens to upload but I am lucky to get 2 or 3 uploaded before I lose the server. In the time I upload 3 I have buffered 15 more! Somethings gotta give or this will never end. We are gonna end up with a billion buffered gens and nowhere to send them! Sitting on about 18 million points and waiting for a nice power outage or disk faliure to make my day complete!!!

Email me about whats going on at XS jimking1@adelphia.net

I'm going to try and offload the directories from 6 machines onto a laptop
and write a little batch script with the -u t and have it try uploading continuously. I hate to resort to that, but I'm not even a big producer and I've got 12000+ to upload and am producing more than I can send everyday.

[magicfan241]

Originally posted by pfb
looks like the stats have died - currently ' as of: Wed Mar 24 02:19:11 2004 EST' which is ~4 hours old...

.


One can only hope that they turned the stats server into an additional server to accept the uploading generations. I would accept 6 hours of no stats a day (8 even), if it would fix the problems by giving them another box to suck up the generations.

Of course, I don't see half the problems most do because I live in the NE USA, putting me pretty darned close to the server box.

If I lived down the street, I'd offer people sending me directories to give to Howard by CD.

magicfan241

[Gumby]

Something has to be done or people are going to give up. Why don't they just bring this generation to an end and start fresh with a new one? They are gong to have to extend the upload of buffered generation up to 2 weeks when this one is over.

[Mikus]

I dial-up to connect to the internet.

I have *one* machine running DF. Currently, with slow responses at the server and time-outs at the client, I need to be CONNECTED for *more* than 12 hours just to complete uploading the generations my machine can create in a 24-hour period.

Normally (except for DF) I'm connected perhaps one hour a day (I'm not into web surfing). Let me tell you, it is a turn-off to encounter a project which imposes such a large (IMHO - unreasonably large) CONNECT time burden on its participants.

[erk]

Originally posted by Mikus
I dial-up to connect to the internet.

I have *one* machine running DF. Currently, with slow responses at the server and time-outs at the client, I need to be CONNECTED for *more* than 12 hours just to complete uploading the generations my machine can create in a 24-hour period.

Normally (except for DF) I'm connected perhaps one hour a day (I'm not into web surfing). Let me tell you, it is a turn-off to encounter a project which imposes such a large (IMHO - unreasonably large) CONNECT time burden on its participants.

The slow generation upload kills a big hunk of my setup as well. I run a diskless linux folding farm, each machine has 256MB of RAM, and because the -rt flag uses so much RAM, I can only get buy with a 64MB RAM disk image, each machine has about 30MB of RAM disk space free for file storage, then bang it crashes I have to reboot and start at gen0. So when the servers can't take my results for a few hours I lose them, it's that simple.

[tpdooley]

can you run a batch file that would stop the DF client if the free space gets too low.. and then run foldtrajlite with the -ut flag until all the buffered gens are uploaded.. and then continue on?

[[DPC]WiZZarD]

make sure that u have sleep.exe in ur system32 folder and run this batch in a copy of ur df dir...

@echo off
:START
foldtrajlite -f protein -n native -u t
echo sleeping for 30 seconds
sleep 30
goto START

dont forget to purgeuploadlist before re-enabling ur client again...

this will work on a windows machine... there are also plenty of linux scripts available on the net... just search