time to reduce the upload bandwidth requirements please

[FoBoT]

for the "new people":

way back when this project started, the amount of stuff uploaded to DF from the client was much, much larger than it is with today's "phase II" client.

sometime during phase I (about half way through maybe?) , Howard was kind enough to make a change to just upload a portion of the data vs. the all of it to reduce the amount of upload bandwidth required.

with this latest "fast" protein's massive data accumulation rate (my work lab is producing results faster than my upload bandwidth allows sending of the data) , I am asking for another change to the volume of DATA the client returns to help alleviate this problem


please, howard, consider reducing the # of KB the client returns for each generation in some way to help us out

DF is my #1 favorite DC project for many reasons and I would like to continue to keep the majority of my assets on DF , but i just don't have the upload bandwidth available during these "fast" protiens.

thank you for your consideration.

[HaloJones]

Seconded.

I'm racking up buffered generations at a frightening rate which has never happened before. Now maybe this is just temporary but at present, the machines are wasting time trying and failing to connect.

[Ned]

I've used a dial-up line monitor to look at the bandwidth for both the current versus beta code for uploading. The beta code returns less than 2K of response versus the 38K plus of the current production code. So the beta should help. However, I suspect its not ready for prime time yet.

It looks like the current protein is too fast for current server. I suggest that the project abandon this protein for a larger/ slower one in the interest in getting some meaningful results in the near future. The 58 protein can be revisited when the architecture has been revamped to cope.

Ned

[FoBoT]

if the "beta" has a 95% reduction in the upload bandwidth requirements, then my request has already been granted

that is what i get for being a moderate stats whore and not participating in the beta

[Stardragon]

The beta is not bogged down simply because there is a tiny number of users uploading data, as well as the fact that the beta protein still remains the slower one. The response size is not the bottleneck.

[Stardragon]

Originally posted by FoBoT

...

please, howard, consider reducing the # of KB the client returns for each generation in some way to help us out

DF is my #1 favorite DC project for many reasons and I would like to continue to keep the majority of my assets on DF , but i just don't have the upload bandwidth available during these "fast" protiens.

thank you for your consideration.

Unfortunately, we have already cut down the content of the uploads to bare bones. If we remove anything else, we will simply not have sufficient data to carry out analysis. I believe that people may still be flushing results and keeping the servers ultra busy. Clearly not everyone is having difficulty getting through, so it is just a matter of time. We cannot miraculously improve the hardware overnight, nor do we have such resources at the moment. We appreciate the patience you have shown so far, and you should be seeing the server loads levelling out fairly soon.

[Beyond]

Originally posted by Stardragon
Unfortunately, we have already cut down the content of the uploads to bare bones. If we remove anything else, we will simply not have sufficient data to carry out analysis. I believe that people may still be flushing results and keeping the servers ultra busy. Clearly not everyone is having difficulty getting through, so it is just a matter of time. We cannot miraculously improve the hardware overnight, nor do we have such resources at the moment. We appreciate the patience you have shown so far, and you should be seeing the server loads levelling out fairly soon.

me "keeps fingers crossed"

[FoBoT]

my concern isn't about the server sides ability to handle the influx of data , that isn't the aim of my post

it is about the number of client PC's i can run on a single conection of XXX KBps upload speed (in the case of my main lab at work, it is 128KBps upload limited)

thanks

[PCZ]

FoBot

I am having a lot of trouble uploading but I am not short of bandwidth.
The slow response of the upload servers is the problem.

Can you monitor your bandwidth usage ?

[tpdooley]

During Phase I, I remember that choosing the option of uploading every 10,000 structures made it upload once every two hours with one of the proteins .. on one of my machines.

Even with the extremely long delays involved in uploading data, I've got a number of machines that are going through more than 250 gens a day.. even though it looked like it was spending half its time trying to upload. (One win98 machine had 220 cached generations after getting kicked off the internet after the client update..)

Instead of uploading once every hour or two - we'll be uploading 10-20 times an hour now with a protein this easy to fold.

If all this is because of handshaking - then perhaps combining the data for 10-20 gens into one file, performing the handshaking, upload the 1 file, then break it up and send us back 10-20 results - would help out. Either that or it would start straining the backend servers dealing with the processing of the data..

[Mikus]

I'm racking up buffered generations at a frightening rate which has never happened before. Now maybe this is just temporary but at present, the machines are wasting time trying and failing to connect.

I'm using dial-up to connect to the internet. I'm being forced to conclude that the DF servers are not adequate to support my participation with this protein.

In the past 24 hours my single machine has accumulated 100+ generations. Meaning that the number of generations it built is over one hundred MORE than it managed to transmit to the servers.

I'm being affected twice: (1) Often, when I dial the internet and start an upload to DF, the server does not have a connection available. The upload process times out. (2) When I *do* reach the server, and at least one generation has been uploaded, sooner or later the server is so slow that the upload process times out.

From error.log:
Fri Mar 19 04:14:47 2004 ERROR: [010.003] {taskapi.c, line 1218} [ReadServerResponse] Timeout waiting for response, got 0 chars.
Fri Mar 19 04:16:48 2004 ERROR: [000.000] {foldtrajlite2.c, line 4933} Error during upload: NO RESPONSE FROM SERVER - WILL TRY AGAIN LATER

I simply do not have the time or the patience to keep dialing many times a day, nor to keep restarting uploads which terminate when there are still filesets to be uploaded.

mikus

[FoBoT]

Originally posted by PCZ

Can you monitor your bandwidth usage ?

not with my current setup

i have a freesco (linux) box acting as the router, it is a floppy distro , so no nice monitoring tools

i should switch it for a "real" router distro and do some monitoring to see where i really am sitting on upload volume

in the past, when dumping offline boxen through this setup, i have not been able to get more than 2 million "points" in a single update on the stats regardless of how many boxen i was dumping through this line, i am currently running close/around that

[Fozzie]

I'm on dial-up too and will probably scale right back until we get another slow protein as this "wait and see" policy is just ridiculous.

Doesn't take an Einstein to know that this changeover would be the fiasco it has turned out to be.

With the cow hordes onboard, and good on them too, our numbers swelled.

With rumours being put about that this would be a fast protein all manner of stat hungry vultures have also tunred back to DF. Don't blame those either.

The upshot, which was obvious to many of us, is that the servers didn't cope, and so far are still not coping.

Looks like we have reached the saturation point for contributors.

I hear there are many other deserving projects with superior backends

[Mikus]

Originally posted by Mikus
In the past 24 hours my single machine has accumulated 100+ generations. Meaning that the number of generations it built is over one hundred MORE than it managed to transmit to the servers.

I take back what I said. Since my last post, things have changed so that once an upload process has begun, it has not timed out, but has continued until all available filesets had been uploaded. Meaning that now the generations I had accumulated in the meantime *have* been accepted by the FD servers.

Thanks, mikus

[Darkness Productions]

Originally posted by Stardragon
We cannot miraculously improve the hardware overnight, nor do we have such resources at the moment. We appreciate the patience you have shown so far, and you should be seeing the server loads levelling out fairly soon.

Care to list what you need/want? I'm sure someone around here might be able to help...