Cannot upload at any reasonable speed on dial-up

[Fozzie]

I know all you broadband users might not have been effected but anyone on narrowband must have seen the really flakey/poor response from the DF server.

I hope it is being looked into as I have 3000+ gens sitting on machines that I cannot upload becase it is taking over 4 hours to upload 500 gens.

I am producing work faster than i can upload it.

Pleas do not allow the changeover to go ahead without this being porperly remedied.

[Chinasaur]

I need an answer on what Fozzie needs to send me so I can upload for him.

What files does he need to send me and will they work cross-platform? ie Windows to OS X or Linux?

TIA.

And Fozzie posted about this at the last update and got slammed for it as if he'd waited till last minute. He is trying to upload way in advance of the cutover and the server limitations are killing him again. This is an ongoing problem that has not been solved.

[Rebels Haven]

Ditto here! I up-load 4 Gens in 5 minutes the the connection drops dead leaving 2196 to go! I won't live long enough to up-load these........ Everyone on our team is just about with all the continual problems....

[bwkaz]

He needs to send:

Everything listed in filelist.txt (or just fold_*, <handle>_* -- that's using the Unix globbing rules)
filelist.txt itself
handle.txt (unless he wants to let you take credit for the work... )
Possibly progress.txt, though I don't think so.

I don't believe there are any other files, though you'd want to see if you can get FoBoT to chime in too (he does sneakernet all the time, and I don't) before you delete those files off a client machine. Or, just try it but don't delete the files -- just to see if they'll upload.

[Chinasaur]

Well it's late in the UK. Fozzie will have to upload the entire zipped directory as adding those files only results in errors I can't get around with purgeuploadlist.

[bwkaz]

Yeah, I saw that after I posted this. Ah well, it looks like you two can figure it out pretty well tomorrow (though that might be cutting it a bit close).

[Chinasaur]

Well this sucks. Everytime I try to go at it I get errors if I do the purgeuploadlist. So the full directory is a must.

I wish one of the old hands at this crap had been around because I'm not up to speed on this @#$%.

[Paratima]

How old a hand do you need?

Fozz needs to send you:
handle*.*,
fold*.bz2 and
filelist.txt.
Add the standard release files, make a handle.txt file with the handle he sent you & pull the chain!

Just be prepared for a wait, 'cause the server is dead slow.

[Chinasaur]

paratima,

Also have the .val files he sent...

I can get it to start scrolling -

Uploading File 1/348 to server
Uploading File 2/348 to server....

gets to end and then says No Response from server..will wait

Then it overwrites the filelist.

Should I just let it overwrite? And then let it sit? how will it wknow what to send?

So I'm there but it seems something is odd..but I've never had to do this before.

[Paratima]

If it has indeed gotten all the way to 348/348, you're golden. I don't grok the problem...

It could be a slightly quicker timeout on the restart, which is what it tries to do at the end. If it actually sent all the structs, hop on over to IB's fridge, 'cause you're there.

[Chinasaur]

No it didn't send the strucs...it said when it got to the end that it could not communicate with the server and then it starts in crunching and overwrites the filelist.txt file.

so questions is ..Do I LET it overwrite the filelist.txt file and then try to connedct after it's done 10000?

Something ain't right here...

[Paratima]

Nothing personal, but you got PM.

[AMD_is_logical]

Originally posted by Chinasaur
it said when it got to the end that it could not communicate with the server and then it starts in crunching

When uploading, I always use the -ut switch. It then uploads and stops. If it didn't upload everything, then it needs to be run again until it finishes.

[Chinasaur]

Using -ut . Not that easy.

Thanks anyway.

[Paratima]

Nah, it was just plowing thru the structs w/o actually uploading them. Sounds like a filelist.txt problem... maybe. Fer shure, it'll have to wait till tomorrow. Long day. Sleepy... :sleepy:

[tpdooley]

with a cable modem (512k down, 128k up, 600+ ping) I noticed one of the two machines I uploaded today going as fast as it used to.. (about 5 secs per upload) and then a few structures later.. slowing down to about 20 seconds per upload.

are the servers overloaded because of end of protein folding uploads going on, networking problems at or near the DF server (possibly caused by MsBlaster clone infections in the area?)

[GHOST]

I had two boxes offline. One has a very fast cable connection and I was getting the same problem. It would upload 3-6 gens then slow down or stop completely and start folding again. Had no problems getting all the generations up except for the speed.

The second box was on a dsl connection with a different isp and took 8 hours to upload about 1300 gens.

[rstarr]

I was having the same problems uploading using DSL. How did I solve it? Well, it was ugly and took time, but I would start DF using only the DOS box and watch it upload, stop timeout and then crunch on without upload the complete list. I would stop DF using Q, wait a few seconds and start the DOS box again. I had to do this like 10 times every 30 mintues or so, but all work finally did upload.

I believe the problem will only get worse today due to everyone trying to upload their off-line boxes.

[Fozzie]

I think there must be something more to this than just traffic.

When it was congested last time it just either worked or it didn't. This is different the upload is stuttering fine then stop then fine again then stop. It seems to be irrespective of what connection you have but everyone's uploads seem to have been effected.

Once again no communication from the DF Team.

As far as looking after their contributors there really is a lot of room for improvement.

[Ned]

Howard,

The bottleneck with the upload process is probably the database update process. Perhaps the upload process could be modified to simply perform a minimal verification edit then a data store into a background process queue. This would perform the edit check required, and still allow acceptance of the data in a more timely fashion. The DB update could then run in the background at a low priority (perhaps on a different machine - depends on what DB you're using).

Ned

[Rebels Haven]

That works, rstar.. But with 4000 generations now buffered, I won't live long enough it to do all...

4 gens seems to be the average. Every once-in-a-while, I get 8-10 uploaded

[Corwin]

09/25/2003

- The next protein update is scheduled for Thursday, October 2, 2003, at 11:00 am EST.


No tentatively this time either.

IMO going ahead with the uploads in the state they are is madness.

Fozzie will lose all his work so will Rebels Haven.

I am sure the countless quiet folders will also be so disadvantaged.

I can't upload mine either at the rate I am allowed.

The short time I have been on this project I can't say I am too impressed with the I'm alright Jack attitude the project team seem to give off and those who never fold offline.

I'm off the project as of today I imagine many others will get discouraged the same way maybe not now but if the team continue to be distant from their own hosted forum I can't see how they will hold on to them.

Looking at registered to active membership maybe they have always been this poor.

[Rebels Haven]

Any explaniation for the very slow up-loads...?

[Stardragon]

A message has been posted on the news page. We are looking into the problem and apologize for the incovinience it has caused thus far. Please watch the news section for updates.

[Fozzie]

extend the upload of old protein wus, so that all the effected users can upload their gens without losing them.

We all know about the 24/48 hour cut offs but I think it wise to let it ride over the weekend due to the problems.

[Brian the Fist]

Originally posted by Fozzie
I think there must be something more to this than just traffic.

When it was congested last time it just either worked or it didn't. This is different the upload is stuttering fine then stop then fine again then stop. It seems to be irrespective of what connection you have but everyone's uploads seem to have been effected.

Once again no communication from the DF Team.

As far as looking after their contributors there really is a lot of room for improvement.

Please remember we are not in the same time zone of some of you. While it may be mid-day or evening for you, we are probably soundly sleeping. :sleepy: Thus don't expect to hear any responses from us before we wake up!! There is no need to panic.

Anyhow, if the problem cannot be resolved quicky we can postpone the protein change, that is fine.

Everything looks fine on out side here so I am puzzled so far. There may be some OTHER network traffic (worms?) bogging down the hospital network, but DFP is certainly not. We are waiting to hear from IT on what may be happening.

[Rebels Haven]

It is coming up on 24 hours that the DF server won't allow up-loads to complete. Doing the upload by hand, as rstar suggested, has allowed anywhere from 3 to 50 generations to be uploaded at once. Now for the last hour, 30 minutes, 2 Gens is it and most of the time they don't really get uploaded...


And that was on a T1 line. I managed to upload about 500 gens in 4 hours by hand, but have about 3500 to go...

Our team is dwindling fast as members are giving up because of the problems that occur each and every time the protein is updated.

[Paratima]

Here is where the user community lives! Would it be too much to ask to post here as well as on the DF site?

This is where we come for news, to find out if others are having the same problems we are, work-arounds. The DF site is, of course, the official home, but it's not where we think to go first. Besides, in times of problems, we are more likely to be able to get to this forum. This is where we "hang out".

All it takes is, "Hi, we heard there's a problem and we're working on it." We're just looking for a signal that you care as much about the situation as we do.

[Fozzie]

this issue was only 1 day old for me but Rebels Haven and others had been noticing it since the 27th.

Now I appreciate that was Saturday, however you and Stardragon are moderators here.

I moderate a forum or 2 too and I monitor each new post that is logged on the forum. It keeps me abreast of what is going on.

It may be worth you doing the same.

Alternatively we could always report a post to get your attention.

Either way, I would suggest that either yourself or Stardragon check the forum daily.

The better your response the better your retention of crunchers.

As we so often say all we need is a "Ok we are on the problem".

Keep your user base happy and you'll reach your goal quicker.

Postponing the changeover would be good for me but I merely would wish longer to upload my gens.

If that could be extended over the weekend then that would be great.

Whichever is easiest for you guys, I think everyone would rather we get on investigating the next protein rather than waiting. We just don't want to lose the credit for the work we have done.

[Rebels Haven]

Forgot to mention that on-line folders don't notice the 2 Gen max, limit, that is occuring because... well, you know....

[Fozzie]

Is it just me or is it getting better?

Just need another couple of days to upload my other 17 boxen.

Hope you can do the weekend extension Howard et al.

[Rebels Haven]

Fozzie, all my online-dial-up folders are uploading like crazy! Problem is over! Now get ready for the update!

[Brian the Fist]

Originally posted by Fozzie
this issue was only 1 day old for me but Rebels Haven and others had been noticing it since the 27th.

It may be worth you doing the same.

Either way, I would suggest that either yourself or Stardragon check the forum daily.

Whichever is easiest for you guys, I think everyone would rather we get on investigating the next protein rather than waiting. We just don't want to lose the credit for the work we have done.

Firstly I resent your comment about checking the forum. I've been reading every message on thus forum since it started about a year ago, every day, usually even on weekends and holidays. Now we both monitor it.

I have seen no indications of a problem posted in this forum since reading yesterday. Please note that we require at least 2 or 3 people to report an alledged server problem before we investigate, or we would never get any work done Once several people start experiencing the same problem, we investigate right away however, and post a message on the News page of the web site and usually here as well.

Ok, that feels better. Now then, we appear to have found a dead/broken web server which MAY have been causing the trouble, we have removed it for now. Please let us know if the network appears to be back to normal now. We will assume it is fixed unless we hear otherwise.

We have changed the protein change day to Friday, 11:00 EST so we have time to ensure everything is working OK again and to give people enough time to upload buffered work that got stuck.

Thanks for your continued patience (those of you who are still here, anyways)

[Rebels Haven]

Howard, it's up-loading like crazy now. Thanks!....

[Fozzie]

I am pleased to see you check the forum regularly. Now we all know that you do.

If you didn't see the issue as early as we did then that's one of those things.

It seems to be sorted and the gens are uploading at a much better rate.

Excellent news that the switch over is on Friday it will give those not in your neck of the woods plenty of time to get all the buffered work uploaded.

[MarcyDarcy]

Howard, i understand the reason to reschedule the next proteinupdate, but the time is not very handy for the users in Europe. Overhere the proteinupdate will take place at approx. 5 o'clock in the afternoon on friday. So there is only 1 option for me (and many more with me)...shut down the Distributed Folding computers. Because leaving them run offline will result is useless data considering the 100% and 50% credit rule for the next 2 days after a switch. Flushing on monday will be to late.

Leaving them online with the 'accept automatic protein/clientupdate' function on, is not an option.

[Chinasaur]

Howard,

We have been talking about this issue for months.

In this thread - http://www.free-dc.org/forum/showthr...urgeuploadlist

- that you replied to, the dial-up issue is explicitly mentioned.

I did a search on -purgeuploadlist and find numerous posts of people complaining about this issue dating back months.

We're glad you have tracked down "a" problem, and that you have understood the need to extend the cutover until Friday.

Hopefully that one server was the culprit.

We are not attacking you, it's simple frustration from not having the issue addressed/fixed in a timely manner, then to see it re-occur, and then be told we've never mentioned it before that gets us a bit "miffed".

Cheers.

[Ned]

Howard,

For the Off-Line folders, is there any way to combine all of the generations of one set into a "package" that could be then sent as one unit to the servers? A lot less load for the receiving end in terms of editing/ DB insertion. It would have to handle partial packages for those incomplete sets.

Ned

[HaloJones]

I'm with MarcyDarcy on this. 5pm on a Friday is lousy timing for European folders. It usually takes a couple of hours for the client to download smoothly and if there are any problems with it, it won't get fixed until after the weekend during which time youwill have severely pissed-off folders.

If you're going to postpone, please postpone til Monday.

[Fozzie]

just in case there are any hitches Monday would be far better.

Also those of us who run no-net boxen at work (firewall issues)we will lose a whole weekends work.

I appreciate you have been very accommodating on this already so if we have to go with friday then that's OK.

I am sure you would be heralded as a top geezer if it was changed to Monday.

It's no skin off the nose of all the Net connected home and work pharmers but it would seriously endear you to all those no-netters.

Cheers for what you have done anyway