return work on CD-R?

[FoBoT]

you think howard would accept return work/structures on CD?
these uploads are killing me! i am going to have to switch back to ECCP soon , very tiny work output and easy to return/no-net

[IronBits]

FoBoT - Do the best you can with each project.
This one has HIGH bandwidth usage.
If that is not an option, then pour it all into ECCp and take 1st place there! :jester:
You done good, no one can fault you.
U da Man :notworthy
Thanks for all you put up with to generate such awesome numbers.
You even made xj10bt cringe in fear :D
:thumbs:

[Paratima]

WHOA, BRO! Don't stop now! You're our guiding light!! :eek:

How about if I sent you a 100 foot CAT-5 cable? Would that do it? 200 foot?

A wireless access point & 37 wireless PC cards?

How about a nice Zip drive?

IronBits is right, of course.

[Chinasaur]

I'm with Paratima...

What do you need and how much $$$ do you want to keep up your production? :)



:cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers: :cheers:

[Scoofy12]

just mail it to me, I can upoad it at school...


what? of course it will be under your handle .. ;)

[FoBoT]

Originally posted by Paratima

How about a nice Zip drive?


nice zip drive? there is no such thing , they are the devil's spawn, evil , evil, eeeeevil ;)

[Paratima]

these uploads are killing me!Hey FoBoT! Just out of curiosity, how HAVE you been doing it? A hundred-pack of floppies?

Eunuch scribes? Parallel Winmodems? :confused:

[FoBoT]

laptop

i create a mondump or tuedump folder, move it to the server in the test lab i am harvesting

then go to each workstations, shutdown df using dfGUI, move the "full" df folder to the tuedump folder on the server, rename it "a" (then "b", then "c", etc) , then copy a fresh/unused df folder with dfGUI included with all the .ini settings already setup back onto the workstation , and fire it off again

then onto the next workstation until all of them in the cluster have been refreshed/harvested.

then i move the tuedump folder back onto the laptop from server and move to the next lab/cluster

i have 4 groups i have to do this with, i try to do it monday/wednesday/friday, but work priorities often take precidence, so i get to it when i can

once the laptop has all the "fruit" of the day, the ***dump folder is usually between 300-900MB , i take the laptop home and setup on my LAN and send it out through my cable modem in the evening or morning while i sit and surf/build/break/fix pc's

gack!!

[IronBits]

:scared: :scared: :scared:
:notworthy :notworthy :notworthy :notworthy
Holy moley FoBoT !!!
Right on! :thumbs: Your a :crazy: bastage. ;)
Have a few :cheers: - but only a few now, we wouldn't want it to interfere with the harvesting or nothing like that. ;) :D
:smoking:

[Paratima]

gack!! is right! Mama mia! :eek:

I can sure see how that could be a bit of a bummer.

Or, in the eloquent words of IronBits, :notworthy :notworthy :notworthy :notworthy

No wonder you're pooped! Grab a :cheers:

And watch out for anyone who says "of course it will be under your handle .. "

[IronBits]

Where is that Dyyryath????
/me wonders if he can fix his "launch and harvest" application he used in another project last year...
That must take you hours to do that FoBoT!
/me :sleepy:

[pointwood]

FoBoT, did you read the last comments from Howard in this thread:
http://www.free-dc.org/forum/showthread.php3?s=&threadid=1549

If he considers it okay in regards to security, the uploads may become a lot smaller when we change to the next protein!

[FoBoT]

yes, i did see that, thanks :)

[Scoofy12]

Are you having to run df/dfgui for each folder you want to upload? like for each "a", "b" etc you have to run DF separately? i wonder if we could write a little script or program that could consolidate all the results for one big "megadump" that could be more easily transported and uploaded


And seriously, if you really dont wanna upload, you can mail me a CD-R. i have access to a nice big multi-megabit uplaod pipe here :)

[FoBoT]

yes, i take the laptop home, plug it into my home LAN, open folder "a" , start df, it uploads all the structures from that folder, i shut down df, delete that "a" folder, open "b" and do it over and over until they are all sent

i am not sure how you would combine them into a single folder, since the .bz2 file names could be identical, although that wouldn't happen often

[FoBoT]

my resolution is as follows until howard reduces the volume of the return

run df on the -nonet boxen from friday afternoon until monday morning

switch them to eccp when i harvest the df results on monday, run eccp from monday to friday afternoon, when i harvest the eccp dplists and start df up for the weekend

that way i only have to do a single large dump from these -nonet boxen on monday evening

i think that is the best i can do for now, we already lost #1 and my output can't get #1 back anyway

[safemode]

there's no need to make the uploads less. Just thread the program. You have a simple thread that gets woken up when N amount of things are completed. This thread uploads N and downloads N + whatever you happen to be up to completed processing. This should keep a regular interval of uploading and downloading even though the amounts downloaded isn't always the same depending on how much you processed while uploading but you always process data 24/7 despite uploading or downloading.

so you always have X - N amount of work to process where X is at least N + amount you can process during an upload. I dont think that's very hard to impliment.

[bwkaz]

That sounds decent on paper, but these are the first couple of things I thought when I read it:

1) It doesn't actually reduce the bandwidth requirements, and the bandwidth requirements are the problem. You will still upload just as much data (actually a little more) per day as you do now. It's spread out a little more, but that might actually be bad.

2) The client doesn't download work. The only things it ever downloads are updates. All work is done by randomly choosing numbers on the client end, and sending the resulting structures to the server. Semantic nonsense probably ( ;) ), but just so you know.

It would be a good idea if it reduced the bandwith requirements (and, incidentally, if every one of the systems it ever ran on were on a 24/7 net connection, which will never happen), but since that isn't the case, I'm not sure it'll happen.

It would be nice for those of us that do have 24/7 net connections, to get a little extra work done, but I have a feeling it wouldn't amount to a ton. A few seconds per upload maybe (... at least, that's for me, YMMV depending on your upstream bandwidth limitations).

[Paratima]

I have envisioned a scheme like that. You have your basic client running no-netted and when it finishes & moves on, you "harvest" the result & result log files, ship them off to another process/box/whatever for uploading. It has the potential for some goodness. Maybe. I think.

The point here is that FoBoT's clusters have no net connections. Therefore, he's already running his basic clients at max warp.

It's not a question of upload efficiency, although as bwkaz points out, it probably wouldn't make enough difference to be worth the effort. The problem is physical. How to get millions (in his case) of finished structures on that northbound train. :D

[MAD-ness]

DfGUI is open source, so if someone wanted to modify it to do whatever they wanted, I am pretty sure they would ok with that. I can't remember what license he released it under, but I had assumed it was just gpl.

While scripts won't decrease the amount of bandwidth that Fobot must use simply to UPLOAD his results, I would think that someone with sufficient skill/time could code up something for him that he could modify for his particular networks which would basically let him plug into the server on each "network" launch the script and then have it all done automatically.

Perhaps if Fobot's production decreases someone here will find the motivation to help him out. ;)

Or maybe someone elsewhere already has a solution or would make one, Fobot is not the only large scale farmer and the same type of scripts he would need would come in handy for smaller farmers and perhaps even on other projects.

Anyways, don't blame you for not wanting to do SO much work and so much uploading Fobot. You are putting up huge numbers, but the sheer amount of data you are having to transfer is really ridiculous.

Gl either way. :)

[FoBoT]

safemode, no offense, but i don't know what you said, i guess i am not that smart

as the others pointed out, my problem is the sheer volume that has to be returned to get credit, its just too much.
the script stuff i could probably do if i wasn't so busy at work right now, but that isn't really the problem, its the amount i have to upload at home on my cable modem. i think my upload speed is 256k bps which under normal circumstances is quite nice, but sending 900MB like i had to on monday took 2-3 hours

i would still like to see something like SETI queue, although i don't fully understand what SETI queue does, my understanding was that it could act as a middleman/central point for client pc's to send work to , which then in turn sends it up to the home office

the daemon that howard added for a common download source for farms is the right idea, except its only one way (down, not up). if that could be altered to handle work back up to DF , then it would be more useful

oh well , we are over half way to 10 billion structures, so we should only have about two more weeks on this protein, maybe the next one will generate less volume of return traffic

if not, there's always

http://www.talonman1.com/hari_kari_2.JPG

;)

crunch on! :D

[MAD-ness]

Fobot: do you mind if I quote your comments in this thread to Howard on the official forum in a thread about getting the proxy to go both ways?

Howard is a good programmer, a good scientist and great at fixing bugs, but he also is rather conservative. If he doesn't see a significant and currently existing problem, he will not be in favor of making changes or additions to the client (most of this is probably due to time constraints and not wanting to add to his current workload). However, every time we have shown him a legitimate need for the project and client to have a feature and/or fix, he has begun working on it once he acknowledges the need for it.

So, if guys like you who are producing massive amounts of work can show an actual need for a certain feature, it is a LOT more likely to convince him.

Perhaps if you do a quick calculation of how much your having to cut back is doing to decrease your production on a weekly basis, it might sort of get the point across. :)

When Howard was undecided on the true "need" for a specific feature, the opinion of Jodie or MilleniumGuy2K1 was often a strong factor in his final decision (IMO, only, of course, as I do not know this for a fact).

[FoBoT]

sure you can quote me, these posts are public
i posted in that thread a week or so ago, as you said, howard wasn't too worried about the whole thing

you can check my weekly production on statsman.org and compare it now, ~49 million to next monday morning, before i do my next big dump. whatever it is then, will be about what i get from my connected boxes at home and a small portion at work that have internet access

so the difference will be the potential loss due to the huge volume of the return structures, about

[MAD-ness]

Fobot: I have been paying attention to these concerns on various parts of the client upload bandwidht requirements and the only aspect which looks like it has much hope for being implemented soon (or easily) would making the proxy work both ways.

I will sort of see what I can find from other major producers, see who has a problem with this (obviously some of these guys have large leased lines and either don't notice the demands or don't care).

[Darkness Productions]

You all are talking about a program that will harvest and upload the DF stuff. I thought WinKDFold had this capability... And if not, I bet we could convince Smiling Carnivore to add this functionality to it. At least that way, FoBoT wouldn't have to go to each machine individually...

[Darkness Productions]

FoBoT - also, like Paratima said (I think it was Paratima) if you want to mail a CD-R(W?) to someone to upload it, that could prolly be arranged easily. How much bandwidth do you think your boxen use on a daily basis just for uploading DF results?

[safemode]

i really dont see how you have upload problems unless you're dealing with a little network of computers working on folding@home exclusively. I only see 60K/s for a few seconds everytime i upload. It's not even noticable. upload requirements would only become an issue if you used a 56k modem. Otherwise it shouldn't be a problem to upload frequently. My suggestion about threading was simply to negate the problem with having your client doing nothing during uploading, even though it only takes a few seconds on anything better than a dialup modem. But from the thread i've gathered that the person complaining about 900MB uploads has a cable modem....how do you accumulate 900MB ...i think you need to increase your upload frequency. A little upload frequently is better than a big upload every now and then. It avoids the saturation aspect on your net connection. Who cares how much it is over time.

[bwkaz]

He's running nonet, because his clients (at least the ones where he works) don't have 'net connections. He lets them accumulate over a couple days, and gets 900MB of results that he has to manually upload from his house.

These results are coming from a fairly large number of workstations, with no net connection... so he's sneakernetting. Which is getting old / tedious / slow. I don't really blame him, actually... :(

Edit: Check out his daily production a minute, courtesy of Dyyryath's stats:

http://www.zerothelement.com/fdc-stats/distfold/index-1.shtml

He's #1 on the team (for the moment... unless xj10bt overtakes him again).