Printable View
Nearly all of my machines have no internet connection. They are managed via pure sneaker netting with a jumpdrive. In Phase I all I did was copy the filelist and completed units to the jumpdrive and then into the internet connected machine to upload. Is this still going to be possible in Phase II? I have looked through the threads but have not seen it specifically addressed, unless I missed it somehow.
It will still be possible, but it will be a little more complicated. The filelist.txt is always there now, even just after you have uploaded. Where before you could just copy all of the data files and the filelist.txt and then delete them, now you have to deal with the filelist.txt changing after the upload but still being around. Sample filelist.txt:
./fold_1_handle_26_handle_protein_25.log.bz2
./handle_1_handle_protein_25_0000027.val
fold_1_handle_48_handle_protein_26.log.bz2
handle_1_handle_protein_26_0000049.val
CurrentStruc 1 51 124 26 1 49 7.287 -1923.725 -118.015 -830.829 41623948.000 1.450 2.700 1337.566 ------------------------------------------------------------------------------------------------
91c45dc6795e6cd38262d53c2aa0b109
I think that to start with you would have to watch what happens on a system that is connected to the internet when you upload and see which files it leaves behind and what the filelist.txt looks like after the upload.
Actually, I just remembered something that will make it harder. You can't edit the filelist.txt because it uses a key to prevent that. So what you would have to do would be stop folding on the unconnected machine, copy all of the data files and the filelist.txt and move them to the upload machine. Upload the data and then move the filelist.txt file back to the unconnected machine and then delete all of the files that got uploaded.
I don't see any way of moving the files on to your drive and then just starting that machine back up, unless you started over on a new set.
I think in order for the new system to continue to be on the forefront of sneaker net distributed computing there would need to be a new function. Some way to upload the data into a folder and then leave the working folder in a state just like it uploaded to the server.
./foldtrajlite -n native -f protein -ut -x:../results
would put the upload part in the results folder and leave the working folder ready to go.
it's a little more complicated than that:
You must not delete filelist.txt (just copy it across).
Same goes for the last .val file in filelist.txt.
That's about it, I think.
In phase I a pair of data files for upload are generated for every # no. of randomly generated structures. The default for # is 5,000.
In phase II the concept of 'generations' is introduced. A pair of data files is generated for each generation. When you stop the client and upload the results the pair of data files for the generation that was being worked on when the client was stopped will be left behind.
Therefore, you'll have to return this remaining pair of data files as well as filelist.txt back to the -nonet machine before restarting the client.
If you were to delete all pairs of data files as well as filelist.txt on the -nonet machine and restart the client it'll start at generation 0 again. This is not what they want. The results of the previous generation are used to start the next generation. The idea is as more generations are produced (hopefully) the better the results. I believe that when generation 250 is reached it'll start the process all over again with generation 0 using a different seed.
Shortfinal
Are these machines on a LAN (Local Area Network) and have access to the one system with Internet access? What OS are you running, (Windoze, UNIX, etc)?Quote:
Originally posted by MgKnight
Nearly all of my machines have no internet connection.
If they have network access to that one system you could set that one system up as a proxy server.
Shortfinal
No, they are all stand alone machines. I am guessing that one viable solution would be to copy the entire directory over, upload and then copy the entire directory back to the original machine. By doing that I will not need to be concerned about deleted and or saved files. That will not be too big a problem as they are all in my office. I haven't networked anything because my theory is that that expense is better put to work by adding another machine. All of them are running WIN2K Pro.
Can't go wrong with that approach.Quote:
Originally posted by MgKnight
I am guessing that one viable solution would be to copy the entire directory over, upload and then copy the entire directory back to the original machine.
Shortfinal
Try checking out this thread also: http://www.free-dc.org/forum/showthr...&threadid=3177Quote:
Originally posted by MgKnight
Nearly all of my machines have no internet connection. They are managed via pure sneaker netting with a jumpdrive. In Phase I all I did was copy the filelist and completed units to the jumpdrive and then into the internet connected machine to upload. Is this still going to be possible in Phase II? I have looked through the threads but have not seen it specifically addressed, unless I missed it somehow.
Best,
RS
I'm fairly lazy. I started to write a couple of batch files for you that would gather work onto the Jumpdrive, organized in folders based on %COMPUTERNAME%. It looked like a lot of work, though, and I wasn't really interested in debugging my code. Then, I had a more brilliant idea:
M0ti has already done most of the work for you and he's working on debugging the rest. I haven't seen the next version of dfQ yet, so I'll describe how you would work with the current version. The dfQ program was made to set-up a client, a server, and an uploader, but you can run all three modules on each individual machine and it will package the data for you in a friendly manner. Reading the documentation for dfQ is also essential; otherwise, my configuration file info below will make no sense.
I wasn't sure exactly what your "jumpdrive" was. I'll assume it's a hot-pluggable USB storage drive so that I don't have to add a bunch of start-up and shut-down instructions to the stuff below. If it's actually a parallel-port hard drive or some other device that can't hot-plug, I'm sure you'll be able to figure-out the specific details for it.
*Configure the dfQClient to upload to localhost two or three times a day. I recommend this "sending" the data several times during the day so that the time you gather data from a particular machine makes less of an impact on your overall sending burden. The dfQClient will be in charge of launching the Distributed Folding client and sending its results to the Server program running on this same host. Configure the client to:
localhost
7070
480
in order to "send" the data three times daily. If the Server isn't running when the client "sends" data, the client will buffer as you've configured in the foldit.bat file. It's a good idea to run the Server program constantly in order to have it package your client's work into more managable files.
*The dfQServer's configuration file holds the TCP port assignments by which the Client program and the Uploader program talk to it. The server will run constantly and it will package the work done by the client program into a .DFQ file whenever the client "sends" its data. (The files will be found in the SERVER directory.) It will make these files available to the Uploader if you want to use it, or you can sneakernet these .DFQ files as they stand.
*The dfQUploader program on a folding client will gather all of the waiting .DFQ files and compress them into a single .A_DFQ file if you would like to take less space on your Jumpdrive. It should be configured to:
1
0
0
localhost
7071
600
and you will only run it when you want to compress the work for transport. Exit the Uploader program when it has finished building the compressed file and deleting the .DFQ files.
When you've gathered all of the .DFQ and/or A_DFQ files from your several client machines onto your Jumpdrive, copy them over to your Internet-Connected machine's UPLOADER directory. If you're also running your internet-connected machine as a folding box, configure the client and server as above. The desired configuration for the connected uploader will depend on whether you want it to sit around and upload files all day or just upload on demand. To avoid personal confusion, I recommend Exiting the Uploader before you copy work off the Jumpdrive.
The first time you upload work, I suggest only copying three files off your Jumpdrive into the UPLOADER directory. If you're particularly paranoid, you can run PERFMON to watch the Total Bytes/Second transferred over the network. You should see three pulses of Internet activity as the structures go off to Canada. After a .DFQ file is sent, the Uploader will delete it. If you're not going to have the uploader running constantly, set a long polling interval for its connection to the server and manually Exit the Uploader when it has finished sending your work.
We will try to put some clear instructions for 'sneaker-netting' in the readme and/or an FAQ after the new client is released, so don't fear.
The next version will have readable config files, amongst other desirable features.Quote:
Originally posted by djp
M0ti has already done most of the work for you and he's working on debugging the rest. I haven't seen the next version of dfQ yet, so I'll describe how you would work with the current version. The dfQ program was made to set-up a client, a server, and an uploader, but you can run all three modules on each individual machine and it will package the data for you in a friendly manner. Reading the documentation for dfQ is also essential; otherwise, my configuration file info below will make no sense.
Howard's been giving me a lot of help with the next version, so hopefully that'll be ready soon.
WOW!! Having been around since the start of the project I know that way back then it was pretty much "Ask and ye shall receive." It is truly amazing that this attitude continues today. :notworthy
Thank you all!! I'll wait patiently for the changeover.