USB Drive to Sneaker-Net harvest offline folding clients' work

[djp]

I warned you I'd do it, but I didn't think I'd have it done this soon...

Here are three .BAT files I've written to harvest work from offline Distributed Folding clients via a USB, Firewire or other hot-pluggable storage medium. To use these programs, do the following:
1. Install and test a folding client on an internet-connected PC.
2. Stop the above-mentioned client.
3. Copy all of the files in the folding client directory into an empty directory on the transport medium. I call mine \Harvest and I recommend the same for you.
4. On the transport medium, sort the directory view by "Date Modified" and remove any folding work files. These should be the most recent files added to the directory. Leave only the original folding client files, the handle.txt file, and proxy.cfg (if needed).
5. Copy the three .BAT files I've listed below into your \Harvest directory on the transport medium.
6. Locate a clock-time pausing program like sleep.exe and install it into your \Harvest directory on the transport medium. I have not included this utility because it is not my code to give away. There are quite a few programs available on the internet that will pause for a specified interval of time.
7. Customize:
a. Edit all three batch files to use your DistributedFolding.Org-assigned handle in place of mine
b. Edit the GATHER.BAT and RELAUNCH.BAT files to change the name of your Distributed Folding Client's work directory, if needed
c. If all of your offline DF clients store their work on the same drive letter, you can simplify the GATHER.BAT and RELAUNCH.BAT files so that they don't prompt the user to enter a drive letter.
d. If you have no clients with multiple CPUs, and a little bit of programming skill, you can optionally remove the portions of my code that deal with these special boxes. If you don't have the programming skill to handle this, don't worry; the multi-CPU stuff doesn't take much time to see that you've only got a single folding directory on your PC
8. Until you are confident that your configuration of these programs works, please make back-up copies of your work-in-progress. I would really hate to impede the progress of this project and/or cost you lost stats credit.

Limitations:
1. I assume that all of your offline folding clients will hold their work in a directory of the same name. If this doesn't suit your environment, you'll need to edit the GATHER.BAT and RELAUNCH.BAT files accordingly.
2. I have done no checking for available storage space on the transport medium. Before running these programs, please be sure that your transport medium has enough space free to accommodate the work that has been dome by the machine whose work you are about to harvest! (The DF client currently takes about 13MB, so if the distribfold directory on your offline folding client - 13MB > free space on your transport medium, stop and upload some work or clear-off idle files before you try and harvest this particular client's work.)
3. I have not included sleep.exe in this submission. You'll have to find your own timed-pause program.
4. I've assumed that you are running the folding client as a foreground application (as I do it). If you're running it as a service, there will need to be some changes to this procedure of stopping the client in the GATHER.BAT file and re-starting it in the RELAUNCH.BAT file.

Here is the program that gathers work from an offline folding client. I call it GATHER.BAT:

@ECHO OFF
REM
REM Usage: GATHER C
REM
REM This program gathers work from an offline Distributed Folding Client
REM computer for Sneaker-Net via USB Flash or some similar removable media
REM of reasonable capacity. It will expect the work files' drive letter on
REM the command line, or else it will prompt the user for one. This will
REM allow harvest from any drive letter without cumbersome code to iterate
REM through the whole stinkin' alphabet.
REM
REM It will figure-out the location of the Sneaker-Net medium on its own:

SET PWD=%CD:~2%
SET FOB=%CD:~0,2%
SET DDRIVE=%1

del /P %FOB%%PWD%\harvest.log

if NOT !%1==! goto GOTDRIVE
SET /P DDRIVE="Which drive letter holds the DISTRIBFOLD directory?"
ECHO.
ECHO Thank you.

:GOTDRIVE
SET DDIR=distribfold
SET INDEX=0
SET HANDLE=4s8cy36o

:DIRLOOP
if not exist %DDRIVE%:\%DDIR%\filelist.txt goto NEXTDIRLOOP

Echo Stop the folding client # %INDEX%
del %DDRIVE%:\%DDIR%\foldtrajlite.lock

REM Pause to allow Folding Client to close-down
:HOLD1
%FOB%%PWD%\sleep.exe 2
if exist %DDRIVE%:\%DDIR%\progress.txt goto HOLD1

%FOB%%PWD%\sleep.exe 3

REM Ready to move files
MD %FOB%%PWD%\%Computername%.%INDEX%
ECHO %Computername%.%INDEX% >> %FOB%%PWD%\queue.txt
move %DDRIVE%:\%DDIR%\filelist.txt %FOB%%PWD%\%Computername%.%INDEX%\
move %DDRIVE%:\%DDIR%\error.log %FOB%%PWD%\%Computername%.%INDEX%\
move %DDRIVE%:\%DDIR%\%handle%*.* %FOB%%PWD%\%Computername%.%INDEX%\
move %DDRIVE%:\%DDIR%\fold_*.* %FOB%%PWD%\%Computername%.%INDEX%\

REM Files are moved

:NEXTDIRLOOP
SET /a INDEX=%INDEX% +1
IF %INDEX% GTR 8 goto DONE
SET DDIR=distribfold%INDEX%
GOTO DIRLOOP

:DONE
ECHO.
ECHO.
ECHO All data are gathered from this PC
%FOB%%PWD%\sleep.exe 2

If you examine the code, you'll see that it first locates its own drive and directory information, then it executes a loop for every instance of the folding client. Inside the loop, it makes a directory for this computername and this folding client instance, adds a note in the queue.txt file, and copies all of the folding work into this directory.

The UPLOAD.BAT program is fairly simple:

@ECHO Off
REM Usage: UPLOAD
REM
REM This program sends work gathered from an offline Distributed Folding
REM Client or clients via USB Flash or some similar removable media
REM 
REM It will need no commandline parameters, since the GATHER.BAT program
REM will have already configured a queue of directories for uploading to
REM DistributedFolding.ORG
REM
SET HANDLE=4s8cy36o
SET PWD=%CD:~2%
SET FOB=%CD:~0,1%

FOR /F "" %%i IN (%FOB%:%PWD%\queue.txt) DO (
move %FOB%:%PWD%\%%i\*.* %FOB%:%PWD%
.\foldtrajlite -f protein -n native -ut -rt
move filelist.txt  %FOB%:%PWD%\%%i\
move %handle%*.* %FOB%:%PWD%\%%i\
move fold_*.* %FOB%:%PWD%\%%i\
)
ECHO.
ECHO.
ECHO If there was some sort of error in uploading, you may wish to keep the
ECHO %FOB%:%PWD%\queue.txt file so that you can correct the problem and try
ECHO uploading again.
ECHO.
ECHO If everything went well, you can delete the %FOB%:%PWD%\queue.txt file
ECHO now. Would you like to
del /P %FOB%:%PWD%\queue.txt

It first locates its own drive and directory information, then it executes a loop for every directory name in the queue.txt file. Inside the loop, it moves the folding files into its scratchpad area, uploads them, and moves the resulting files back into the directory from whence they came. Finally, it allows you to retain the queue.txt file if there was some error that needs manual correction.

The RELAUNCH.BAT program is also pretty simple:

@ECHO ON
REM
REM Usage: relaunch C
REM
REM This program returns work from an internet-connected uploader via USB
REM Flash or other removable media. It will expect the work files'
REM destination drive letter on the command line, or else it will prompt
REM the user for one. This will allow harvest and return from any drive
REM letter without cumbersome code to iterate through the whole stinkin'
REM alphabet.
REM
REM It will figure-out the location of the Sneaker-Net medium on its own.

SET PWD=%CD:~2%
SET FOB=%CD:~0,2%
SET DDRIVE=%1

if NOT !%1==! goto GOTDRIVE
SET /P DDRIVE="Which drive letter holds the DISTRIBFOLD directory?"

:GOTDRIVE
SET DDIR=distribfold
SET DEST=%DDIR%
SET INDEX=0
SET HANDLE=4s8cy36o

:DIRLOOP
if not exist %COMPUTERNAME%.%INDEX%\filelist.txt goto NEXTDIRLOOP
move %COMPUTERNAME%.%INDEX%\*.* %DDRIVE%:\%DEST%\
rd %COMPUTERNAME%.%INDEX%
%DDRIVE%:
cd \%DEST%
start /MIN cmd.exe /c .\foldit.bat
%FOB%

:NEXTDIRLOOP
SET /a INDEX=%INDEX% +1
IF %INDEX% GTR 8 goto DONE
SET DEST=%DDIR%%INDEX%
GOTO DIRLOOP

:DONE

If you examine the code, you'll see that it first locates its own drive and directory information, then it executes a loop. Inside the loop, it returns the remaining files from the upload back to their original source directories. It then re-launches the folding client in a minimized window.

[[DPC]Mobster]

Looks usefull, but as long the 'Generation 337'-bug isn't resolved I will not run DFolding on my offline clients. It already costed me 180k points and within DPC it costed us over a million points...

[djp]

Regular harvesting of the work from your offline clients will prevent the Generation 337 bug from ever becoming a problem for you.

It isn't that the client stops at generation 337 and leaves you having to start over; the client stops when it reaches 337 generations buffered. That's a few days' work, even for a very fast processor with unusual luck in folding.

[[DPC]Mobster]

I started a XP2700+ on friday just before I left the office. When I returned on tuesday I was unable to upload them. Even with all the tips from everybody. And going through those threads I've seen that it is not only me having the problem

I think that IS a problem... but we'll all have to be patient I guess...

[Ned]

If Your machine is so fast that you cannot upload the results from an offline machine with your current schedule, then RUN MULTIPLE COPIES in those machines. Each copy runs that much slower, you can download the results in time to avoid the Gen 338 problem, and your data does not get wasted. You also get more chances to get those great results!!!

Ned

[[DPC]Mobster]

Cheers Ned. Somebody else also gave that suggestion. I'll see what I'll do.

Still, it would be easier to just run the client normally. Still stuck with 180k points though...

[Ned]

For experimentation sake, I tried to use two copies in a W2K machine. DON'T BOTHER with that effort. While the second copy gets some computer time, in general, one copy will get MOST of the time EVEN THOUGH I SET PRIORITY EQUAL.

I had been using two copies in a Linux environment for quite a while now. Linux divides the CPU up almost exactly evenly which I foolishly expected W2K to do...

Ned