dfQ Release Candidate 1 now available

[m0ti]

Hi Everyone,

dfQ is now at RC1.

Here's what's new:


1) Fixed DF Client Hang Bug.

2) More robust error-handling:
1) Correction of common error in filelist.txt
2) Automatic filelist.txt creation for orphaned .bz2 files
3) Creation of zip files for .bz2 files with no match

3) File-formats slightly change (needed to support orphan recovery).

A brief summary of features:

- Computers without an internet connection can upload results to a server

- From the server they can be automatically uploaded (if it has an internet connection)

- If the server has no internet access all results (unlimited number) can be stored in a single file

- File can be uploaded from any computer with an internet-connection

- Reliable: nothing gets deleted until it's saved somewhere else

- Can run hidden

- Automatically handles orphaned .bz2 files

- Works with other applications (dfGUI, dfDetect, etc)

- Cross-platform (Win/Linux/other)

dfQ can be downloaded here

[MarcyDarcy]

great work
I'm soon going to test it out

just a little question. This also means there will be memberstats? As in: multiple users who are flushing under the same name and they 'flush' to the dfQ it can generate IP-bases userstats?

[m0ti]

In the current version it doesn't do stats, just basic logging.

I guess I'll count those as feature requests:

1) Saving stats (num-structures by IP, num-structs per user, per day, etc).
2) Serving up stats (simple web-server? or auto-generate an .html file to a specific location?)


It shouldn't be too hard to start saving this information.

I could also add in support for multiple handles; just remember that you only want to share your handle with someone you trust! Any requests for this?

---------

If anybody else wants these (or other) features let me know!

The more people press for specific features the more likely they are of being included.

[MarcyDarcy]

wow that's great

i'm going to post it over at our teamforum and i know some big subteams are very interested in this new features

i think this tool can help finding new users for the project

[m0ti]

Seeing as I'll be doing stats I'll also implement a mini-team feature.

[IronBits]

Says the zip file is corrupted

[m0ti]

Oops... I made a couple of minor bug fixes and rereleased it, guess there's a small problem...

Guess some problem when I uploaded it.

...

Ok, all fixed! Thanks for the heads up!

[IronBits]

Thanks!
Will it handle the DFbeta clients?

[m0ti]

Currently it doesn't support the beta clients.

I will of course be supporting them in the future, since they will start being the regular clients.

I've already planned the move and have updated some of the code appropriately.

The next release (rc2) will probably be configurable to run either the beta or the regular code.

[matrix_fan]

what is it????(dfq)

[Welnic]

Originally posted by Opteron Me
what is it????(dfq)

Pertinant info from the readme (edited):

dfQ - provided by Team Anandtech

dfQ is a queue program for uploading results from the Distributed Folding client.

dfQ should only be used under one user handle. Several people folding under several different handles should not use the same installation of dfQ.

At this time dfQ is only being released for Windows, though it can be easily used in Linux as well (see below). dfQ requires a Java installation on the machine.

[matrix_fan]

Thanks...That will be REALLY useful;l....Thanks alot for taking time to tell me what it is

[djp]

I feel so dumb, but if I've got the question, I'm sure someone else does, too....

I understand that dfQ requires Java. All of my client boxes are running Win2k with IE 5.5 (including the Microsoft Java Virtual Machine).

I've got about 100 machines on a couple of internal networks that are not connected to the Internet, so I'm really excited by the prospect of puting them to work via dfQ. I wrote a collection of .bat files to harvest the results manually, but I've got to baby-sit the process and I have written no facility for recovery of bad files. dfQ could save me some serious blocks of time!

When I launch dfQclient.bat, I get an error message telling me that 'java' is not a recognized command.

How can I get my isolated client boxes working with the least amount of additional file downloading and/or program installation?

[Digital Parasite]

Your MS IE 5.5 with Java VM only allows you to run Java applets within IE I believe. You will need to install Sun's Java runtime (JRE) which can be found here: http://java.sun.com/j2se/1.4.1/download.html

Or for other versions of it here:
http://java.sun.com/j2se/downloads.html

Jeff.

[aptarasc]

Originally posted by djp
I feel so dumb, but if I've got the question, I'm sure someone else does, too....

I understand that dfQ requires Java. All of my client boxes are running Win2k with IE 5.5 (including the Microsoft Java Virtual Machine).

I've got about 100 machines on a couple of internal networks that are not connected to the Internet, so I'm really excited by the prospect of puting them to work via dfQ. I wrote a collection of .bat files to harvest the results manually, but I've got to baby-sit the process and I have written no facility for recovery of bad files. dfQ could save me some serious blocks of time!

When I launch dfQclient.bat, I get an error message telling me that 'java' is not a recognized command.

How can I get my isolated client boxes working with the least amount of additional file downloading and/or program installation?

check your PM

[djp]

Sorry, I don't speak acronym well. What is a PM?

[aptarasc]

Originally posted by djp
Sorry, I don't speak acronym well. What is a PM?

PM = Private message

Too bad considering you work for a company that prides itself on acronyms

[FoBoT]

download and install the Sun Java runtime environment
all of my dfQ testing was done that way

after you install the Sun Java, windows will then know that the "java" line is to execute a .jar file , etc, etc

[bwkaz]

FoBoT -- does the JRE (or JDK; either one should work) installer edit your PATH, too?

[djp]

OK, I've reluctantly loaded the Sun JRE on my folding clients, server, and uploader. I had hoped to install as little as possible on these machines so that I might hope to have minimal impact on their performance.

I haven't tried loading the dfQ package on any of my multi-CPU boxes yet. Is there any feature in dfQ for keeping all 8 microprocessors busy on my bigest box? Since I've only got a couple of machines that hot, is there at least a chance of keeping both virtual processors in a hyperthreaded box busy?

-djp

[m0ti]

You should get the JRE, since you don't need the JDK.

I believe that it does alter your path for you.

<If after installing java you still get the unrecognized command you can do the following:

Write down the full path where you installed java. THis should look like:

C:\Program Files\JavaSoft\jre1.4.1
or something similar. This directory should have a sub-directory called "bin".

What you want to write down is:
C:\Program Files\JavaSoft\jre1.4.1\bin


Go to control panel, and open up System.

Go to advanced, and click on the "Environment Variables" button at the bottom.

Find the PATH variable in the list at the bottom. Hit "Edit".

Go to the end of the list (second text input area). If there isn't a semi-colon (";") at the end of the list, add one, and then enter in the path you wrote down (C:\Program Files\JavaSoft\jre1.4.1\bin). Hit OK, until you've closed everything down.

Now you're good to go!>

There isn't any feature yet in dfQ for running multiple instances of DF. You'll have to use multiple dfQ clients on a multi-proc/hyperthreaded machine. Note that since they share a library the memory consumption shouldln't be too high.


In the next version (rc2) which will be ready sometime (haven't had any breathing time in a long time), I'll add in a feature to the client where it can run multiple instances of DF. Also you'll be able to run dfQ as a service.

I had been hoping to get rc2 out sometime in May. (Un)fortunately I've got a new interesting project which is very time consuming, so it may be until July before I can get it out. Overall though, I think I should be able to do an August release (for version 1) which should also support multiple-handles and stats.

[djp]

Originally posted by m0ti
There isn't any feature yet in dfQ for running multiple instances of DF. You'll have to use multiple dfQ clients on a multi-proc/hyperthreaded machine. Note that since they share a library the memory consumption shouldln't be too high.

OK, I can run a second client, but how do I get it to send its results to the Server machine?

A second plain client with -if will hold the results locally. (If I had it directly connected to the Internet, I would have no need of dfQ, no would I?) The dfQclient.bat file invokes the java program where I thought the client's pathname was hard-coded.

-djp

[m0ti]

Oops, yes it is hard coded.

Hmm.

In the meantime, just place multiple installations of dfQ in two different directories.

I'll do my best to do get out some sort of an rc1.1 which will do SMP properly (and service installation).

[djp]

I've got quite a folding farm on my hands. . . .

Imagine a network with so many dfQClients running that the one poor little dfQUploader is on the edge of being over-run with data. (During the interval between one polling of the Server and the next, more work has been generated than the uploader can transmit in the next interval.) I'm not at this point yet, but I haven't finished converting all of my folding clients to dfQ from my older system.

I can't divide the work between two uploaders (because I may not put another box onto the real Internet). I will need to balance my workload better. I can see two variables that I can change: The time between the dfQClients' uploads to the server and the time between the dfQUploader's collections of work from the dfQServer.

I'm hoping that if I push the dfQClient's upload interval out to a longer period I will have fewer partial batches of data transmitted by each dfQClient in a day, resulting in fewer work units per client per day at the dfQUploader.

Does this sound reasonable?

To make things a little more complicated, many of my dfQClient boxes are not folding during the day-time due to their dedication to "real work." Because of this dual use, I end-up stopping and starting a large block of clients simultaneously, but some days it will be Group A of clients that get switched and other days it will be Group B, C, D, or whatever. The switch-over time will also vary. Will the fact that several clients will have large workloads to drop onto the dfQServer at the same time complicate anything, or is the Server code robust enough to handle several concurrent transfers? If the Uploader has a spike of too much work between its Server-polling events, will it freak-out?

I also have SMP boxes with 2, 4, and 8 CPUs running n-instances of the dfQClient. If I start all the instances on a CPU at the same time, should I use different upload intervals on each client to avoid the possibility of the client's several streams of data getting confused at the server?

[m0ti]

First off, if the one uploader is getting hit hard, probably the best thing to do is to increase the amount of time between uploads to the server from the clients. In any case, there is a fixed overhead to the transfer.

The server should be able to handle a large amount of clients. I've never tested how many simultaneous clients it can support but it should be fairly large (guess I'll have to add in a thread pool to the next version for people with very large farms). And I might have to add in a server-bridge or something (clients can upload to it, it uploads to another server), though that should only be for very very large networks. In its current form, it should be able to support a fairly large number of clients. If a client can't contact the server the work will remain at the client location and it'll try again the next time an upload is scheduled.

Overall, the whole system should be robust enough that there shouldn't be many problems. The uploader is the largest potential bottleneck in the system IMO, so extending the time between uploads from the client to the server is the best way to go.

I'll update the uploader too for the next version, so that you can set it up for multiple concurrent uploads to DF. I'll set a max of, say, 20 DF directories under Uploader. It'll use these to upload.

Ok, planned new features for next release (v1 rc2):

1) Stats is getting pushed back a bit. Stats is more of a convenience than a feature resulting in more work getting pushed out. The only thing that'll make it in is a listing of IP's / host names and a timestamp for when their last peice of work has has been received. This'll most likely just be a text file on the server.

2) Service installation

3) Support of multiple handles

4) Support of SMP on the client

5) Support of multiple servers for client and uploader

6) Multiple concurrent uploads from the uploader.

7) More robust server.

8) Improved (more reliable) reporting and log size kept down.

rc3 will see stats in some form, and likely some bug fixes. rc4 will be an absolute feature freeze, and from there it will be any bug fixes, and then v1 will be officially ready.

[djp]

Minimal stats are nice for more of us than just the registered Stats H... er... uh... Enthusiasts.

I've got a few machines that seem to be at the same point every time I check on their progress. It could be that my "free time" to check them happens to hit nearly the same point in their compute- send- clear- compute-again cycle. It could also be that these particular machines are so slow (relatively) and I've set such a long interval between transmissions to the server that any time I looked, I would always see them working on that first batch of 5000. I suspect they might just be sitting idle. If I had a text file at the Server or the Uploader with just a basic listing of IP addresses, timestamps and volume of work submited, I could build a database or spreadsheet to tell me the rest of the fancy info I might desire..