DFProgress

[MerePeer]

Want to monitor all your DF folding machines in one screen? Here's an application called DFProgress that does that, and it is portable and only dependent on the java runtime (1.4.2 or higher).

There's a 0readme.txt in the zip to help you get started. I'll post updates & fixes by editing the attachment on this first post so that you have a quick place to find the latest.

Mar 28 update:
- newer zip in subsequent post in this thread; version 1.02
- new STATS panel shows on display. It grabs your stats + 2 people above you from macnn pg. Configure your username using a new DFProgress.ini file you should create. Add one line w/your username, like "username=MerePeer". This panel updates every 15 mins by default. See 0readme.txt for 3 other new .ini file options.
- Update points... option no longer uses blueprint team page, now grabs exact latest points as well as your latest total standing and when that was posted.
- Note: The Options menu items may block activity briefly if the web sites they are hitting on are slow. It may timeout as well.
- Updated DFProgressNode.reg, the AppParameters is supposed to begin with -Xrs so if you already installed a node as a win service you'll need to modify your reg entry.
- Note: DFProgressNode.class is unchanged and you do not need to update your nodes

Mar 27 update:
- Newer zip file attached, same version.
- I updated the 0readme.txt and included a screenprint in the zip.
- I added files, and instructions, for making a windows service so it starts automatically.

Whoops -- just found out I can't delete, then add attachment. No attachment options after delete & save! Check down this thread for subsequent post.

[FoBoT]

:drums:

i gotta get something on my work setup one of these days, right now i have to remote into all of them :bonk:

[rofn]

:|party|: :cheers:

good to hear that someone else can take use of such feature :)

perhaps we should let dy or someone else let the thread move into the 3rd party programs forum in the official df forum???
then sure more people will get rid of dfprogress and your good efforts ;)

rofn

[MerePeer]

This is not the oldest zip, keep lookin at posts below.

[MerePeer]

Here's pic

[MerePeer]

New version 1.03 fixes 2 bugs:

- corrected in {DFProgressNode.class} the "Set" column was not being updated for any node whose generation 250 was unable to post and was therefore buffered.

- corrected in {DFProgress.*} the real-time points calculation was not adding points from generation 250 when the generation reverted to 0 to begin a new set.

NOTE: I can't modify my earlier posts because of a Forum Edit restriction timer. I'll just post new versions as a new reply w/attachment for now.
You wouldn't get notified if I just edited a previous reply anyway, right?
I'd like to keep 3 things available each release: the latest zip, the prior zip (in case you dont trust the latest or discover a problem in the latest), and the screenprint (now out of date).

[rofn]

heya...

looks like dfprogress is worth its name ;)

ok i've tested it on my local machines...i'll have to get to my 2nd network to test out the forwarding but it looks like that this should be ok for working...

anyway...i found one issue so far...

i dont know why but the dfprogress client just greps my statistics from macnn not +2 above...
is there something else i have to put into the .ini to get this?

btw. +2 -2 would be great or even a configurable # how much he should display...afaik the grep procedure should be the same for that...

here is a screenshot of dfprogress only getting my own stats

http://members.nextra.at/rofo/dfprogress.png

perhaps you have an idea merepeer?

so far for this...

since those command prompt boxes quite annoyed me i wrote a short simple .bat script for windows just for the starting procedure of dfprogress & dfprogressnode...i'll attach them to this post...

you would also be able to add all your dfprogress & dfprogressnode commands into a single script...i splitted them as example into 2
so there will be no output for the client (except the window itself) & the node... ;)

rofn

[MerePeer]

Here's a quick fix until I have time to pkg everything. Problem was the header line above you after #30 Anteraan. So you can either pass him and it will work ;) , or you can grab this version which I only tested using your username but should work for everyone.

I can make #above configurable, #below a bit more work, but if I knew what the status of the stats page development was perhaps we could write our own stats "page" which is really just delimited text to provide all the information for DFProgress to then format -- user could choose which fields to show in the ui or something. I want to show the stats of the non-freedc users, so next version I'll be grabbing stuff from this url too ... that way you can watch your progress through the overall ranks! http://statsman.ww-testsites.co.uk/distfoldingstats/html/users.html

Attached 1.03a

[rofn]

thanks for the quick fix :) works now ;)

passing anteraan will take some time i think...

rofn

[Anteraan]

Originally posted by rofn
thanks for the quick fix :) works now ;)

passing anteraan will take some time i think...

rofn

It depends on ability to upload, which I can't do much of right now. That said, you still have more GHz on this thing now than I do, so something has to give. my long-stated goal is to be in the top 30, and if I can pull off a pass or two, I should be ok for a bit.

[rofn]

thats my goal, too

and since we have this fast protein a bit longer i should get through it...

anyway...

dunno but since the big troubles in the beginning i dont have huge upload problems...well ok sometimes it buffers 2 or 3 gens but thats it...and for the longer upload cycles i run a backup client on priority 20 for not loosing cpu time...the main clients run @ 0 on my workstation and -20 on my just crunching machines...
this solves the lost cpu time problem for me quite well..

rofn

[Welnic]

DFProgress is working fine for me at work. I am running it on OSX to monitor my linux farm.

The only problem that I have with it is the time that it reports as to when it updates. This is the time that the node updates. It is actually the time that the update happens, there is no arguing that. But I don't think that is the most useful time to report. As long as the node is updating this time will be updated. But one of the things that I need to monitor is if folding is still happening. Some of the nodes on my farm have a history of just shutting down, and the only real clue is that progress.txt hasn't updated for an hour. I would prefer if the update time was the time that progress.txt was last updated.

Whew! That seemed long winded. :D I really like the program and am quite impressed with how little processor time it sucks up.

[MerePeer]

Here's another version, 1.04.

DFProgressNode
- a minor change to address a startup condition I witnessed: progress.txt existed but fold_* files did not. The "set" was marked as 0 and didn't get updated when the fold_* file finally got created -- until the next set began. Now it will continue to look for the fold_* file until it first sees one. Would be nice if progress.txt had the # of sets in it -- I already asked on the official thread.

DFProgress
- the socket/URL timeouts are no longer reported as a popup 'ok' dialog, I added a status bar. Needs some tuneup but works for now.
- {per rofn request} I added 4 lines above and 3 lines below in the stats area. I didn't make these configurable yet.
I experimented with having the stats area display a URL that could be configured to your favorite -- and all I would do is scroll the window down to your username, if it was there. I was major disappointed with the way that the java JEditorPane renders HTML -- almost unreadable in some cases. It doesnt seem to deal with things like CSS, embedded tables, etc. too well. I've read that java 1.5 may address this. So then I experimented with pulling in the statsman "top 1000", but it was taking about 30 seconds to render it, and it looked awful, and I got concerned that some users might not have the internet bandwidth to pull this in every <configurable> minutes. So this "display of stats" is still under development and I've reverted to the macnn entries for just freedc for now.
- {per Welnic request} I know what you mean, I've seen this happen too: sometimes as simple as killing the foldtrajlite process, leaves the progress.txt out there, leaves the .lck file out there, and of course the node is still sending datagrams.
As you probably have seen, I already have two checks in place for lack of progress: (1) background turns orange on the update time if the time gets stale (for instance when DFProgressNode stopped, system shut off, lan down); (2) background turns red if the node is missing its progress.txt (for instance foldit.bat has been Quit).
I think a safer, and less expensive way to observe this new 'foldtrajlite bombed' condition (vs. I/O checking file dates on the node) is to not change the node, and instead we watch the structure # arriving at DFProgress and see if it has changed in some amount of <configurable> time. So that is what I have added: your structure # will now show on an orange (warning) background if it has not changed in the number of minutes. The default is 15 minutes but if you add a line to your DFProgress.ini you can change that (min. is 1):
structurechangewarning=15

Also included rofn's .bat files for windows.

[Welnic]

Thanks for the update. I have been tempted to turn one of my nodes off just so I can see the warning color show up, but I have been able to wait for the real thing so far.

[Scoofy12]

question: does DFprogress support running from a different working directory? ie, heres what I did and the crash I got on my linux box, running just 1 node locally:

xds@xdsdell:~$ java software/distribfold/dfprogress/DFProgress
Exception in thread "main" java.lang.NoClassDefFoundError: software/distribfold/dfprogress/DFProgress (wrong name: DFProgress)
at java.lang.ClassLoader.defineClass0(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$100(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)

then I cd into the proper dir, and it worked:

xds@xdsdell:~$ cd software/distribfold/dfprogress/
xds@xdsdell:~/software/distribfold/dfprogress$ java DFProgress
DFProgress v1.04

[Scoofy12]

Also I got an error message that Scoofy12 was not found on the stats page, although I am there, number 41.

One suggestion if possible is to make the status bar on the bottom resizable. if you increase the window size it doesnt change, and it also seems to be what limits making the window smaller.

[rofn]

Originally posted by Scoofy12
question: does DFprogress support running from a different working directory?


try:
java -cp /where/ever/the/directory/is/ DFProgress

[MerePeer]

As rofn correctly shows, you have the option of locating the DFProgressNode.class file itself somewhere else than where your default is. You use the classpath option, -cp {dirname} and then a space and then DFProgressNode and then a space and then whatever progress node options you are using.
This approach I have found useful if I want to run the node on box A but not copy over the .class file to box A and just reference it on a share mounted drive from box B. So I might do this: java -cp \\boxb\sharename\dirname DFProgressNode -sendto boxc -name boxa -dfdir c:\distribfold2


As you see in that example, and answering your original question, yes you can specify the location of the distributed folding dir on the command line using the -dfdir {dirname} argument.

So in total you can have a default directory which is different than the location of your DFProgressNode.class file which is different than the location of your df dir. Mucho flexible.

[MerePeer]

@Scoofy12
Just realized your question was about DFProgress, not DFProgressNode. Same answer really, -cp is what you needed. There is no -dfdir option for DFProgress, only DFProgressNode. (That's because DFProgress doesnt care about df directories.)

re: no Scoofy12 in stats
I just modified my DFProgress.ini file to have
username=Scoofy12
And when I started DFProgress it showed your record and the ones around you. Try this and let me know if problem. Maybe its not finding the .ini if you are running it in another dir?

re: sizable stats area
There is a draggable resizing line between the (top) list of nodes and the (bottom) stats area. There are also two tiny arrow icons on the left side of it. Try that and let me know if its what you wanted.

[Hagar]

I noticed that you need to specify -name for the node if you run 2 or more nodes on the same machine. Otherwise it can't distinct one node from another. Might be nice to put that in the readme.

After changing the teamID I even got the stats displayed ;)
Nice job :)

[jand]

suggestion:

1) could you include a feature to publish the results into html?
2) this one's a wee bit more difficult; upgrading the dfprogress so it will have 2 modes:
- server (viewers can connect to this port and update progress relayed to them through object serialization)
- viewer (connects to one or multiple servers and displays the progress onto the GUI)

the (2) makes it very scalable and if you were to include html outputs as well, it'll make dfProgress so far, the best df monitoring tool out there.

[MerePeer]

@Hagar
I just updated the readme for the 'next' version, whenever that happens (next protein probably).

I'll try to put the teamid into the .ini one of these days, make it easier for cows to graze I guess...:rolleyes:

[MerePeer]

@jand
I may not be understanding your situation, it might help to have an example. Did you notice that we have a DFProgressForwarder? I think your description "server" is really a collector of local node info. What you can do is have that node forward its results either to (a) another node or (b) the final monitoring node. So if you had NODEwork1 and NODEwork2 just regular nodes doing a -sendto NODEworkWithInternetAccess (which itself could be running DFProgressNode -sendto localhost), then you can also run DFProgressForwarder on NODEworkWithInternetAccess and have it -sendto NODEhome. The tree of nodes can be built up infinitely: forwarding to forwarders, end nodes to forwarders. In the end you direct them to your DFProgress machine which is just a receiver: never acknowledging packets arrival, never connecting out to get them.

Regarding the html.
The information changes every 10 seconds so I'm wondering how useful this would be. I've considered making DFProgress into an applet which could display within an html page, but that would mean the node packets still need to arrive on your desktop, not the html server, so I'm not sure it solves anything connection-wise.

[jand]

well, the situation here is that only 1 dfprogress can be monitoring.
i.e:

nodework1, nodework2 -> monitorwork1 -> monitorhome1

I have a few ppl folding under the same handle and was hoping we could have a shared view of our pooled machines

with the current method, i can only relay the results to a specific host. that'll mean i'll have to create many copies of the same node to forward results to other monitors

[Hagar]

:D

Originally posted by jand
suggestion:

1) could you include a feature to publish the results into html?
2) this one's a wee bit more difficult; upgrading the dfprogress so it will have 2 modes:
- server (viewers can connect to this port and update progress relayed to them through object serialization)
- viewer (connects to one or multiple servers and displays the progress onto the GUI)

the (2) makes it very scalable and if you were to include html outputs as well, it'll make dfProgress so far, the best df monitoring tool out there. I was thinking something similar. I have a small home network and some of the people running DF on their system would also like to see the stats from other clients in our network. A server client model for the GUI would allow this very easily and would save an incredible amount of time (and recources) setting up all the nodes.

@MerePeer: Instead of running a node for each GUI listener on every DF box you would only need one node per DF box.

current: (nodes represents 10 DF clients)
nodes1 --> GUI user 1
nodes2 --> GUI user 2

you need 20 nodes to monitor 10 DF clients for 2 GUI users

suggestion:
nodes --> server --> GUI user 1 ... x

1 node for 1 DF client and an "unlimited" amount of GUI users

[MerePeer]

It sounds like if DFProgressForwarder could "copy" the node info and send out multiple UDP packets for each incoming packet it might address your scenario:

1) nodework1, nodework2 -> port5030monitorwork1
2) forwarderOnMonitorWork1 -listenport 5030
-sendto monitorwork1 -sendport 5031
-sendto2 monitorhome1 -sendport2 5030
3) monitorWork1 -port 5031
4) monitorHome1 -port 5030

[jand]

Hagar: you took the words right out of my mouth. You're the man! :cheers:

MerePeer: basically, what we're looking for is a one-to-many relationship instead of the current one-to-one.

Polygamy works best on the Internet
;)

[jand]

Originally posted by MerePeer
It sounds like if DFProgressForwarder could "copy" the node info and send out multiple UDP packets for each incoming packet it might address your scenario:

1) nodework1, nodework2 -> port5030monitorwork1
2) forwarderOnMonitorWork1 -listenport 5030
-sendto monitorwork1 -sendport 5031
-sendto2 monitorhome1 -sendport2 5030
3) monitorWork1 -port 5031
4) monitorHome1 -port 5030

but therein lies the problem of dynamic ips. :(

[Hagar]

@Merepeer: If a new user wants to get the data you would have to reconfigure the forwarder. With the model described in my previous post the GUI would simply retrieve the data from the server instead of the forwarder sending it to the GUI. That way it doesn't matter how many GUI's are used (like a webserver)

[MerePeer]

I'll give this some thought. Maybe if the "forwarder" could be dynamically told "send some to me!" by a GUI who provided an IP, then the "forwarder" would add that IP:port to its list of whom to "cc" the node messages. Someday maybe the GUI could even specify which node(s) it cares about. Perhaps a timeout every hour for these dynamic dudettes -- i.e. no sense sending these packets out if no one listening; so GUI once an hour needs to 'remind' forwarder it is still listening. This would avoid the overhead of a server hosting all the node info and needing to respond to incoming requests with the latest refresh.

[Hagar]

That would provide the same functionality and is a lot easier to implement in your current code. The only reason I can think of to use a server instead of a forwarder would be that the "network" has less listeners and would be more secure. But I'm pretty sure nobody will be running this as root so that shouldn't be a problem either ;)

[MerePeer]

Here's version 1.05.

1) No change to DFProgressNode. {You dont have to change your crunchers.}
2) Since this protein is so pokey I added a "ETC" column to the DFProgress GUI. This (Estimated Time to Complete) value represents the number of hours:minutes:seconds to reach structure 100 (or 10000 for gen 0). It does not account for the time to finish the Gen, nor upload it. Each gen it starts over.
3) The DFProgress.ini file now supports "teamid=n" (teamid=5 for FreeDC) to override the default team when retrieving the MACNN stats.
4) I enhanced the DFProgressForwarder so that it can dynamically forward node information to multiple DFProgress GUIs. Dynamically meaning that it will listen for a request from the GUI (new Options/Request Forwarding) and then add the GUI to its "distribution" for forwarding. The forwarding expires after 60 mins. More detail + examples in the 0ReadMe.txt under forwarder note d. This new functionality has been tested on a LAN -- and is useful if you want to skip around to different boxes in the house and "check in" on the progress. I am not able to test this outside the LAN, specifically through NAT, so I expect jand and Hagar will give it a shot.

[MerePeer]

DFProgress version 1.06

* no change to progress node; dont need to chg crunching machines, just restart DFProgress monitor after unzipping.
* estimate (ETC) and progress bar for gen 0 now expects 30000.
* minor bug fix for background coloring after reordering columns.

[magnav0x]

Will DFProgress resolve hostnames?

[MerePeer]

Yes, the argument you give "-sendto" can be either an IP # or a name.