Page 1 of 2 12 LastLast
Results 1 to 40 of 51

Thread: FADProgress

  1. #1
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771

    FADProgress

    I finished migrating D2OLProgress into FADProgress. Its a simple display of the progress of your FAD nodes. The GUI uses Java so it can be run on any O/S; each FAD node running the FAD "Server" runs a progress daemon which sends the progress information to the GUI.

    FADSpy is a much more thorough application which I recommend if you are using a Windows desktop. I personally use a Linux desktop and wanted a native GUI.

    This is an initial release and so there will likely be issues. I've tried it on Win2k, Winxp, and Debian.
    Attached Images Attached Images
    Last edited by MerePeer; 04-07-2005 at 08:46 PM.

  2. #2
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    Zip attachment has doc, binaries, source.
    Attached Files Attached Files

  3. #3
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    I should mention it currently makes a connection to http://stats.find-a-drug.com/livequery1.html to obtain the target-to-project mappings. It will do this each time it sees a new target has been reported by one of the nodes if it hasnt seen that target before. Since it is not currently obtaining the 'oldquery' list, it is possible that the Project column will have blanks if your nodes are working on old targets that have finished. I could add a startup option to shutoff that (http connection) feature if that's important to anyone, since I know some folks need proxies etc.

  4. #4
    Administrator Bok's Avatar
    Join Date
    Oct 2003
    Location
    Wake Forest, North Carolina, United States
    Posts
    22,165
    Blog Entries
    13
    Woohoo,

    thanks MerePeer. Been waiting for this!!

    I'll install it when I get back to town.

    Do you want it on its own webpage in the utilities section?

    Bok

  5. #5
    Not here rsbriggs's Avatar
    Join Date
    Dec 2002
    Location
    Utah
    Posts
    1,400
    There's a utilities section ????
    FreeDC Mercenary


  6. #6
    dismembered Scoofy12's Avatar
    Join Date
    Apr 2002
    Location
    Between keyboard and chair
    Posts
    608
    thats cool... does the GUI always have to be running to catch the packets sent by the nodes or what? how does that work?

  7. #7
    Senior Member
    Join Date
    Apr 2004
    Location
    So Cal
    Posts
    268
    Cool app. Thanks.

  8. #8
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    @Bok -- at your convenience we could setup a page like we did for http://www.free-dc.org/utilities/d2olprogress/ to make it easier to find the latest download.

    @Scoofy12 -- the answer is you do not have to run the GUI if you have other nodes running the daemon/service sending packets (obviously you won't see your information however). If the GUI is not running the packets are just 'lost in thin air' aka dropped after they arrive. UDP packets are not acknowledged, unlike TCP packets. Note: the default update rate (which you can change via a command line arg (or in the .INI file for win svc)) is 10 seconds for both the daemons sending the packets, and for the GUI to refresh itself.

    I should also mention that the GUI can produce an HTML file which mirrors the table of information. Currently you have to run the GUI to produce the HTML file, although you could minimize it. I will soon add a way to specify the directory+name of the HTML file, it currently defaults to FADProgress.html in the same directory. Also I don't think the special "stale datetime background coloring" in the GUI has been implemented in this HTML yet.

  9. #9
    Not here rsbriggs's Avatar
    Join Date
    Dec 2002
    Location
    Utah
    Posts
    1,400
    And putting up a nice page here so we could see what's "underneath" would be nice

    http://www.free-dc.org/utilities/
    FreeDC Mercenary


  10. #10
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    Great work as always

  11. #11
    =>Team Joker<= LAURENU2's Avatar
    Join Date
    Dec 2004
    Location
    Chicago IL USA
    Posts
    5,477
    Blog Entries
    1
    Originally posted by rsbriggs
    And putting up a nice page here so we could see what's "underneath" would be nice

    http://www.free-dc.org/utilities/
    I get
    Forbidden
    You don't have permission to access /utilities/ on this server.

    on that link

  12. #12
    Not here rsbriggs's Avatar
    Join Date
    Dec 2002
    Location
    Utah
    Posts
    1,400
    Exactly my point
    FreeDC Mercenary


  13. #13
    Administrator Bok's Avatar
    Join Date
    Oct 2003
    Location
    Wake Forest, North Carolina, United States
    Posts
    22,165
    Blog Entries
    13
    yes, we haven't made a page there yet. Not my forte......

    And I have apache setup to always deny directory listings.

    Bok

  14. #14
    Target Butt IronBits's Avatar
    Join Date
    Dec 2001
    Location
    Morrisville, NC
    Posts
    8,619
    Fixed!

    Not pretty, but fixed for now


  15. #15
    Not here rsbriggs's Avatar
    Join Date
    Dec 2002
    Location
    Utah
    Posts
    1,400
    Cool. Thanks. Now we have a page for utilities to show up on....

    Umm.... Will there be any?
    FreeDC Mercenary


  16. #16
    =>Team Joker<= LAURENU2's Avatar
    Join Date
    Dec 2004
    Location
    Chicago IL USA
    Posts
    5,477
    Blog Entries
    1
    Thats what I like about this place Fast Friendly Service And it even come with a

  17. #17
    dismembered Scoofy12's Avatar
    Join Date
    Apr 2002
    Location
    Between keyboard and chair
    Posts
    608
    one other question.... what files does it read to get its information? I have a directory that used to be shared among several nodes, but now only services one (i found that i got better speeds if each node used a local directory). so FADProgress reports all the leftover jobs that were unfinished when I killed the other nodes and moved them. I deleted the .log files in the "job" field, and some other similarly-named files, but the dead jobs still show up in FADProgress. What else can I do so they don't show up?

    (or, since these are unfinished jobs, can I force the currently-running process on the local node to take them over and finish them?)

  18. #18
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    Scoofy12

    You can put the old jobs on hold and reassign them to the node that is running.

  19. #19
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    MerePeer

    I have fadprogress monitoring all my nodes, a mix of windows and linux.
    The deamons run fine and the program does what is says on the tin !

    I have a request, can we have some colour in the html output.
    It would be good to have at least RED on stalled nodes.

    I have a web server running using the fadprogress html output and can monitor my nodes anywhere there is web access.

  20. #20
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    @Scoofy12 -- PCZ has the right answer. Another alternative is to just remove them.
    ./server -remove j1234
    The main file it reads is server.dat. You can read server.dat with a command like
    ./server -list


    @PCZ -- html coloring coming up. I'm thinking we keep the yellow background currently showing when the last activity timestamp or the progress updated timestamp is 3 minutes stale. Then we replace the yellow w/a red background when, say, 15 minutes stale. I'm wondering if the node name should receive the same background color too.

    I may as well add the html output dir/filename cmd arg too.

    And I've moved the Rating column to be #2 next to the node name -- since it is a (fad) rating of the node, not the server. Of course you can always drag it to a diff. position.

  21. #21
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    Thanks
    It is a lot easier to monitor with a bit of colour.

    How about a green background for a running node.
    Red for stalled and yellow for stale.

    3 mins is a little bit on the low side to turn yellow.
    From experience the clocks on the nodes can easily slip 3 mins, I regularly have to use the date command on the PXE nodes.

    Could you add variables for the traffic light system in the ini ?

  22. #22
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    Ok: configurable yellow_delay and red_delay.

    I think I'll play with CSS stuff after getting this version out -- html output too bland.

    I know you use LTSP nodes whereas my nodes boot a custom Linux O/S from the NFS share. So I set mine up to run "ntp-simple" (network time daemon) and that keeps them in synch. Is that possible with your LTSP setup? Might save you those date command issues.

  23. #23
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    Yes NTP should work.

    I have just synced my router up with time.nist.gov and it should work as an NTP time source.
    Just need to load up an NTP daemon on the ltsp nodes.

    I will probably put it in rc.local.
    Whats the executable called ?

    PS
    My nodes boot the ltsp kernel but that is about all thats left from LTSP what they mount is a mix of RH 9 and some newer libs from fedora core 2.

    The boot server is a Sparc running Aurora

  24. #24
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    Executable is ntpd. Startup file is /etc/ntpd on redhat (ntp-server on debian). Config file is /etc/ntp.conf. Yum/RH package appears to be called ntp (debian needs both ntp-simple and ntp, and server only needs ntp-server).

    I have my Server setup to sync with the internet
    server pool.ntp.org

    Then I setup my server to broadcast its time to the local subnet
    broadcast 192.168.1.255

    Then each node also runs ntpd but each node's ntp.conf file has
    server {name or IP of your server}
    Last edited by MerePeer; 04-08-2005 at 07:16 PM.

  25. #25
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    And you can make Windows nodes use your own NTP server,

    WinXP Control Panel/Date and Time/Internet Tab (enable checkbox, chg server name)

    Win2K a bit more difficult. Might be best to doublecheck me via google for "net time ntpserver 2000". Basic steps including starting the service called "Windows Time", then adding registry entries under HLKM\System\ControlSEt001\Services\W32Time\Parameters for key "type" a REG_SZ with value of NTP and key "ntpserver" with REG_SZ value of {your server IP or name}. The DOS command "NET TIME" also relevant, but always stubborn for me.

  26. #26
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    No problem with the windows servers.

    I sync all the servers at work to an NTP time source.
    The domain members get time from the DC's so no need to touch them but the workgroup servers are set to go to an NTP server.

    The internet facing routers sync up with a stratum 1 clock and as the time filters down to the access layer routers it is Stratum 3.

    net time /setsntp:ipaddress
    no need to edit the registry

    Set the time service to auto, it is set to manual on non domain machines.

    I have never used NTP on nix boxes before, I don't admin those at work.

    I have set the PXE nodes to run
    ntpdate -u serverip
    They should sync up whilst booting should be good enough.


    Anyway this is getting a bit of topic back to colours !!

  27. #27
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    v1.01
    You dont need to change the daemons, only the gui has changed.

    * I moved the rating column next to the node name, since the node is what is rated
    * If either the updated or activity dates get 'stale' then the node name now also chgs background color
    * Added general html coloring
    * Added 'stale' html coloring
    * Added these new FADProgress.ini optional values you can include:
    html_file=/home/user/myfile.html - specify directory and filename of html output file
    projectlist_enabled=false - do not perform http query to FAD site to obtain project names
    stale_delay=3 - wait this many minutes before considering a date stale and showing it in orange
    ancient_delay=20 - wait this many minutes ... ancient ... in red
    * Html output column order now matches gui, i.e. if you drag/change column order it will now be reflected in html
    Attached Files Attached Files

  28. #28
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    MerePeer

    I have a bit of a problem with the html output.
    Basically it only runs for about 3 hrs then stops updating.

    If i restart the Fadmonitor then the html output works again for a few hours.

    Edit
    Seems to be behaving now

  29. #29
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    Hmm. I haven't seen mine stop but that doesnt mean yours didnt. It could involve a date, or time change, for instance as it rolls past midnight. Might even be different types of jobs or queue status codes.

    The good news is you'll know it stops when the "Generated:" date at the bottom is stale.

    I'm wondering if you also see the FADProgress GUI stop updating itself i.e. can you confirm html is stopped but GUI is still updating properly next time?

    Any chance that external activity might have the file locked up -- for instance if you are http-serving it via a web server? Maybe FADProgress isnt handling that interaction properly -- you could try writing a looping script that accesses it (wget perhaps) so that a better chance of regenerating the prob occurs.

    Also, if you could start up the FADProgress in a shell (either DOS command window or Linux xterm) then after the html stops, look in the shell and see if any stack dump exists -- the shell is where stdout/stderr msgs will go for that gui. Post the trace if you see one so I can try to isolate it.

  30. #30
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    I've duplicated the problem by letting mine run for a few days. I'll check into it.

  31. #31

    temp.jgo project groups

    I notice you use the livequery to find the project name.

    First your using the .com which is the backup server you should use http://www.find-a-drug.org.uk/livequery1.html which is the current primary stats server.

    But there is a much quicker way (and probably a better way)

    use the temp.jgo file in the FaD folder

    Code:
    1  10 1rzj-q1  0      1        1    51352        1
    3   5 1ptz-q1  0      1        1     4097        1
    3  10 1hzj-q1  0      1        1     4097        1
    0  10 1uu3-q1  0      1        1    51352        1
    0   5 1nkp-q3  0      1        1     4097        1
    0  10 1c4z-q1  0      1        1     4097        1
    12 25 1qm3-q3  0      1        1    51352        1
    8   1 1jqe-q1  0      1        1     4097        1
    8  20 1irj-q2  0      1        1     4097        1
    You'll see something like the above with other bits after it.

    This shows you
    Project - Priority - Query - ? - ? - Start of mollie range - End of mollie range - ?

    Project numbers are
    0 Cancer
    1 HIV
    2 Agrochemicals
    3 Proteome
    4 Anti-bioterrorism
    5 Methodology
    6 Reserved
    7 Reserved
    8 Respiratory disease
    9 Multiple sclerosis
    10 Malaria
    11 Commercial
    12 CJD

    Hope that makes it faster and useful for people with no/restricted connenctions.

    (not having not run your program as I have no Java RT installed i'll waffle on)
    One thing to note is if a beta client is in progress then they may have jobs for beta only, these come from a different file (but are renamed our end) so if you run a mixed beta / gold(release) setup then you may run into problems
    You may want to get the info from the local temp.jgo to that clients server.exe location rather than rely on just the one on the computer your using.

    Adds more complication but I see you also have a service to send udp signals, so it could be easier than i'm thinking :-D


    EDIT: you should post your program on the FaD board
    Last edited by FluffyChicken; 05-05-2005 at 10:07 AM.

  32. #32
    Senior Member
    Join Date
    Feb 2004
    Location
    Maryland, USA
    Posts
    771
    I've posted a fix for the 'html output stops sometimes', and I changed the livequery url per above. You can grab v1.02 (only the GUI has chgd) from the new page that Bok installed for me here: http://www.free-dc.org/utilities/fadprogress/ (thx Bok) .

    ===========================

    @PCZ - let me know if that html issue happens again. I had the fix running for 2 days without probs.

    @FluffyChicken - thanks for the details on the temp.jgo file however I want to stay away from hardcoding lists of FAD projects/numbers into this app because then they'll need to be kept up to date. It would be nice if the FAD folks/software included the project name in the temp.jgo or better yet, in server.dat! In fact, doing so would also solve the issue that some projects are actually in the "old targets" url, making it difficult to obtain a complete list whcih is necessary because some PCs are still crunching projects which were live, yet have since been made old. Maybe they will do this someday.
    (Note: the fadprogress livequery can be shut off using an ini setting.)
    Regarding your idea about posting outside of Free-DC: a bigger audience is more work than I care for. As you can see we've got enough requests with the handful of local Free-DCers using this to keep me quite busy, and then I owe a D2OLProgress update after that!
    Thanks for your input!
    .

  33. #33
    You wouldn't need to hardcode that much, as you could leave create a seperate file for people to edit themsleves in the future ;-)
    The Project codes will not alter (they will only be added to like CJD was added)

    One problem with your way is that it takes a stats update for the newer projects to get added to the live query page, so you have the potential of ~6 hours without a project type known when you could have got it from the temp.jgo


    As for older job names, temp.jgo only gets updated when the client recieves so if a query type dissapears then it'll only be any running jobs that may loose the name (as any in the que will should be purged fromthe list), but if you keep a temporary file with the data in until it's not needed anymore that would get around that.


    I believe the queries files do contain the project name, but you'll need to crack the encryption


    I do see the benefit of your way, but it does add extra bandwidth usage to FaD's servers (they can take it but as usual it cost money)



    Anyways,

  34. #34
    dismembered Scoofy12's Avatar
    Join Date
    Apr 2002
    Location
    Between keyboard and chair
    Posts
    608
    I think i great addition to FADProgress would be some sort of summary view, estimation of points/MFLOPS, total and per unit time or something... (i'm still not quite clear on which is the "real" metric and what their relationship is with the CPU rating), maybe something along the lines of what DFGui had, only aggregated. maybe even integration with FAD stats would be cool (or failing that, you could just enter your point total and it could calculate based on that).

    i could even offer programming assistance except that i've never done any java. (if its easy enough to figure out based on C++ experience i could handle it, but i expect there are a few class libraries i would have to learn) Anything on the C side i could probably help with though.

  35. #35
    SIDE:
    Originally posted by Scoofy12
    ... (i'm still not quite clear on which is the "real" metric and what their relationship is with the CPU rating)
    CPU Rating is the 'Work Done' in One Hour of CPU Processing Time
    Note : Points = GFLOPS / 8

    So a rating of CPU Rating of 200 means you get 200 Points every 1 hour of CPU crunching Time (not real time)

  36. #36
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428

    Thumbs up

    Thanks MerePeer

    The html fix is working for me.
    Been up a couple of days now without stopping.

  37. #37
    Not here rsbriggs's Avatar
    Join Date
    Dec 2002
    Location
    Utah
    Posts
    1,400

    FADProgress

    Just wanted to say "thanks" - it's turned into a great program, and the ability to monitor both Windows and Linux nodes makes it more useful to me than FADSpy....
    FreeDC Mercenary


  38. #38
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    What I find most useful is the html output.
    I can keep tabs on my farm from anywhere there is an internet connection.

    rsbriggs
    FadSpy gives more info and I still use it at home but it is a bit buggy and you have to run version 0.45 with use pcname files deselected if you run linux clients.

  39. #39
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    MerePeer

    Is it possible to have a setting in the ini file to change the sort order ?

    The reason for asking is that I have to run the gui and toggle the activity column.
    This means logging in to the server and leaving the session running with the gui.

    I can run FadProgress as a service and this is the way I would prefer to run it, but it means using the default column sort.

    Changing the sort order by activity means any stalled/stopped nodes go to the top of the list and are easily spotted.
    I can view the html with a reduced window size and still see if any nodes have stalled.

  40. #40
    dismembered Scoofy12's Avatar
    Join Date
    Apr 2002
    Location
    Between keyboard and chair
    Posts
    608
    Bump!
    i just had to dig this out, so figured i would make it easier to find for the rest of us.

    maybe we should make this a sticky, its a nice program. maybe one of these days i'll learn java and add the ideas i had earlier
    C-x C-c

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •