ARS horde are at the gate.

[PCZ]

Title says it all


Brian

[magnav0x]

booby traps anyone? I'll protect from a distance :sniper:

[Digital Parasite]

Did I forget to mention that dfGUI v3.x secretly changes your handle.txt file to an Ars account when it detects that FreeDC is almost going to be over taken?

Hehehe, I'm just kidding... :jester:

Jeff.

[rsbriggs]

Just kidding ????? Whew. You mean you didn't get that feature snuck into version 3.1 ??? :spank: Hey wait a minute - whose side are you on, anyway? It should change ARS handles to a FreeDC handle for 3 out of 5 generations when ARS is catching up....

( I figured that was the reason that FreeDC anonymous was starting to come up through the ranks.... )

Not much I can do to help hold off ARS - all my puny little systems are going full blast... They seem to be overtaking us by about 100K per hour.... :scared:

[Moogie]

Well, I've recently added another boxen (borg)...but have been having some problems with my home systems...did some fixing up...hopefully that fixes the problem.

Goooooooooooooooooo team!!!!!

[Paratima]

Wellllllll, I been foldin' as fast as I can (gonna pass thru 1,000,000 sometime in the morning), but I may actually have to take down a couple of my best systems at the office if they don't get that memory-leak problem fixed. :( Seein' as how I'm gonna be out for a week or more, I just can't afford the chance of those boxen freezing up. I'll leave 'em up till the last minute tho, just in case Howard and Elena find a solution.

I guess the other fix would be a rig up a timer for stopping & restarting every couple of days. Hmmm. I may give that a shot. :idea:

[Moogie]

Ugh Paratima....that's certainly a rough one. Hope you can find some kind of solution! Be safe on your trip!

[Paratima]

If my numbers Tuesday approximate my numbers Monday, you'll know I worked it out. :D

[Moogie]

Okis...I'll make sure I check it out. Right now, I'm having problems of my own and am not sure what is going on. For some reason...production is way down but there seems to be no reason for it.

....off to go take a closer look....

[IronBits]

I've got everything on DF I could find.
This client does not produce steady WU flow. Some days you get alot, some days you swear it's broken... just ride the wave, you're not alone ;)

Looks like Dyyryath and now maybe xj10bt have been bitten hard by the memory BUG :cry:

We are in this for the LONG HAUL, so keep up the good work everyone! :D
:cheers:

[pointwood]

Originally posted by IronBits
We are in this for the LONG HAUL, so keep up the good work everyone! :D
:cheers: That's certainly good to hear :thumbs:

I didn't expect anything else though ;)

Sorry to hear that some people have problems with the client - good thing we got Howard, he's usually able to fix bugs pretty quickly :cool:

We are only slightly outproducing you right now so that could easily change...let's hope for a great race! :bouncy:

Now, I'll return to enjoy my vacation - first week's been great :)

[Dyyryath]

Unfortunately, the problems with the client are affecting us quite a bit more than Ars. :(

We're a much smaller team so we rely more on individuals running lots of machines than they do and right now, the client's just too flaky for some of us to trust. However, as Pointwood pointed out, they're not outproducing us by too much, so we'll see how it goes. We may not have nearly as many 'small' crunchers as they do, but ours are definately dedicated, and they're keeping us in the race! :thumbs:

Once Howard & his crew get things straightened out, I expect our output will increase. Substantially. :cool:

[Moogie]

Originally posted by IronBits
I've got everything on DF I could find.
This client does not produce steady WU flow. Some days you get alot, some days you swear it's broken... just ride the wave, you're not alone ;)

We are in this for the LONG HAUL, so keep up the good work everyone! :D
:cheers:

That's the part that's killing me right now! Yesterday, my production went down by half! Argghhhh! Checked all boxen and it looks like they are all working fine.

You were right! I thought it was broken.

But, as you just pointed out, it's not steady.

Now, I need to call the doctor to increase my meds so I don't go insane trying to analyze the numbers.

(this is a curse we finance folks have to live with)

[IronBits]

Originally posted by Dyyryath
Unfortunately, the problems with the client are affecting us quite a bit more than Ars. :(
Once Howard & his crew get things straightened out, I expect our output will increase. Substantially. :cool: YA!! :D
:notworthy :notworthy :notworthy :thumbs:

[Paratima]

Originally posted by Dyyryath
Substantially. :cool: :thumbs: I love it when he talks dirty like that! :p

[magicfan241]

I was going to get scared. Then I realized that this isin't my project. :smoking:

But I'll be waiting if you guys can pass us in my home project (UD).

Just to warn you, a couple more ARS people might get it going as soon as the S||B gauntlet ends (july 9th). You guysbetterfind that production by then or you may be cut down:
The fight will look like this:


:machgun: :machgun: :machgun: :machgun: :machgun: :machgun: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun: :machgun::machgun: :machgun: :machgun: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun::machgun: :machgun: :machgun: :machgun: :sniper: :sniper: :sniper: :sniper:

(ARS is in yellow, FDC in black)

Waiting for the next shipment of snipers? (j/k)

[Dyyryath]

Actually, I think you have it reversed...it usually looks more like this:

:machgun: :machgun: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper:
:machgun: :machgun: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper: :sniper:

(Free-DC is yellow, Ars is black)

We're being overwhelmed by the sheer number of small guys you have. :D

[rsbriggs]

And that bguinto1 guy.....:shocked:

[magicfan241]

Small guys?? I have 4 GHZ! of processing power. That gives me a yellow guy. So nah!

[Dyyryath]

Yeah, they do have a couple of brutes over there. :D

However, if you take their output over the last 24 hours and divide it by the number of active users they have, then do the same with Free-DC, you begin to see what's happening here:

Free-DC Average WU per User: 140,850
Ars TSF Average WU per User: 28,696

If you take that big bruiser bigunto1 out of the equation, you get this:

ArsTSF Average WU per User (no bigunto1): 20,714

We're putting up a hell of a fight, but right now, they're crushing us with sheer numbers. ;)

[IronBits]

They got a bigger jump on us...
Arachnid Stats (http://stats.zerothelement.com/cgi-bin/distributed-folding/cd-graph-vs-team-custom.pl?teams=5&teams=78&data=LastUpdateStruct&time=1&points=14&title=Arachnid+Stats+-+Custom+Graph&xsize=600&ysize=400&scale=AUTO&maxScale=.1&maxAbs=&minScale=.1&minAbs=&type=LINE&transparency=0&dimension=0&lineWidth=2&submit=Create+My+Graph)

[magicfan241]

I think you Free-Dc guys are having TROUBLE WITH TRIBBLES!!

I am not running DF ATM, because of the often mentioned memory problem. So that's 4 les Ghz you have to deal with.
You know, you could get Howard to make a secret fix, and then only give it to you guys for a week or so, that would solve the problem...

[rsbriggs]

I still don't know what memory problem people are talking about. I've got at least one machine that has been running DF since 5 minutes after phase 2 was released, and I have never seen a single memory glitch. (Umm.... Actually I took that machine down once when dfGUI 3.1 was released, to upgrade....)

So, there is something very curious about the whole memory issue. Why are my machines so special that they haven't ever seen the problem? They range from an 866 Mhz P4, through a 2.66 Ghz P4, one AMD 1600+, one AMD 1800+, and a couple of other various boxes. Got one 2*Xeon HT box running 4 copies of the client, and they show over 72 hours of CPU time, and the highest peak memory utilization that task manager shows for any of them is 172K.

So, maybe Howard is in the same state - puzzling over what people are talking about, because he doesn't see the problem there.

Am I just missing the problem because I'm not looking at the right thing on these windows boxes ? I look at memory usage, peak memory usage, and VM size via task manager - nothing seems out of whack anywhere.....

[rshepard]

No, you're not missing anything-- my boxes are doing the same thing as yours-- they haven't had a memory error since the project rolled out. :confused:

[Digital Parasite]

Most of us are not talking about memory errors but more about Memory Usage.

rsbriggs, if you look at your DF clients that have been running, how much RAM are they taking up right now? The client when it first starts (if you are using the -rt switch) uses around 8900K. Are any of your clients using more than 89000K?

For every single machine that I have, after each generation is completed, it uses more and more RAM.

Jeff.

[Dyyryath]

I seem to be having a terrible time keeping the clients running. :(

I had to delete the filelist.txt file and restart all but 2 of them this morning. :bang:

I really hate this new client. :swear: :swear: :swear: :swear:

[Angus]

rsbriggs -

Forget the 'Peak memory usage' - look at (or add) the 'memory usage' column and see what your foldtrajlite.ex processes are using.

Are you allowing the client to upload automatically, or running nonet and using the dfGUI 'UPLOAD' button? The dfGUI method will stop and start the client, starting over at the 89MB memory level.

Your dual Xeon HT box is showing 172MB per process after three days... you don't think that is normal to almost double the memory footprint in 3 days? What if you left that machine alone for 2 weeks? What do you think the memory usage would be? As it fills your available RAM, and has a fit with page faults, the crunching will just sloooooowwww down until things stop or crash.

Personally, I think a lot of the crashes we are seeing are boxes running out of memory and corrupting things that get written back to disk. No scientific or programming analysis here, just a gut feeling.

[rsbriggs]

I'm not talking about memory errors, either. Since I use the -rt switch that allows them to use up to 150 megabytes of extra ram, I wouldn't find memory use in the general vicinity of 150 megs at all unusual. All of my boxes have minimum 1Gig ram, anyway.

Here are some task manager readings:

CPU / Memory usage / Peak Memory usage / VM size / CPU time (uptime)

2.66 Ghz P4 / 151,800 / 152,700 / 152,700 / 74:14:19
2.4 Ghz P4 / 164,800 / 165,548 / 168,944 / 93:44:56
AMD 1800+ / 102,240 / 102,892 / 110,394 / 65:30:34

The 2.4 Ghz box currently has the longest run time and highest memory usage of any of the 15 clients I'm running (it's on an out-of-the-way server that I don't pay much attention to - haven't even logged onto the box in two days).

8 of the clients on an 4*MP+HT box have uptimes of under 6 hours (uploaded them this morning), and current memory use of about 98K. 4 of the clients on a 2*MP+HT box have uptimes of about 36 hours and memory use in the 120K range.

[Dyyryath]

I'm noticing that I'm primarily having problems with dual CPU boxes. I'm going to look into it a little further this afternoon.

As for memory usage, I'd be perfectly happy with the numbers that rsbriggs is seeing. Unfortunately, mine tend to climb above 400MB per client over time. Even with 1GB of RAM (or more) in each box, two clients using 450MB per is unacceptable. :(

[Angus]

Originally posted by rsbriggs
8 of the clients on an 4*MP+HT box have uptimes of under 6 hours (uploaded them this morning), and current memory use of about 98K. 4 of the clients on a 2*MP+HT box have uptimes of about 36 hours and memory use in the 120K range.

It sounds like you are running with the nonet option ( -i f switch) and restarting the clients when you upload. Is that correct?

If so, you would not experience the drastic memory increase that occurs if you just start the client and let it crunch and upload automatically at the end of each generation, and never stop and start it.

[rsbriggs]

Your dual Xeon HT box is showing 172MB per process after three days... you don't think that is normal to almost double the memory footprint in 3 days? What if you left that machine alone for 2 weeks? What do you think the memory usage would be? As it fills your available RAM, and has a fit with page faults, the crunching will just sloooooowwww down until things stop or crash.

I'm not at all concerned about the memory footprint doubling in 3 days. I have the -rt flag on so I expect the processes to use "up to 150 megabytes of extra ram." I didn't say that the processes on the dual Xeon-HT box are using 170K - I said 120K - I don't have any clients using as much as 178K anywhere.

I'd have to check in another three days to see just how much memory usage has increased, if any. The fact that the 4 processes are using 480 megs of memory out of 2 gigs on the box isn't worrisome at this point. And, certainly if I checked back in another three days and found that they were using 1/2 the memory on the box, I'd restart the clients...

Now I have no doubts that people are seeing memory problems - I just don't see them here. And I stopped having corruption problems once I stopped using dfGUI on the multi-processor boxes. Jeff made it known that dfGUI has problems if you run more than a single copy of a client on a box, and use ANY action button in dfGUI. You can use them to monitor, but pressing start/stop/upload buttons on a copy of dfGUI running on a box which is running more than one client process is TROUBLE....

If you run dual CPU boxes, or an HT box, and run multiple copies of DF on the box, DO NOT USE dfGUI on the box to do anything but monitor. DO NOT start the clients with dfGUI, DO NOT stop the clients with dfGUI, and DO NOT use dfGUI to upload results, DO NOT even use dfGUI to hide client windows or make them visible - you will end up with corrupted files, phantom copies of the client process running, and/or a host of other problems.

===bob

EDIT: P.S. I use -if only on the 8x box. All my other machines auto-upload. Unfortunately, and due to circumstances beyond my control, the internet connection to the 8x box is only available during a couple of fairly short windows during the day. I have to run those clients -if and can only upload when a connection is available.

[rshepard]

Hmm- I must have missed that thread about dfGUI having issues on the duallies- I think I'll try running mine for a while w/o it and see if it makes any difference.

[Dyyryath]

I'd read that, but unfortunately, mine are all Linux boxes so I'm not using dfGUI on them anyway. I'm experimenting with different ways to 'monitor' the status (up/down and mem usage) of the client now so that I can have it restart automatically.

[Digital Parasite]

Originally posted by rsbriggs
I'm not at all concerned about the memory footprint doubling in 3 days. I have the -rt flag on so I expect the processes to use "up to 150 megabytes of extra ram." I didn't say that the processes on the dual Xeon-HT box are using 170K - I said 120K - I don't have any clients using as much as 178K anywhere.

I think you are missing something. When Howard said "Up To 150MB of RAM" that meant depending on the protein size. The current protein is not supposed to use any more than 89MB of RAM. Even if you turn off the -rt option, you will see it grows and uses more than the regular 20MB of RAM over time.

While it might not be a problem for you if you restart the client (by rebooting, starting/stopping or whatever) every so often so you never see it grow above "150MB" of RAM, many of our boxes don't get restarted so they start taking up 200+MB and 300+MB, etc... Since this is not supposed to happen, it would be nice if the problem was fixed...

Jeff.

[Digital Parasite]

Originally posted by rsbriggs
Now I have no doubts that people are seeing memory problems - I just don't see them here. And I stopped having corruption problems once I stopped using dfGUI on the multi-processor boxes. Jeff made it known that dfGUI has problems if you run more than a single copy of a client on a box, and use ANY action button in dfGUI. You can use them to monitor, but pressing start/stop/upload buttons on a copy of dfGUI running on a box which is running more than one client process is TROUBLE....

I guess I should be more specific here about running multiple copies of dfGUI. There may be a problem with the UPLOAD button on SMP systems, I haven't had time to check it out yet. That is the big one people might want to avoid for now since other people who use the UPLOAD button such as rsbriggs on SMP systems were seeing problems.

The STOP button in all cases should be safe since all it does is delete the foldtrajlite.lock file and lets the DF Client itself close gracefully.

The START button in Service mode should always be safe. The START button in non-service mode should also be safe but the Visible/Hide buttons may not work as expected if you have multiple copies of the DF client running since it won't know which window to hide/unhide.

The only thing in dfGUI that might cause corruption problems on systems running multiple DF clients is the UPLOAD button while the DF client is running. I'm not sure why yet but there may have been a change to how the DF client stops deletes its files so dfGUI might be automatically re-starting the DF client after upload a little too early so the new instance tries to write to the filelist.txt before/after the old copy has finished updating it, etc... That means the UPLOAD button might also be unsafe even on non-SMP systems. I'm looking into it. If the DF client is stopped and you click on UPLOAD, dfGUI won't try to restart the client so it seems to be safe all the times I have used it (every few days to dump my offline machines).

Jeff.

[the-mk]

Have you ever seen soooooo much boxen?

http://translate.google.com/translate?u=http%3A%2F%2Fwww.aei-potsdam.mpg.de%2Fabout%2Fnews%2Fmerlin01072003%2Fimages.html&langpair=de%7Cen&hl=de&ie=UTF-8&oe=UTF-8&prev=%2Flanguage_tools
original site (http://www.aei-potsdam.mpg.de/about/news/merlin01072003/images.html)

[rsbriggs]

You snuck into IronBits' garage, did you? :spank:

[magnav0x]

Ars has finaly overtaken us :bang: :trash:

[PCZ]

Oh no they haven't

Distributed Folding Project Statistics
as of: Wed Jul 2 18:02:01 2003 EST

FreeDC Team Page (Rank: 1)
(* = team leader)

Total team structures: 97,475,584

[Darkness Productions]

w00t w00t! But is it just a matter of time again?

[QIbHom]

Oh, man! This trading back and forth between FreeDC and Ars is so much fun, I want to be on both teams!

One of the reasons I love DF is that we have such friendly rivalry going on. There is nearly none of the nasty, spoiled brat name calling and dissing I saw wither other DC projects.

Despite the danger to my blood pressure, I hope this back and forth stuff goes on for a long, long time. It calls out the best in all of us.

Be well,
Lissa

one of the Ars little guys