As the official team leader of Team Stir Fry, I'm submitting this message partially on behalf of the requests of my teammates. The beta testing procedures for software client updates need to be reviewed very carefully to avoid fiascos such as the current one involving the Windows client. The situation that needs particular scruntiny is that involving automatic software updates of the client. A buggy software client can esentially shut all the distributed computing testing down and drive away users.
I would suggest that in the future, that before any software client updates are switched to in an automatic changeover, the software client update be available somewhere on the official site for download at least 2 days in advance. This would allow a smaller number of users to test and verify that there are no bugs with the client before it is unleashed on the general public. (Some warning about the client not being completely tested for bugs yet could be placed somewhere on the download site.) It also might be worthwhile to do the same with protein updates by allowing users to switchover in advance of the main protein switchover by at least a few hours. These steps should prevent the recent troubles surrounding software updates to the client software.
A member of TSF http://teamstirfry.net/
I'm not a team leader but I second the motion anyway and also volunteer to help with the testing.
I'll go even further and say that the results of the testing wouldn't have to count, as attempting to do scoring would make the whole process unwieldy. I'll dedicate one box to this, a couple of days prior to a changeover, and I'll bet that several others would, also. It could be set up exactly like we did the beta testing for Phase II.
Actually for software testing, I don't see any reason why the results couldn't count, although it Howard doesn't want to do so that's fine. Protein beta testing may be a different matter though.Originally posted by Paratima
I'm not a team leader but I second the motion anyway and also volunteer to help with the testing.
I'll go even further and say that the results of the testing wouldn't have to count, as attempting to do scoring would make the whole process unwieldy. I'll dedicate one box to this, a couple of days prior to a changeover, and I'll bet that several others would, also. It could be set up exactly like we did the beta testing for Phase II.
A member of TSF http://teamstirfry.net/
I'll make that a 3rd!
I have no problem setting up a w2k or XP or both for testing releases, before it is released.
I like the -beta switch idea, which would make it look to another server for updates, and if there is one, snag it.
STATS? We don't need no stinking STATS for testing pre-release candidates
After all, we are testing the 'release' mechanism and the clients proper operation after a succesful update right?
With the -beta switch, we could try it, check it out, delete it, put the old client back, add the -beta switch and try it again and again and again and again
I was just thinking of the problems of maintaining multiple result databases and the fact that changeovers don't usually occur without protein changes. (I believe this was a first.) Don't know if their setup will accomodate such.
Whatever, as long as we prevent catastrophes, that's the main goal. And I'll see IB's W2K and WXP and raise him a RHL!
I'd be glad to help out with testing as well-
I can throw in Win2k servers on duallies, a Win2K pro box, and a couple of Mandrake Linux boxes--If it can be broken, I'll find a way
( just ask my sysadmin)
Another vote for that - actually I already requested it yesterday in another thread:
http://free-dc.org/forum/showthread....highlight=beta
LOLOriginally posted by IronBits
And I'll volunteer a couple of Dyyryath's *nix boxen
Thanks, IB...
"So utterly at variance is destiny with all the little plans of men." - H.G. Wells
Such actions should be a part of any major software update. I have a bunch of unhappy teammates although since TGC was/is heavily involved with Stanford, we've come to expect such things
Whatever, better testing is required and a willingness to defer the changeover if there are any problems with it. Releasing unoptimized code when it could have just been postponed instead was a major FU.
I've got a dual XP1800 box for testing Linux and BeOS candidates and an OSX box for Macintosh.
Testing these beasties prior to unleashing them is essential. As is releasing them a the start of a day rather than the end of a day. Come in at "oh-dark-thirty" and do the deed rather than 1600 PST.
Agent Smith was right!: "I hate this place. This zoo. This prison. This reality, whatever you want to call it, I can't stand it any longer. It's the smell! If there is such a thing. I feel saturated by it. I can taste your stink and every time I do, I fear that I've somehow been infected by it."
We agree it would be great to have such a facility. Since the beta testing of the phase 2 client, I have thought about what may be a way to accomplish this.
The easiest way to implement something like this would be to run a parallel beta project at beta.distributedfolding.org, complete with its own auto-update and versioning, and completely independent of the main project (but sharing the same user database initially of course). Then a beta could be released as an autoupdate say 2 days before the 'real' update to allow some testing, since the code itself would be identical in both clients.
This would require us to set up another database to store the beta data, of course, but we could probably scrounge up the extra disk space on the server for this since presumably it would be minimal with only a dozen or two folders going.
It would also require us to do twice as much work every time we made a change to the algorithm/program. This is where I hesitate. It already takes us two full days to prepare and deploy a protein update. With only one person on the project (now that Elena is here, I am trying to start new work...) we cannot really afford that right now.
Also, the software is constantly changing and (hopefully) improving - every time we make a change we'd have to do another beta test or it would be pointless.
We will endeavour to test the Windows client more thoroughly immediately before release, ensuring it can at least make it through to gen. 1 with no problems.
Like we said this was a highly unanticipated problem and the code WAS all tested and working fine (albeit with a few still unresolved bugs from the previous version) before we switched compilers (the only difference between what you are running now and what we released is we are back to MSVC6). We will be more cautious about switching back to .NET and make sure everything is working properly before doing so.
In the future when and if we have a few more staff working on the project we can hopefully set up a beta testing facility somewhat as described above.
Howard Feldman
Just to add to what Howard was saying, unfortunately there is only one of me for now. I am still learning some of the ropes around the mighty DF, so a full-fledged beta site might hinder us from serious progress on updating and upgrading the client itself.
I promise to test and double test and triple test the windows updates/releases before unleashing them in the future. If only the world consisted of Unix users - the code would be so much smaller and my life would be so much easier
I will try and think of some smaller-scale ways that the updates may be tested ahead of time, but even I need to sleep every now and then under my computer desk
Elena Garderman
There is an old Project Controls shibboleth that goes -
"We don't have enought time/resources to do it right, but we do have enough time/resources to do it over"
Agent Smith was right!: "I hate this place. This zoo. This prison. This reality, whatever you want to call it, I can't stand it any longer. It's the smell! If there is such a thing. I feel saturated by it. I can taste your stink and every time I do, I fear that I've somehow been infected by it."
Why would it take any more days than normal, other than the upfront costs of setting up the beta server and allowing for a -beta switch (or something of that nature).
If the -beta switch is in place, it checks to see if there is a new client on beta.distributedfolding.org. IF there is, it fetches that and starts working, storing data anywhere you want it (beta database, regular database, etc). All you would need is to copy the "beta" client to your beta site, and away you go.
You have no idea what we have to go through to prepare an update, so just take my word on it.
Howard Feldman
You're correct, Darkness Productions don't know - he still have a point though and it's a very important point.
If something like happens again, you'll most likely loose *a lot* of crunchers. You've probably already lost a lot. I don't want this to sound like a threat but it's the simple truth as I see it. There are lots of other projects around and people will either stop doing DC at all or start supporting another project if the client doesn't *just work* from now on.
You might end up dropping Phase 2 on the floor because of it and no one wants that to happen
That's why Darkness Productions and the rest of us, stresses the importance of proper testing before anything is released.
Last, a friendly (and I mean it friendly!) note: Be careful when replying - your reply to Darkness Productions could easily offend him and others which I don't really think was your intention.
![Send a message via Yahoo to [DPC]Mobster](images/misc/im_yahoo.gif)
Without judging anybody, I think that Howard already has taken a lot of crap from all of us. I (used to) participate in over 10 different projects and never did I see a project leader get this much crap over him. I think he has been very kind to some of us who are really taking a piss at him.
I think his reaction was quite normal and that other project leaders wouldn't behave (in a good way) as well as he did!
Howard (and the rest), you're doing great work and don't let the tight asses among us ruin it for you!
Back to folding now
Proud member of the Dutch Power Cows
The vast majority of posts over the last week have been reasonable, I believe. I haven't seen too many posts personalising their displeasure. What I have seen is posts offering valid advice and offers of help to ensure this does not happen again.
Look at the top 100. 11 crunchers have not returned a single unit for this new protein. 11% of the most dedicated have apparently quit! 388 out of the top 1000!
This project simply cannot afford to have another changeover like this! I, along with others, have offered to pre-test any new client. All I am asking is that before the new client is put on the website for download, it gets e-mailed to a few volunteers who check it runs without errors. Some of us have very fast machines which will produce errors at least as quickly as Howard can and the more of us who test, the faster Howard can be sure he has a working client.
I don't want or need stats for such a beta. I would have thought that running the beta client through an install and an off-line first 50 generations or so would catch such as this week's problems.
If the new client is known to have a problem, I do not understand why it was rolled out, instead of being left for another day.
Finally, the point about a replacement new client auto-updating is not a lot to ask.
I'm not having a go at anyone. Historically, Howard has been enormously responsive to the community and in that spirit, I am offering advice as one software professional to another.
Just wanted to say that I absolutely have no intentions of flaming Howard in any way. I think Howard and the rest of the DF "gang" is doing a great job and been very responsive and I have a lot of respect for the efforts that have been put into making this project work. My previous post was meant to be constructive critique. Maybe my english isn't good enough (not my primary language) or maybe I was just too tired when I wrote it (it was pretty late), I don't know.
I don't want to see this project fall apart and I believe it will if things like this happens again. I have no doubt lots of top producers will then say enough is enough and leave and everyone loses
It's not possible to completely avoid issues/bugs, but you can minimize the risk a lot.
I just wanted to mention that it is official. Our friends at NCBI have confirmed that there is a serious bug in the Microsoft .NET and .NET 2003 (both) compiler optimizers. They were able to construct a simple piece of code which, when compiled with optimization, produces incorrect machine code output. This was indeed the cause of the problems during the most recent changeover. Of course it is still our fault for not testing the EXE after the compiler switch so we can't shift all the blame to Micro$oft (but we will still shift as much as morally possible) Linux: 1 Windows: 0
Howard Feldman
Howard, would it be possible to SEE that simple piece of code? I don't ask this to question you in any way, rather, something like this would be a matter of utmost interest to the company I work for. We are in the process of conventing millions of lines of straight 'C' and C++ code over to C# and the .Net framework....Originally posted by Brian the Fist
I just wanted to mention that it is official. Our friends at NCBI have confirmed that there is a serious bug in the Microsoft .NET and .NET 2003 (both) compiler optimizers. They were able to construct a simple piece of code which, when compiled with optimization, produces incorrect machine code output. This was indeed the cause of the problems during the most recent changeover. Of course it is still our fault for not testing the EXE after the compiler switch so we can't shift all the blame to Micro$oft (but we will still shift as much as morally possible
FreeDC Mercenary
Actually yeah, if you can get a hold of this simple piece of code that compiles badly, it would also be of use to me since I have been using the .NET compiler for stuff (so far without problems) and might have to switch to 2003 soon.
Now the question is, does anyone here have good enough contacts at MS to bring this bug to their attention and hopefully get it fixed?
Jeff.
Well, the C#(.Net) compiler is completely different from the C++.Net compiler. If there's a bug in the optimizer for C++.Net, then it wouldn't necessarily affect the C# compiler (though of course, none of us can see the .Net compiler code, so there's no way to know for sure other than by trying to compile the same code using the C# compiler).
At work, we are also considering switching from VB6 (well, mostly -- some old programs are still VB4) to either VB.Net or C#.Net (I'd prefer the latter, but we'll see...), so yeah, if we could get this code for testing, it'd be helpful.
Jeff -- we could all use our (employer's) credit cards (or MSDN subscriptions, if you happen to have one of those) for the cost of a support call, and nag their tech support people about it. Endlessly.
Jeff -- we could all use our (employer's) credit cards (or MSDN subscriptions, if you happen to have one of those) for the cost of a support call, and nag their tech support people about it. Endlessly.
FreeDC Mercenary
Since I didn't find the bug I didn't want to go around publicizing it and taking credit for it but let me just ask the guy who did find it if he minds first. If he's OK with it I'll be happy to post the example.
Howard Feldman
Here is the example, which reproduces the error in MSVC. It is supposedly known by Microsoft and will be fixed in the 'next' version:
#include <iostream>
#include <tchar.h>
int _tmain(int argc, _TCHAR* argv[])
{
char c[4];
c[0]='a';
c[1]='b';
c[2]='c';
c[3]='d';
std::cout << "before: " << c[0] << c[1] << c[2] << c[3] << std::endl;
for (int n=2 ; n>0; n--)
{
for (int i=0; i<3; i++)
{
c[i] = c[i+1];
}
c[3] = 'x';
}
std::cout << "after: " << c[0] << c[1] << c[2] << c[3] << std::endl;
return 0;
}
Correct output is
before: abcd
after: cdxx
Compiled with "max.speed", the output is
before: abcd
after: cxxx
In fact, this cycle gets compiled as
for (int n=2 ; n>0; n--)
{
c[3] = 'x';
for (int i=0; i<3; i++)
{
c[i] = c[i+1];
}
}
Howard Feldman
Yeah, I'd definitely say that was a problem...
Quite curious. Cute little problem with the optimizer, though...
FreeDC Mercenary
Posting Permissions
Reply With Quote
