Query over new 'scoring' method

[pfb]

Looking at the info in the progress.txt files for the new algorithm I am getting, for example, the following:

Building structure 44 generation 3
6 until next generation
1 generations buffered
Best Energy so far: 45.599

I assume that we should be getting higher energy values for gen 1+ then we were with the 'old' RMS scoring...? Also, the High-Flyers page shows the best RMS still (for example, mine says 14.63 - the lowest energy recorded locally is 45.599) - will this change to include / reflect the scores we are seeing locally?

[shortfinal]

They hadn't finished updating the web-site to the new protein. The 'High Flyers' page was still showing scores for the old protein.

Looks like they're finished now updating the site to the new protein.

Your 'local' score will be uploaded when you upload your results. Right now it should show a score of 100 on the 'High Flyers' page indicating you haven't uploaded any results yet.

Finally, unless I'm wrong, they want lower best energy just like they did for RMSD.


Shortfinal

[pfb]

I understand that - my query is regarding the correlation between the energy score being shown by progress.txt (and whichever 3rd party util used to monitor DF - dfGUI/dfMon/etc.) and the score on the site (which is RMS)...

I have 5 clients who have the following best energy scores:



yet my High Flyer page shows my RMS to be 13.45 - which of those 5 energy scores is that?

Not really concerned but would be interesting to know how things are now begin calculated as we appear to have 2 scores - the RMS on the site and the energy score locally...

Also, the energy score is now up and down rather than the smooth downward trend that RMS score locally gave us - example below from dfGUI:

[Brian the Fist]

Firstly, RMSD is not Energy - two different things - ideally they are correlated and we want to minimize both.

Secondly, remember this is a new experiment, testing a slightly new algorithm (we have tried it before only during phase II beta testing). Thus we don't know exactly what will happen. That is what is so wonderful about science - discovery!

It may do great, it may not, but the important difference now is we are not using ANY information about the true structure now, it is true 'ab initio' prediction. We are treading on unexplored territory now. Thus even if the energy goes down, the RMSD may not, and vice-versa. We will watch how this run goes carefully, and 'tweak' parameters as we feel necessary for the next protein to try to improve things even more.

[pfb]

Thanks for the feedback

Will there be any plans to show the Energy score (alongside or instead of RMS) on the site if it proves to be the way forward?

[Blackeagle]

From what I can see this new scoring system has slashed scoring per day by 2/3's.

That will in effect enshrine those on the top half of the front page for good. A new team try'n out DF will be quick to detect this and move on to a differant project that does not treat them this way.

All scores should be wiped clean. And it's to bad this wasn't run parralel to the old scoreing system first until it could be tweaked, not waiting to tweak it until after the fact.

This new scoreing is going to turn away new members and teams as well as possibly causing some currant users to drop out.

This is bad for the teams who will now find it MUCh harder to recurit new membes vs other project teams. And loss of currant members AND potential new members is very bad for the advancement of the medical research involved.

Not acceptable.

[Paratima]

Originally posted by Blackeagle
<snip> That will in effect enshrine those on the top half of the front page for good. A new team try'n out DF will be quick to detect this and move on to a differant project that does not treat them this way.
This is bad for the teams who will now find it MUCh harder to recurit new membes vs other project teams. And loss of currant members AND potential new members is very bad for the advancement of the medical research involved.
Not acceptable.

So how do you reward those of us who have been running the client since the beginning?

Sorry, Blackeagle. Those are the rules of this particular game. There are many games out there. If the rules here aren't acceptable, you would probably be happier on a different project.

Edit: BTW, you can go back over the history in this forum and see where that, and every other aspect of this project has been debated, sometimes to death.

[bwkaz]

Blackeagle, I think you may be confused.

Originally posted by Blackeagle
From what I can see this new scoring system has slashed scoring per day by 2/3's.

That will in effect enshrine those on the top half of the front page for good. (...)

Yes, this protein runs slower. If low production forces people away from the project, though, then maybe this is a good time for the rest of us to sneak up on them?

And it's to bad this wasn't run parralel to the old scoreing system first until it could be tweaked, not waiting to tweak it until after the fact.

There is no "tweaking" going on with the points calculations; they've used the same algorithm since the start of phase 2. You get 50 points for completing generation 0, then 50*sqrt(gen number) for any higher gen.

----

I think you're confusing the way stats are calculated with the way the algorithm works. A candidate fold's "score" is currently a combination of its energy, its secondary structure, and its Rgyr (don't ask me what that stuff is; I just read it in one of Howard's posts ). The protein with the best "score" is what's chosen at the end of each generation to use as the base for the next generation. With the last protein, the "score" of a candidate was its RMS deviation from the known structure of the real protein. This scoring is what's being experimented with.

Your personal score, your points, were explained above. The algorithm used to calculate those has not changed since the start of phase 2.

The other side of this is, the algorithm used to determine the score of a fold has very little effect on your production. The biggest thing slowing this protein down is not the changes to the way folds are scored; it's the general size and shape of the molecule in question. (Since it doubles-back on itself so often, the client gets stuck trying to place amino acids a lot more often, so it takes longer to finish a structure than it did before.)

[PCZ]

Blackeagle

I totally agree with you.
The current Protein scores very low and anybody trying to come into DF would as you say quickly realise they have no chance of catching up the established teams.

I and a lot of others are taking a holiday from DF and doing other projects
the scores are so low no one can catch up.

There doesn't have to be any difference in scores between the fast and slow proteins it simply needs an adjustment to the credit for each generation.

[Nofinger]

Yep, You guy's are right, it's a slow scoring protein and it is hard now to pass anyone.....

But it's not for the stat's we do this project or am I wrong

I know the stat's are big in a project but I didn't thought it was the main reason for someone to join a project.

Aren't we in this project for the science, I know I am.

[Grumpy]

There are always new people starting up, so there will always be those on equal ground to compete with, if you can get enough Boxen together, you can catch anyone Our top Folder for OCWorkbench, Bong88, went from last to where he is in 3 months..yes, last out of all the Folders here to 8th. Nothing is impossible

[Hua Luo Han]

its gonna take longer to hit the goal at 5 bil for the current protein.

Can we make it b4 X'mas ?

[PCZ]

Grumpy

Bong achieved his rapid rise because the previous protein was fast.
If he started now it would take him a very long time to catch up.

When the protein changes to a fast one you will see people returning and within a few days all the hard work of the folks plugging there way through this slow protein will be undone.

The point I want to make is that it shouldn't matter which protein people are working on the scores should be adjusted to ensure a similar output per generation.

[TheOtherPhil]

Originally posted by PCZ


The point I want to make is that it shouldn't matter which protein people are working on the scores should be adjusted to ensure a similar output per generation.

This is exactly how it is done now. As bwkaz has already explained:

You get 50 points for completing generation 0, then 50*sqrt(gen number) for any higher gen.

Every generation that you complete is scored exactly the same. The only difference is the speed in which you complete generations. If you stick around long enough, you will get a faster protein and you too will enjoy the speed (and personal stats score) increase.

[PCZ]

TheOtherPhil

I've been around long enough to have scored over half a billion points and 3rd place overall.

Perhaps you should remove that foot from your mouth now.

[HaloJones]

Since you've been around for so long, PCZ, you should know that this is a futile argument which has been going on as long as the project. Howard isn't going to change the scoring method, it's the same for everyone, slow proteins slow your catch-up to those you're catching and slow the catch-up for those catching you. Must we go through this every time we get a slow protein?

[Stardragon]

Just to clarify some matters:

What has changed: the way we evaluate the fitness of each generated structure, also referred to as the "fitness score".

What has NOT changed: the way we allocate points to users; i.e., the way a user's personal score gets calculated remains the SAME.

Also a couple of things to remind you all:
- the protein size will fluctuate back and forth, as it is impossible to find exclusively small and fast proteins for us to work on - if nature were that generous, we would have figured out folding a long time ago
- the first protein of Phase II was 96 residues, and the current protein is 108 residues - the difference is insignificant and this is well within the protein size range that we normally fold
- the current protein was already worked on in Phase I with the old algorithm, so we need to be able to compare our results - not to mention the fact that all our folders have gotten through it just fine the first time around, so I have fatih in you guys

Those of you who have been around long enough know that protein speeds will fluctuate as we find larger or smaller proteins to work on - this is just one more in an already established trend, so bear with us, you've been nothing but awesome so far

[PCZ]

Yes
when the solution is so simple.

[HansArne]

Originally posted by Blackeagle
From what I can see this new scoring system has slashed scoring per day by 2/3's.

That will in effect enshrine those on the top half of the front page for good. A new team try'n out DF will be quick to detect this and move on to a differant project that does not treat them this way.

Do other projects clear the states once a month because a new team might join?
If you get late into any project you will start at the bottom.
You can't expect the stats to start form zero, just because you didn't get in from the start

[TheOtherPhil]

Originally posted by PCZ
TheOtherPhil

I've been around long enough to have scored over half a billion points and 3rd place overall.

Perhaps you should remove that foot from your mouth now.

I am in the company of greatness.

My last sentence was directed at the people who keep saying they are leaving because of the way the scores are allocated and they don't stand a chance. I guess I could have worded it a little better.

[Blackeagle]

Originally posted by PCZ
Grumpy

Bong achieved his rapid rise because the previous protein was fast.
If he started now it would take him a very long time to catch up.

When the protein changes to a fast one you will see people returning and within a few days all the hard work of the folks plugging there way through this slow protein will be undone.

Agreed!

And they wouldn't leave if a method for balancing of the scoring system were in place. So this protien would be finished faster and more would be learned in a shorter time.

[Blackeagle]

Originally posted by HansArne
Do other projects clear the states once a month because a new team might join?
If you get late into any project you will start at the bottom.
You can't expect the stats to start form zero, just because you didn't get in from the start

First, I started near the beginning of phase 2, so that has no bearing on what I've posted.

My interest is a balanced scoring system, regardless of the differance in each protien. A balanced system would help to reward those who stay involved, and would help to get people to stay involved in each and all protiens.

I don't find it surprising this debate has come up repeatedly. And I suspect it will continue to come up, unless a balanced score system is put in place.

[bwkaz]

Originally posted by Blackeagle
...And I suspect it will continue to come up, unless a balanced score system is put in place.

"Balanced" according to who? You? Me? Howard?

There is no reason whatsoever (IMO) to scale the scores by taking the protein size into account. Because right now, everyone is running the same slow protein.

If you're not catching up to people on this protein, then you wouldn't have caught up to them on any other protein either, because they most likely just have more GHz running the client. This fact has nothing at all to do with protein size or speed; it has to do with the fact that if they're pulling away from you, then regardless of the protein speed, they would be doing the exact same thing. If you're catching up to them, then you would be catching up regardless of the protein speed. The only thing that the protein speed affects is how fast you catch up, or how fast they pull away.

Guess what: people that put more (in terms of GHz) into this project are going to get more (in terms of points) out of it. That's a simple fact of life, applicable to just about anything, and it's not going to change no matter how much people complain about it. Sorry.

[FoBoT]

Originally posted by HaloJones
Since you've been around for so long, PCZ, you should know that this is a futile argument which has been going on as long as the project. Howard isn't going to change the scoring method, it's the same for everyone, slow proteins slow your catch-up to those you're catching and slow the catch-up for those catching you. Must we go through this every time we get a slow protein?

what he said

[Paratima]

Cool off, I say! Ah... oops. Heh, sorry. Thought that was the fire extinguisher!

[Grumpy]

If we changed the value of Gravity, those who can jump higher than me will still do so, as will I jump higher than those who I could best before. One must learn how to crawl, Little Grasshopper, before one can run

[rsbriggs]

It would be very trivial to adjust the scoring based on protein size. Settle on some standard length, say 70, and multiply the score per generation by protein size / standard length.

This has a number of results.

1.) It equalizes things for people coming into the project during a slow protein. I'm certain that ARS, for example, has troubles getting/keeping new people involved - once a newbie notices that that the person immediately ahead of them on the team has 50 million points, and they find out they can produce maybe 1000 points per hour on a top end machine, well, who is going to want to stick things out for the next 6 years to move up one place on a team?

2.) It makes people more willing to crunch longer proteins, since there is no scoring "penalty" for doing so. You would get about the same score per time-unit of crunching performed - it wouldn't matter if the protein is size 64 long, or size 4000.

3.) It would probably help keep the project from losing crunchers (including those in the top 100) which is happening at an alarming rate. Competition keeps the project interesting and people motivated. The simple fact is, the vast majority of the people crunching do not have a doctorate degree in biology or biochemistry and don't understand the science well enough to get excited about it - the stats are the only thing keeping them entertained (unless they are entertained by reading threads like this, but I would point out that they can do that without doing any folding.) I certainly fall into this category.

And your gravity example is all very interesting, but consider it in the following way - you are racing up an endless flight of stairs. Previous gravity was such that people could jump up 100 stairs at a single bound. Current gravity is such that people can just barely manage to jump up a single stair. Would you enter that up-the-stairs race at this point with that disadvantage, or would you just go find another "more fair" race where the leaders aren't 3 billion steps out in front of you, by virtue of having done a lot of jumping when the gravity was different? Just something to think about...

I personally, along with a number of other top-100, soon-to-be-ex-folders, am out finding and participating in those different races, and we're finding them to be a heck of a lot more competitive (and just plain fun).

[tpdooley]

rsbriggs:
basing the scoring only on size isn't going to work - as we've had something like a 105AA protein that was faster than .. an 89AA or 65AA protein in Phase I. (Speed isn't soley based on size, it's also determined by how many times the protein gets twisted up and stuck.) They'd have to do something like figure out the time it took the "standard" protein to create 10 sets of gen 0; and then on the same hardware find out how long it took future proteins to have 10 sets of gen 0 created - and base the Scoring Factor on that. If the Scoring factor has to be based on gen1-250 creation, then it needs to be averaged from a large body of identical machines. (since the exact same hardware can take twice as long through one set of 250 gens as another set.)



I can see benefits both in having a Scoring Factor (if it actually keeps more of the folks folding on the project) and in not having it (those of us that stick to the project through fast and slow end up catching up to those ahead of us..

More important for the project, in my eyes, is to finish getting the client to the reliability that it had during the end of Phase I. Then things like speed of algorithm, the speed of our scores, and algorithm's ability to create the smallest rmsd structure would move to the prime focus of the coding..