YES!! Another poll for secondpass

[vjs]

YES!! Another poll for secondpass

Well we have been running secondpass for quite some time now, bringing the secondpass level up from ~2.5M to 5M. It appears that the creators are dilgently working on ensuring we have matching residues for all k/n pairs for non prime k's with n<5M.

My hats off to you guys good work

Did this effort pay off? Yes and then some, k=4847 fell to the second round of beating and who knows how high we would have had to test this k???
The work saved during this effort was well worth any frustration.

With the above in mind I though it was time to create another poll... what is the question? Same as before

How far would you like to see secondpass get before the firstpass que is repopulated?

Previous results:

3.5M I think we are far enough now... repopulate the que 6 11.76%
4M (4,000,000) 8 15.69%
4.5M (4,500,000) 2 3.92%
5M (5,000,000) 17 33.33%
6M (6,000,000) 8 15.69%
7M (7,000,000) 10 19.61%

A little less than 2/3'rds of the voters at that time thought we should repopulate at 5M... Note that this was before the missing prime was uncovered.

I ask the same question again...



How far would you like to see secondpass get before the firstpass que is repopulated?

I think we are far enough now... repopulate the que (5M)
5M
6M
7M
8M
Take one or two k's up to 8M first

I've allowed multiple choices, please choose 1 of the top 4 and decide on selecting the 5th or not selecting the fifth.

[Nuri]

I think 5m is enough for the time being...

Still, we might consider hand picking and retesting the remaining k/n pairs within the 5m < n < 10m range for;

- the client versions where the error rate is significantly high (>20% ?) and
- the users who has very high error rates (>20% ?)

[vjs]

I can see stopping at 5M however I'd personally like to see secondpass reach 6M and it wouldn't take that long. Only reason I say this is our current client seems stable and the next client may produce different residues etc. It would be a shame to tripple check everything later.

Personally I voted for 6M and the running one heavy k or two light k's out to 8M.

My reasoning behind the 1 or 2 k's is quite simple, they would be a fairly representive sample of what we will get as secondpass increases.

We could then handpick as you suggested.


Inaddition our current secondpass effort has given us a good head start on the P-1 effort, P-1 testing everything between 10M and 10.5M. We have also increased the 90% sieve level from 1017T to ~1057T during these efforts. So needless to say when prp starts up again there will be less wasted tests.

[Joh14vers6]

For the project it would be wisely to choose to go up till 10M and from then on start to populate under all users second-pass tests after 1 month after the first-pass result came in and so on start to populate under all users error-fix tests after 1 month after second-pass came in.

With the actualy scoringsystem in my mind, all members of DPC are already tired of doing these second-pass and error-fix tests. That explains the high score for populating the first-pass queue at 5M or earlier.

[Greenbank]

5M for me, but then I'll be switching all of my machines back to QQQsecondpass anyway :-)

[Greenbank]

Not long until min n of error-fix reaches that of the dropped-tests and then second-pass queues and tests get assigned from all 3 of these queues.

dropped-tests should be pretty sparse (I'd guess only 150 of the 1261 tests currently in this queue are below 5M).

That leaves 1326 in error-fix and 2965 in second-pass.

Also remember that this is number of tests awaiting assignment. It may be some time before all of the tests are returned as completed. There must be a few slow clients out there still grabbing tests.

I know I accidently dropped a test for k=55459,n=806386. I can't expire it as I had originally grabbed it using the supersecret account. There are probably a few hundred tests out there below 5M that will take a while (up to a month) before they expire into the dropped-tests queue.

[Keroberts1]

i just don't understnad where all of the new testrs that were added to the DC queue came fro m after we just about finished off n less than 5 milllion I thought orror fix was non-matching residues so what are these values from? Are these triple checks?

[Greenbank]

Here are the queues:-

First-pass: Tests that haven't been performed at all yet.

Dropped-tests: Tests that were assigned to someone but then dropped before being completed. No residue exists for these tests.

Second-Pass: Tests that have been performed once (in first-pass). They need to be performed again to see if the residues match.

Error-Fix: A second-pass test was performed but the residues did not match. Another test is required to find a matching residue.


There are so many tests in error-fix because there are so many tests coming back with residues that do not match the residue of the first-pass test.

Where we are now (3M to 5M) about 1 in 20 of the tests are dubious and therefore need retesting. See error rate discussion here:-

http://www.free-dc.org/forum/showthr...5&pagenumber=1 )

[Greenbank]

And the bit I forgot to add.

Imagine we have just completed first-pass all the way up to 5M and no second-pass testing has been performed. Say there are 50,000 tests between 0 and 5M.

After we finished first-pass the second-pass queue would hold all 50,000 tests.

Now consider the error rate is 5% (pretty much what we've seen).

So 5% of the first-pass tests will be incorrect, but since we don't have a second-pass test to compare this to we believe they are all correct.

Now we test all 50,000 again. 5% again (but maybe not the same 5%) of the tests will produce incorrect residues.

So 47500 first-pass tests will be correct and 2500 incorrect.

On average 5% of 5% of the tests (125 of them) will have obtained incorrect results in both tests.

So what does this mean for the queues.

95% of the 5% of tests that were incorrect first time will be computed correctly second time. 50,000 * 0.05 * 0.95 = 2375

95% of the 5% of tests that were correct first time will be computed incorrectly in second-pass. 50,000 * 0.05 * 0.95 = 2375

So, after this second-pass stage we will have:-

2375 + 2375 = 4750 tests which we have two residues for, one correct and one incorrect.

These 4750 tests will sit in the error-fix queue.

125 tests will have been computed incorrectly both times, and again these tests will sit in the error-fix queue.

The remaining tests (50,000 - (4750+125)) = 45125 will have two matching residues and therefore will be removed from all queues.

So with an error rate of 5% after two rounds of testing we will have 9.75% of tests reamaining in the queues.

Next stage we have two groups to consider. 4750 tests where we have one correct residue and 125 tests where we have no correct residues.

If we do a third-pass on each of these tests the following will happen:

From 4750 tests ~ 4512 tests will be computed correctly and we will have two matching residues.

237 (rounding down) will be computed incorrectly again. These go back into the error-fix queue.

Of the 125 tests where both previous tests failed, we still need to perform these tests at least twice more (since both residues we currently have are both incorrect).

What's worse is that of these 125 tests, roughly 6 of them will produce incorrect results again and we still won't have a correctly computed residue for them.

Eventually though we will get two correctly computed residues for every k,n pair and all of them will be removed from the current queues.

[vjs]

Great analysis Greenbank hopefully others will follow:

The numbers also match what we are seeing with que population as far as I can see. 5% error rate also doesn't seem that bad, I'm sure this value will increase and decrease depending on alot of factors (clients, n, CPU speeds etc).

Perhaps someone would like to calculate the optimal secondpass level considering a 5% error rate:

Things to consider during this calculation:

- 5% error rate basically means one test in 20 are wrong.
- There is a greater chance of finding a prime at a lower n (prime density)
This would factor towards higher secondpass level
- Work lost due to missed prime (this is hard to judge)
- Work lost by conducting firstpass before factor is found, (I think we could assume that 1 in 33 firstpass tests (3%) will be factored before they reach secondpass, ??optimistic??)
I think this can be guesstimated from Mikes pages.

-----------------------

My very simple guess before was 1 in 20... Now considering the above...

Perhaps this can be over simplified by just guessing that a minimum error rate is 5% which will only increase. Then assuming everything else prime density, lost work, factors, simply brings the value of a second pass up by a factor of two.

I.e. 1 in 10 or a 10% rate of return on secondpass

I.E. Time to complete one firstpass test should equal 10x the time to complete a secondpass... we know it should be less than 20.

So if n=11m what is the time requirements for one test t(11m) and where should secondpass n be in order to complete 10, 15 , 20 secondpass tests in the same time period.

What is n for:

t(11m) / 10 = t(?n)
t(11m) / 15 = t(?n)
t(11m) / 20 = t(?n)

[vjs]

Greenbank,

I know I accidently dropped a test for k=55459,n=806386. I can't expire it as I had originally grabbed it using the supersecret account. There are probably a few hundred tests out there below 5M that will take a while (up to a month) before they expire into the dropped-tests queue.

This one may stick in the que for some time... if your sure about it you could always do the lost test. You don't need the z file, simply change your name to supersecret the k and n in the registry and don't submit intermediate blocks.

Basically you would complete the test and remove it from the que.

------------------

ANother point is if louie or dave (someone who knows the secret, garbage, supersecret, 31337, account) If they could go in an drop all tests which are not being processed this may be a good move at this point.

If you e-mailed me the passwords I'd take care of it for you and from time to time if you wish.

[Greenbank]

I suppose you could fiddle with the registry on a windows machine and see how long it takes for a few blocks of a test for each n you want and then extrapolate.

I don't want to interrupt any of my machines to do this (plus I'm off home soon as it is 6:30pm).

[vjs]

I was thinking about doing this but I was sort of hoping someone could simply check their logs etc...

To be honest with you it's been some time since I actually ran prp, I mostly do sieve or P-1, since I'm totally AMD here. It would be nice if someone with a P4 or Athlon64 etc could do it.

[Greenbank]

But no-one has crunched any tests 10M (end of first-pass) < n < 13M (largest prime queue).

You also need the results for the same machine, there's no point even comparing times from a P4 3.0 GHz Prescott and a P4 3.0 GHz Northwood as they differ so much due to FP instruction latencies and memory bandwidth speeds.

[Greenbank]

By the way, there are still 2 FIRST PASS tests of n < 5M that haven't been completed yet :-)

33661•2^4757472+1
assigned: Oct 5 2003
last update: Nov 11 2005 18:30
percent complete: 55%
cEMs/s: 896

22699•2^3625174+1
assigned: May 6 2003
last update: Nov 12 2005 13:40
percent complete: 93%
cEMs/s: 559

Yup, that last one is a whopping 559 cEMs/sec compared to the 720690 cEMs/sec my 3.0 GHz Xeon kicks out. ;-)

[Keroberts1]

i understand the Double check and theerror rates effecting the error fix queue. What i was curious about was after we returned to DC we had already obtained a first residue for all tests under 5000000. So why was it that after we pushed DC up to 5000000 another 4,000 tests were then added to the DC queue and 4000 to the error fix queue. Are we testing some that already have matching residues? Are these tests where we had lost or never saved the original residues? The best response for this would be from one of the admins because obviously there was some manuel intervention here and only they could definatly know what they did.

Also i believe there are residues for those ultra low tests below % million (from first pass) I believe these tests were reassigned a long time ago. There should probably be a limit to how long a single tests can be out like 6 months or something before it expires reguardless of intermediate block submition. Especially if there appears to be so little work going into it that it will take another few years to complete.

[Joh14vers6]

Originally posted by Greenbank
By the way, there are still 2 FIRST PASS tests of n < 5M that haven't been completed yet :-)

33661•2^4757472+1
assigned: Oct 5 2003
last update: Nov 11 2005 18:30
percent complete: 55%
cEMs/s: 896

22699•2^3625174+1
assigned: May 6 2003
last update: Nov 12 2005 13:40
percent complete: 93%
cEMs/s: 559

Yup, that last one is a whopping 559 cEMs/sec compared to the 720690 cEMs/sec my 3.0 GHz Xeon kicks out. ;-)

No, this where the garbage-queue kicks in. Reading the definition of the garbage-queue these tests above are fallen in the garbage-queue.

[Greenbank]

Indeed, must engage brain next time.

[Greenbank]

About 2 days left at the current rate before the queues are exhausted of tests below 5M...

[maddog1]

Well, now I can see tests up to 6m in the 2nd pass test queue...guess this answers what the admin's decision is

[vjs]

I've closed the poll since the injection of n<6M thanks for voting.

16 people "we have gone far enough with secondpass", 18 people wanted 6M or higher...

I'm not including the 11 who wanted to see 1 or 2 k's populated to 8M in the stats but I guess they would have voted for 6M or higher not sure...

The interesting divide is less than 50% of the people who voted wanted to see secondpass extend beyond 7M. I think this is reasonable reflection of how the admins populated the que. I guess we will see what happens by 6M.

[Matt]

Do we ever get any "official" word from admins about any of this stuff? It seems to me we all seem to discuss and analyse what happens and presumably the admins read what we say and take some notice of it, is there often any feedback from the admins in the forums? I'm not even sure who really does what, some people seem to know a lot about what goes on but I don't know if they're admins or just a bit obsessed!

[Alien88]

Originally posted by Matt
Do we ever get any "official" word from admins about any of this stuff? It seems to me we all seem to discuss and analyse what happens and presumably the admins read what we say and take some notice of it, is there often any feedback from the admins in the forums? I'm not even sure who really does what, some people seem to know a lot about what goes on but I don't know if they're admins or just a bit obsessed!

The only admins are myself, louie, and dave (kugano). I frequent the forums and pass feedback to Dave and Louie quite often, they visit every now and then, mostly depending on time.

[Matt]

oh cool! cheers for clearing that up