Time To Repopulate A Que ~7 Days Of Tests

[vjs]

Well I know this was mentioned elsewhere but here are the facts.

Monday there are 5800 tests left in secondpass and we are doing about 800 tests perday. This mean it will probably run out sometime this coming weekend or next monday.

One of the que's need to be repopulated as soon as possible. I'd prefer we populate secondpass to 7 or 8M. This the que will be stocked with tests for some time after that.

Then at some time we can populate the firstpass que.

The other possibility is if secondpass has the identical priority as firstpass we could populate both. This way we don't run out of either.

I'm a little concerned about secondpass testing still... The dubious rate I know it's not error rate but it was quite high in the 7 and 8 M's. I'm wondering if it's not a good idea to to bring secondpass upto this level.

I also think now is the best time to doublecheck that we have matching residues for all tests n<5M. Not simply two tests (if there is some problem with residue versions etc) but actually two matches. Even if we had to do a thirdpass it would take too long and we would be done with these little ones once and for all.

[vjs]

I didn't mention that we have 1100 tests in the dropped que, this would give us a little extra window.

Either way I'm sure Alien will repopulate soon.

[Alien88]

Dave, Louie, and I are discussing how we want to proceed. However, keep in mind that we have a lot longer than 7 days. The tests are going to take longer and longer to complete as the N gets bigger.

[vjs]

Thanks for the quick reply Alien, we just don't have anyway of judging what's in the dropped test que besides 1163 tests with n>5.29M and how long it would take before those run out.

http://www.seventeenorbust.com/secret

I'm sure you guys will come to the correct conclusion, and thank everyone on our behalfs for doing an outstanding job. I know you guys are always told, do this do that, how come ???, error rates BLAH. IMHO you don't get enough praise for what you've done and we are all quick to see problems and comment. If you guys need any help or anything just post.

[Nuri]

I'm sure you haven't forgotten those, but two minor things for the to do list..

- Increase the max number for the graphs from 10m to, say 20m(?)
- Clean up the dropped-tests queue from 4847 tests, if any

[Frodo42]

We are almost down to 2500 tests in the secondpass queue now.
I am starting to feel a bit uneasy about it ... a runaway client or two and people would start getting "strange tests" or even worse they might stop to crunch (I am not sure what will happen if the second-pass queue should run out).

[Nuri]

I guess the server will start assigning the tests from the dropped-tests, error fix and garbage queues based on the pre-defined priorities. It will take at least two weeks for those tests to be finished after the secondpass queue is emptied.

In fact, it's a good opportunity to clean up those stuff.

Still, I agree that there is the risk of a runaway client. I guess in case we're caught off-guard to such a situation and all these three queues get emptied very quickly due to a runaway client, then the server will start assigning largest-prime (n=13.x m) tests.. ;)

[ShoeLace]

i wouldn't worry about it..

if they are slow populating first/second pass queues
there are 11331 tests in the largest-prime queue that will get handed out...

true thare are n=13M+ which is a bit of a jump frmo 10M but its still something.

[Joh14vers6]

Originally posted by Nuri
I guess the server will start assigning the tests from the dropped-tests, error fix and garbage queues based on the pre-defined priorities. It will take at least two weeks for those tests to be finished after the secondpass queue is emptied.

In fact, it's a good opportunity to clean up those stuff.

Still, I agree that there is the risk of a runaway client. I guess in case we're caught off-guard to such a situation and all these three queues get emptied very quickly due to a runaway client, then the server will start assigning largest-prime (n=13.x m) tests.. ;)
Originally posted by ShoeLace
i wouldn't worry about it..

if they are slow populating first/second pass queues
there are 11331 tests in the largest-prime queue that will get handed out...

true thare are n=13M+ which is a bit of a jump frmo 10M but its still something.
That is not the way the queues are configured right now. After assigning the tests from the dropped-tests queue there is nothing to assign for normal users.

http://www.free-dc.org/forum/showthread.php?s=&threadid=6717

[ShoeLace]

I see your point about the queue configuration.. but unless you know something I dont my understanding was that evey queue had a prioirty and that proirity was set by your login.

ie for normal users ,the GLOBAL queue was highest priorty.. then first-pass, dropped and garbage, then (not specified but) seconf pass and largest prime

if you use a username iwth QQQ<directive> then it altered the queue prioityies.. but not the available queues.


so (to restate) my understanding was that EVERYONE had EVERY queue just ordered in different prioirties.

Dave

[jaat]

Originally posted by ShoeLace

so (to restate) my understanding was that EVERYONE had EVERY queue just ordered in different prioirties.

Dave

That is not the case. Last time when first pass went empty, clients did not start getting tests from other queues automatically but things had to changed.

jaat

[Nuri]

I remember that... I was under the impression that this problem was solved that last time when first pass went empty... May be a bit of wishful thinking... ;)

Soon we'll see... cross your fingers!!!

[Mystwalker]

660 tests done in the last 24 hours, 694 tests remain...

[Nuri]

568 left.... and counting ;)

[Mystwalker]

542...
Anyone want to set up a poll? :D

[Nuri]

I just want to count... and feel the stresssssssssss :looney:

[Vato]

...459...

[Vato]

...228...

I've swapped my QQQsecondpass machine back to normal, since I've no idea what will happen when this runs out....

[Keroberts1]

it looks as if the queue has been populated with al of the tests that do not have matching residues. Perhaps this will give us a better idea ofthe error rates. These tests should not last long thoug hbecause many of them are very small and will only take an hour or so a piece to finish.

[Nuri]

I retired my slow machine a couple of days ago when I thought it finished it's last n<5m test... it seems like there is more to be done (although not much), so she's back for a couple of days ;)

[Keroberts1]

my guess is that the queue will still empty later today

[Nuri]

it will probably take 2-3 days... we'll see ;)

[Greenbank]

Luckily I'm really busy at work so I can't spend hours staring at the stats pages ;-)

85 minutes until my P4 finishes it's 4.9M n and goes back to ~3.3M. Oooh.

[Nuri]

I feel like it will get something within the range of ~3.6m... :p

[Matt]

Yeah so what IS gonna happen to QQQsecondpass machines when the queue runs out? These machines - in my case - are slower machines and will takes AGES to complete a first pass, will there be no place for them :'(

[Nuri]

Which ks are you getting??

I do not have access to my PRP running machines right now, but I feel like something strange is happening.. It looks like only k=55459 is being redistributed when I look at the the pending distribution tests graphs... Is that the case? If so, any ideas?

[Greenbank]

The second-pass queue should not run out and, in theory, will never run out until the 17th prime is found.

I would expect the plan is something like this for QQQsecondpass.

Assign from second-pass, if second-pass empty, assign from error-fix.

Obviously some of the tests completed will be 'dubious' and will go back into one of these queues.

Eventually both of these will be cleared out (there are still 9413 tests below 5M that haven't even been assigned yet, and probably another 3000 pending).

In which case the second-pass queue can be populated with all of the completed first-pass tests where 5M < n < 10M.

The admins could do this now but then they want to keep the queue sizes down so that we can all see what is going on. (i.e. 3711 second-pass tests left < 5M and 5702 in error-fix).

Yes slow machines will take ages to do some of the tests, but then that doesn't matter. It's only a problem if they take a really huge amount of time to complete a test.

If you've still got any REALLY slow machines lying around (i.e. more than 100 days for a test at 5M) then the best idea is to turn it off, use it as a home webserver (if you don't already have one), give it to charity or just throw it away.

It seems kind of wierd to give something away that works perfectly especially as, at the time, it was probably a quick machine and cost lots of money. I remember spending 1000 UKP on a 486 DX2/66 at University. I threw that away two years ago.

My slowest machine I have (my work laptop that I've resisted an upgrade on for years because it just works) is a mighty PIII 700MHz. It's current test is n=4.35M and it will take around 20 days to do that which is fine.

[ EDIT ]

Before people jump down my throat about not using REALLY slow machines, consider two things.

1) The longer a machine runs the greater the chances for a random error to flip a bit somewhere and produce a dubious result.
2) How much does it cost to run that machine for a year? How many tests can it complete in that time?

At a rough guess a high end PC will use about 50 UKP of electricity (UK prices obviously) a year. An old Pentium 120MHz will still use about 30 UKP of electricity a year and probably complete 2 tests at n=5M a year! 15 UKP a test!

[Greenbank]

Originally posted by Nuri
Which ks are you getting??

I do not have access to my PRP running machines right now, but I feel like something strange is happening.. It looks like only k=55459 is being redistributed when I look at the the pending distribution tests graphs... Is that the case? If so, any ideas?

k=21181, n=3499868 obtained 60 seconds ago.

Don't forget you can go to www.seventeenorbust.com -> Preferences -> "Pending Test Management" to see how your PRP machines are progressing.

[Nuri]

ok. thx.

[vjs]

Humm,

Some interesting stuff going on... wonder what's up.

Also for those using slow machines, there is still sieve a p-400 will actually crunch through a 500G sieve range in a reasonable period of time. Matt's done a fantastic job getting sieve mainstream with his reservation system. IMHO everything less than 1G should probably be sieving.

Also wondering if we should put a couple machines onto garbage presently? I know we don't get personal credit but those n<1M tests look interesting. Probably best to leave them in secondpass however since it looks like there was some manual que seeding. Pretty obvious that secondpass is probably 3rd pas or ???

[Greenbank]

By garbage you mean supersecret? ;-)

[Nuri]

which user grabs from error-fix queue?

[Greenbank]

OK, so both VJS and myself were right. Sorry VJS!

"
The secret and supersecret users get tests from the second-pass and error-fix queues.

The garbage user gets tests from the garbage, dropped-tests and error-fix queues.
"

For people who don't care about personal stats you can use either of these.

I've just moved over one of my machines to supersecret. The 718706 test is 9.74% complete after 112 seconds :-)

[vjs]

dropped-tests 1237 4709191 13467862 30 +2022624
error-fix 5697 927778 4996571 1 +223406
first-pass 2 5372 6223 9 +∞
garbage 1058 5201806 7594947 0 n/a
second-pass 3384 3880901 4999946 997 +3696574

These ques are a little interesting...

firstpass 2 tests n=5372 and n=6223 humm... those would be quick!!!
secondpass 3384 tests between 3.8M and 5M looks like a handpicked population or ???

And does anyone remember the number of tests in either of the error-fix and garbage ques were... say a month ago?

I'm thinking that they are making darn sure we have two matching residues for everything n<5M before we proceed any further. I'm also very happy that these things are being cleaned up.

It might be useful to truncate the database for all n<5M and create new ques, this might be what's going on. Regardless looks like we should know by next week.

[kelman66]

Originally posted by Greenbank
If you've still got any REALLY slow machines lying around (i.e. more than 100 days for a test at 5M) then the best idea is to turn it off, use it as a home webserver (if you don't already have one), give it to charity or just throw it away.[/B]

OMG, surely you must have meant to say to RECYCLE the computer.

Throwing it away would be ignorant.

[Joh14vers6]

Originally posted by vjs
dropped-tests 1237 4709191 13467862 30 +2022624
error-fix 5697 927778 4996571 1 +223406
first-pass 2 5372 6223 9 +∞
garbage 1058 5201806 7594947 0 n/a
second-pass 3384 3880901 4999946 997 +3696574

These ques are a little interesting...

firstpass 2 tests n=5372 and n=6223 humm... those would be quick!!!
secondpass 3384 tests between 3.8M and 5M looks like a handpicked population or ???

And does anyone remember the number of tests in either of the error-fix and garbage ques were... say a month ago?

I'm thinking that they are making darn sure we have two matching residues for everything n<5M before we proceed any further. I'm also very happy that these things are being cleaned up.

It might be useful to truncate the database for all n<5M and create new ques, this might be what's going on. Regardless looks like we should know by next week.
How do I (we) get first-pass tests?

With a n that low, we should have crunched that already.:(

[Nuri]

cool

[vjs]

Nuri,

I'm totally guessing on the next week its just what things look like. Anyone else have opinions or beter yet does anyone want to tell us exactly what's going on?

[Joh14vers6]

Originally posted by Joh14vers6
How do I (we) get first-pass tests?

With a n that low, we should have crunched that already.:(

The first-pass queue is again empty also the largest-prime queue is empty.

[Keroberts1]

now it seems error fix has been populated also. This is funny i though we were doing error fix work when the DC queue was refilled again. If anyone knows where r these new tests comming from? Do we already have residues for these? maybe 2? Or maybe are they just for error rate determination?

[Greenbank]

Originally posted by kelman66
OMG, surely you must have meant to say to RECYCLE the computer.

Throwing it away would be ignorant.

Yes, I should have said recycle. I think I said that in the other thread I posted something like this in.

The local dump has a recycling area. There's a seperate section for TVs, computers and other electrical stuff. It has a nice big sign outside saying "You are not allowed to remove items from this section. Doing so will result in prosecution for theft." Any time I've been down there there have been 20-30 computers sitting there (mostly gutted). It's nice to add to that pile.

I see a few other people have picked up on doing error-fix work, my Xeon has done 14 tests in the 14 hours it's been running on supersecret :-)

[Greenbank]

Originally posted by Keroberts1
now it seems error fix has been populated also. This is funny i though we were doing error fix work when the DC queue was refilled again. If anyone knows where r these new tests comming from? Do we already have residues for these? maybe 2? Or maybe are they just for error rate determination?

OK, here's how it works.

first-pass tests for n < 5M were completed a while back. All of these were added to the second-pass test queue. So for each of these tests we have a residue, lets call it F.

Now, these tests are being pulled from second-pass and another test is performed. Let's say it comes back with residue S.

If F == S (i.e. they are the same) then the test is removed from all queues.

If F != S then it is placed in the error-fix queue.

(In the posts about error rates you'll remember they talked about 'dubious' tests. Since these two results (F and S) do not match then both of them are marked as dubious.)

Now, someone pulls this test from the error-fix queue, completes it with a residue we call E.

if F == E (i.e. the residues of the first-pass and error-fix tests match) then the test performed by the client that got it from the second-pass queue was incorrect. Since we have two matching residues we can remove this test from all queues and consider it not prime.

(Because F == E both would be marked as correct (F was marked as dubious) and S would be move from 'dubious' to 'error'.)

if S == E (i.e. the residues of the second-pass and error-fix tests match) then the test performed by the client that got it from the first-pass queue was incorrect. Since we have two matching residues we can remove this test from all queues and consider it not prime.

(S and E would be marked as 'correct' and the first-pass test F would move from 'dubious' to error').

if S != E and F != E (i.e. none of the residues match) then we are nowhere closer to knowing if it is definitely composite (we don't have two residues that match) or definitely prime (all 3 tests S, F, and E could all be incorrect) so it goes back into the error-fix queue.

Someone will pick up this test, complete it, and return yet another residue (lets call it E2).

If E2 matches any of S, E or F then we can remove it from the queues, otherwise it goes back into error-fix again.

3 tests I can imagine, it would be quite a rare occurence for more than 3 tests to be required before two residues matched.

To calculate error rates you have to know how many tests were either 'correct' or 'error'. Counting 'dubious' tests proves nothing, if you have two differing residues from a test (A and B) you don't know if one of them is correct (and the next test will confirm that) or whether BOTH A and B are incorrect.