I'm curious why the n prp line has stopped to a crawl today. Has the server started reassigning tests from the V2.20 mishap?
Some v2.2 retests are in play. Some tests for error rate estimation are in play. Some tests are reassigned for residues we don't have confidence in.
We're doing a lot of dabbling with the queues right now. Got to make sure a prime can't get past us.
Cheers,
Louie
In the meantime check out this neat graph of n vs. assignment time. Gives an interesting look at where we've been testing over the years. The major break in November 2002 is when we went from testing one multiplier to testing seventeen multipliers.
Bonus points to those who can figure out some of the other "quirks" of this graph =)
good I've been wanting to se that for a while
there alot of DC going out for the main line i see a huge speed increase there
Dunno what you're looking at, but I don't think it's the second-pass line (the real second-pass line doesn't show a recent increase at all... it hasn't been long enough yet).
Challenge questions to keep you busy:
[list=1][*]Why are there three or four "tiers" in the data at the upper threshold?[*]What is the curve which starts in June 2004 and runs to the present at around n = 4.5M?[*]What is the spike in the upper threshold in June 2003?[*]What are the long, totally flat horizontal lines?[*]What are the short "squiggly" lines that intersect the slowly-increasing line at the bottom, starting around June 2004?[*]What is the month-long gap in the data at the upper threshold in January 2004? (I don't know the answer to this myself)[*]What is the vertical line at the far right edge?[/list=1]
#7, my guess is the recent reassignment of a lot of error tests, and double checking for the error ratio, these test cover most of the already checked n range.
#4, is a slow computer reporting on an exponent over the period of many months
#2, garbage account?
#1, dropped tests?
#3, runaway client?
#6, there is also a gap like this in jan of 03 (smaller), maybe it has to do with something getting cleared out of the database after a prime is found?
not sure about #5
Last edited by pixl97; 01-11-2005 at 01:35 AM.
where are these graphs youre talking about? I was looking at www.seventeenorbust.com/secret/ and it says the second pass que has incresed by almost 100,000 in the last 24 hours
What? You saw it. The link from my post, the third one in this thread.where are these graphs youre talking about?
Here's a much bigger version of the same graph. A bit easier to pick out fine details.
I'll give it a go...
But first off, There is one assumption that has to be made regarding the data, These points either represent a completed reside returned, the n of returned blocks, or assigned tests. (There is a difference)
1. Assuming it has to due with returned residues or blocks, it probably the difference between the clients or machines. (i.e.) Time processing time differences between all P4's, AMD's, older slower machines.
That's why these lines and at large exponent are slightly "smugged" compared to those of low exponents (many computers different times of completion vs a 1 or 2 computers constant time of completion). Also the reason why the difference between theses lines is less at lower n.
2. The garbage que
3. ?? Someone doing a massive dump on manually inputted k/n sequence using a que program. Submitting all results in a short peroid of time once prp caught up.
~june 2003 someone on a university campus or school messing around with an entire lab once school is out, etc... They probably started hording back in May etc.
You can actually see it's effect on the main line slighly exagerated slope until prp catches up to their high n?
4. Very curious unless the graph reports submitted blocks, if it's submitted blocks it's probably a slow computer working on a paticular k/n test from, old secret = 2m, old garbage -4M. If it's residues, someone submitting the same residue over and over, or a k/n pair stuck in the server que continully being reassigned (your validity check etc???, "command line testing" (lets leave it at that).
5. The short squiggly lines are those for the, residue-recovery, supersecret testing.
I know exactly what those hairy lines are starting around mid april 2004, you will have to PM me if you want an answer... But you are obviously checking those for a third time...
Look at mid-april 04 to early june 2004, they were later done a third time, mid june.
All of these short curls correspond to tripple checks later...
7. THE END OF THE GRAPH
Seriously... your error checks, these points must represent returned blocks right not residues, correct.
6. This is interesting... Ignored/lost resides by the server... it's after the prime, perhaps that prime or k/n pair was being handed out alot by accident lowest n etc. Later the reisdues were ignored...
Smaller such break at june 2004, again at late jan 2003, sort of sept 04, there are quite a few of them, but some are few small. SOmething to due with backups, factor submissions removing tests etc. Are factors applied to the DB daily to remove residues/tests or is that a manual thing once a month or at these breaks...
Let me know do I get an A, B, etc..., elevated to guru or class clown
Also if it is blocks you don't have the graph extending upto 14M, I know some of the ~13.3M tests have been done.
Also you can see a line around n=5054502, is this possibly people testing the,
prime found: 5359•2^5054502+1 over and over again,
Same goes for the other primes.
The line at n=2.6M has me wondering, what is that??
It's also interesting to note the progress of residue-recovery.
Not going to comment yet on the "correct" answers, but here's a few clarifications.
First, a point means a test was assigned with that n at that time. Period. It does not show "intermediate" block reports, or submissions... just assignments. The graph gives no information on how long it took to finish the test after it was assigned, or even whether or not the test was completed at all.
It only goes up to 9M because I clipped it that way. There are indeed data points up around 13M from the "largest-prime" queue... I just cut them off because I didn't think they were very interesting.
BTW, some very good guesses so far =) And some not-quite-right guesses...
O.K. so one point per test, regarless of submission or block or residue, simply assignment.
But is it really "assignment by the server" or "one point represents acknowledgement by the server of a user doing a test at that n and time"?
It really makes a difference in the interpretation, also how are factors handled and how do they effect these points...
10M is cut off so I can't tell if those 10M tests inserted actually show up on this graph. Would have those 10M tests shown up on the graph if they weren't deleted???
And also would a factor remove the point from the graph, i.e. record of the test from the server?
This is very interesting... thanks alot of the ?game? it put a smile on my face this morning. Is there going to be an answer sheet later or a who done it ending...
Yes, I'll give all the correct answers eventually...
Factors don't affect this graph in any way. We have never deleted, nor will we ever delete, test histories, even if a factor is found for a number that was already tested. The very first tests ever assigned (on April 1, 2002) show up on this graph. The points are assignments, not "acknowledgements of work done by a client." If someone reported a test that the server didn't assign, it will not appear on this graph.
There will be a point at (x, y) if and only if the server assigned a test to some client with n = y at time t = x.
Put another way, the points come from this SQL query:
And yes, if I hadn't clipped the graph, you would have seen points above 10M.Code:SELECT assign_time, n FROM proth_tests;
7) The 100 double checks just seeded into the main queue
If these points were completions instead of assignments then 1) could represent the different speed machines. Could it be the diffferent K's? Are the tests handed out it strict n sequence? or something else?
ps
What program/package did you use to create the plot?
Joe O
Maybe... but the points aren't completions, they're assignments. In any case I doubt the lines would be so distinct and regular if that were the case... there's a huge variety of machines. It would probably look largely random.If these points were completions instead of assignments then 1) could represent the different speed machines.
gnuplot.What program/package did you use to create the plot?
O.K. then alot of my above points are only half correct...
1)
Looking at the large version one can see, a significant number of tests assigned are from the dropped que (second tier), and assigned a third time in the third tier, all points below etc... The spacing between these tiers is approximately 30 days. So that seems to fit, common n=y tier 1 and common n=y teir 2 have a spacing of ~30 days in X=time teir 1 vs X1=time tier 2.
You can also see this effect at the highest x,y of people dropping tests.
Not sure how I feel about this.
This is also exemplified by the crossing of garbage with the n=4M tests in december.
I don't mean to be rude, but if they are truly assignments "only by the server" can you explain. Whats happening in the region of:
April 20,2004 to june 5th 2004
From n=doublecheck to n=1m?
A reverse question, humm...
This thread is was makes SoB such a great project, how often do you get to see what's behind the scenes!!!!
I haven't a clue... it started just before we implemented the new queue system on June 7, 2004, so it's a lot harder for me to tell what's going on there. (I have another version of this graph color-coded by what queue the test came from. I'll post that when I post the correct answers to the seven questions.)I don't mean to be rude, but if they are truly assignments "only by the server" can you explain. Whats happening in the region of:
April 20,2004 to june 5th 2004
From n=doublecheck to n=1m?
The tests you're talking about go up to exactly n = 800,000, and they stopped exactly when the new queue system was implemented. Extra bonus questions for whoever figures this one out! =P
When I look at www.seventeenorbust.com/secret/ I see that the numbers at Overall are incorrect. What means there is 1 or more queue.Originally posted by Keroberts1
where are these graphs youre talking about? I was looking at www.seventeenorbust.com/secret/ and it says the second pass que has incresed by almost 100,000 in the last 24 hours
The sum of #tests of all shown queues is lower than the number at Overall.
At Min n at Overall is a lower n-factor than seen in all queues.
The sum of #Done tests of all shown tests is lower than the number at Overall.
The sum of n increase of all shown tests is higher than the number at Overall.
Looking into http://www.seventeenorbust.com/stats/textStats.mhtml I found:
User holepatch is getting directly work out the queue "residue-recovery".
User Clitest (speaks for it self).
User lowk (speaks for it self).
No clue for a missing queue.
Answer for bonus points...
http://www.free-dc.org/forum/showthr...5&pagenumber=1
http://www.free-dc.org/forum/showthr...&threadid=6499
And the graph is of submitted residues not assigned tests.
Like I said It's me so the only thing I'll get is a booby prize.
4. Con't
Also those horizontal lines of tests are probably a client that is running and not updating the registry... How this can happen I know but shouldn't post, or they have the registry locked down so that it can't be updated by a program, etc etc etc
7. I'll agree with Joe, I didn't realize there was a time lag in the graph these are completed tests from those you injected.
6. The month long gap has to due with extending the reassign time out after the prime. Did this happen or not???
5. Some of the Short squiggly lines are attributable to one thing some another. The ones going up above doublecheck are those of people creating their own tests for a particular k. They are not all me...
What's happening around mid June 2004 has 2 results from 3 actions... (picture below.
a. Main test (from any user first pass prp reside) then Double check (supersecret, secret, user only second pass residue) --> server understands a first pass residue and doublecheck residue has been made they match and advances the DC-prp line accordingly.
b. Main test (from any user first pass prp reside) vs Double check (non "proper account" second pass prp reside) --> server recongises a test has been tested twice residues match and advances DC-prp line accordingly not assigning the test for the third time.
c. Manual server update, Main test residues are compared against second pass resides looking for errors. All residues from any user first pass prp are compared only against those resides from supersecret, secret, garbage. Second test reside is not found for test completed by another account although correct.
d. Those tests that don't have second residues from supersecret, secret, garbage are reinserted into the que system.
e. The DC line advances quickly since ther are only a few.
Last edited by vjs; 01-11-2005 at 05:28 PM.
I didn't want to comment on these accounts since these are the (previously) unknown non-public accounts...
User holepatch is getting directly work out the queue "residue-recovery".
User Clitest (speaks for itself)<-- this would be responside for some of the horizontal lines since there are currently 4 active pending tests (what I didn't want to post). I only see 3 lines however.
User lowk (speaks for itself) this is the progress line, step wise ending at 1.4M.
One thing that I didn't consider is the quad xeon that tested all of those results really turned in garbage and every test reside was a mismatch...
If this is the case it's even more disturbing, since I did all of those test twice to validate the machines integrety and none of those test mismatched internally.
Only one set of residues was returned for each test.
With all of the above being said, I'm not sure if the server is correctly identifiying double "tested" tests based upon residues, perhaps this is part of the confusion...
Server is thinking match residues submitted twice, and reassigning seconds based upon que assignment to a user...
The above is probably only causing a problem if the test was never assigned in the first place. But not considering that a residue could be submitted for a test it had never assigned in the first place.
The only problem with the above logic is the 3rd and 4th tiers (curves) at higher values. If I were to guess, I'd say what's happening is a user is taking longer than 30 day to complete the test and it's reassigned. This is great but if the second user also takes longer than 30 days it's reassigned again, fourth time also but it's harder to make out the fourth tier "curve" and only possible at high n.
Now is the tests only "not reassigned" once a residue is submitted? I'd answer yes. So what this leads to is a tremendous number of double checks done above the n=1m double check limit. Which is just fine...
But we this mean that we must consider "compare" all resides submitted from all users on a first returned residue bases, to a second residue returned by any user.
I'm pretty sure this is being done already and it's what you have stated on this board before. I just wonder if it's worth double checking if the server is behaving correctly.
ALso on question 3...
I'm very confidant that it's some user subitting all values "first pass prp"<n<4M for 1 k.
No!! Again, the graph is not of submitted residues, it's of assigned tests.And the graph is of submitted residues not assigned tests.
I'm also curious what happened mid may 2004 with the first pass prp.
Alot of test at various n on one day then nothing for 3 days following...
Network outage for one of the heavy crunchers possibly, or the main server down???
6) Louie changed the check in time from 8 days to 30 days in January 2004. No tests expired so no tests were re-assigned.
?) December 2002, the check in time was lowered from 21? days to 3 days and then increased to 8 days.
5) Residue recovery, recheck, population of the queues with tests that had only been done by other researchers.
3) Runaway client grabbed a bunch of tests.
4) Garbage used to assign the same number repeatedly Some other "user" for testing could also have the same characteristic
2) Garbage after server assignment queues changed.
7) The 100 double checks just seeded into the main queue
1) Tests reassigned after failure to check in Next tier when it happens again spread because sometimes their is a liitle activity (couple of check ins) before it is dropped and fails to check in.
Joe O
Just to make it more interesting: here's the huge, color-coded version.
Just out of interest, from which queue are 'main' PRP tests being taken right now? It's not first pass, that's for sure, but neither dropped-tests nor error-fix (which seem to be the two logical alternatives) seem to be behaving as one might expect if they had become the effective 'main' queue.
And do we have any rough estimates on when all the 2.20 re-issues will all have been err.....re-issued?
For the last day or two, the general pool has been getting a pretty even split of dropped-tests and v2.2 retests. There are 208 v2.2 retests left, which should take just over one more day to finish handing out. Then it'll jump back to the usual first-pass & dropped-tests.
Is the 'n increase' for the second pass queue working correctly on this page ? Currently it shows an 'n increase' of +101032, and an 'n min' of 1276063. A little over 24 hours ago I was given a test with n = 1268914. Doesn't this give an increase of something more like +7000?
Possibly there's the same sort of problem with the error-fix queue.
It only shows +10766 right now, which is correct. Are you sure you didn't accidentally add a digit somewhere?
Nope, I did a cut and paste from the page. That's why it seemed so strange, I'd seen it at +90K, and was wondering when it was going to start to come back down.Are you sure you didn't accidentally add a digit somewhere?
It's working now, that all that matters.
Posting Permissions
Reply With Quote