Percent Complete (Phase 2)

[wirthi]

Hi, anybody recognized that the "Percent Complete (Phase 2)" on the OGR-25 stats page dropped from 44% to 28% ? Perhaps my brain is cheating on my but I'm sure that's what happened a few days ago. Can anybody confirm this or tell me why that happened?

[Guilherme]

You're right, the status was almost 44% some days ago.

[alpha]

Seems to be fixed.

[the-mk]

currently at ~45.85%

[Guilherme]

I am receiving very short WUs lately, like these:

[Apr 08 18:48:50 UTC] OGR-P2 #a: Completed 25/44-29-10-2-1-6 (0.03 stats units)
0.00:00:02.18 - [12,561,489 nodes/s]
[Apr 08 18:48:50 UTC] OGR-P2 #a: 25/44-29-10-2-1-6 [27,471,977 nodes]

[Apr 08 18:49:21 UTC] OGR-P2 #b: Completed 25/44-29-11-2-1-6 (0.03 stats units)
0.00:00:03.09 - [11,240,717 nodes/s]
[Apr 08 18:49:21 UTC] OGR-P2 #b: 25/44-29-11-2-1-6 [34,778,780 nodes]


I hope this indicates that the project will end soon.

[alpha]

Nope, unfortunately we're not even half way through phase 2 yet.

Stub size generally indicates nothing with regards to project progress. The dnet team don't know how long a stub will take to crunch when it is distributed to participants - this is discovered during computation.

We actually had a thread here a while back (http://www.free-dc.org/forum/showthread.php?t=2602) comparing smallest stub sizes. That is a small one you've found, though! :)

[Guilherme]

First time I get these messages:

[Apr 08 20:57:52 UTC] The keyserver says: have run out of OGR-P2 work.
[Apr 08 20:57:52 UTC] NetUpdate::keyserver does not have packets available.
[Apr 08 20:57:53 UTC] NetUpdate::keyserver does not have packets available.

[Apr 08 20:58:24 UTC] NetUpdate::keyserver does not have packets available.
[Apr 08 20:58:24 UTC] OGR-P2: Sent 1 packet (0.07 stats units) to server.
[Apr 08 21:13:55 UTC] NetUpdate::keyserver does not have packets available.
[Apr 08 21:13:55 UTC] NetUpdate::keyserver does not have packets available.
[Apr 08 21:15:37 UTC] *Break* Shutting down...
[Apr 08 21:15:38 UTC] Shutdown complete.

Now, two hours later, I am receiving WUs again.

[alpha]

Yeah, sometimes a keyserver can dry up, but there are loads of them (keyservers) so it should grab work from a different one eventually.

Alternatively, you could use "stats.free-dc.org" as your keyserver (which is highly unlikely to dry up), then you can track your stats here (http://stats.open-dc.org/dnet/ogr/byemail.html) throughout the day! :thumbs:

[wirthi]

During the last days the servers were handing out really small stubs. My average dropped to less than 3 Gnodes per stub. Since it is the same work for the server no matter if the stub is small or huge, right now he has to do a lot of work since each client is requesting stub after stub.

[wirthi]

The Project should reach 50% today!

[the-mk]

Maybe I'm not that optimistic :D

Do you really think we'll be >= 50% by next stats update?

We will see...

:cheers:

[wirthi]

The stats say: Percent Complete (Phase 2): ~49.94%

Well, I guess I was too optimistic without actually calculating it. Perhaps we only reach it a day or two later.

Edit: we're doing about 0.02 % every day. So it should take 3-4 days.

[TeeJay]

So, rough gestimate... It will take 2,503 days to complete this project ?
Like a little more than 7 years ? :looney:




The stats say: Percent Complete (Phase 2): ~49.94%

Well, I guess I was too optimistic without actually calculating it. Perhaps we only reach it a day or two later.

Edit: we're doing about 0.02 % every day. So it should take 3-4 days.

[Guilherme]

Percent Complete (Phase 2): ~50.05%

We're doing more than 0.02% lately. Some days we're doing 0.06%, others days 0.14%, 0.08%, etc.

[alpha]

Let's talk real numbers.

On 16th March 2007, the-mk posted that we were at 45.85%.
Today, 14th April 2007 we are at 50.05%.

That spans 29 days in which 4.2% was completed.
That amounts to 0.14% per day. At that rate, it will take another 357 days to reach 100%.

However, as we all know, there are several reasons why it probably won't take that long.

BTW TeeJay, if the (way off) 2503 day estimate scared you, what do you make of RC5-72? :lmao:

[the-mk]

On 16th March 2007, the-mk posted that we were at 45.85%.
Today, 14th April 2007 we are at 50.05%.
That's the reason why I was not optimistic :D

That spans 29 days in which 4.2% was completed.
That amounts to 0.14% per day.
I didn't calculate this number...
--> it was high enough to break the 50% mark :D

:cheers:

[wirthi]

Then there is some glitch in the Data at http://stats.distributed.net/projects.php?project_id=25 and I guess I already know where.

What I did:

Gnodes we tested yesterday: 14.000.000.
Overall testes GNodes: 37.000.000.000
Since we are roughly 50% through the nodespace, there must be 74.000.000.000 nodes overall.

14.000.000 / 74.000.000.000 * 100 % = 0.019 % ~= 0.02 %

The glitch will be, that we are 50 % through phase 2, but that the "overall tested nodes" and the "time working" will be counted for the overall project, not for phase two. Then, what I'm doing is comparing apples with peaches. Then we're doing more percent of phase two every day, what is obviously (and luckily) the case.

So, we should finish this project much faster.

[the-mk]

I never liked complicated maths :D

[alpha]

Since we are roughly 50% through the nodespace, there must be 74.000.000.000 nodes overall.
This isn't right! We don't know how many nodes there will be. Compare the front page for the OGR-25 stats with the RC5-72 stats. RC5-72 has a fixed keyspace, so they show statistics about how long the project will take to exhaust the keyspace at the current rate (today it's 350,052 days). OGR-25 stats don't have these details because there is no known "keyspace". Stubs are of an unknown length, so it is impossible to determine how many nodes each stub consists of or how long it will take to crunch.

As far as I was aware, the % completion measures the number of tested stubs. That would mean we have now tested just over half of the total number of phase 2 stubs. Quoted from the FAQ (http://faq.distributed.net/cache/230.html):


The higher-diff stubs overlap with the smaller-diff stubs from later sub-phases, so there is less work (fewer average nodes per stub) left for the later sub-phases. This makes the overall project percentage for Phase 2 (which is measured in stubs not nodes) seem to progress more slowly in the early subphases, than the later subphases.

[wirthi]

Well, yes, I know that. But, we have already done phase 1. We learned how many nodes there were during phase one. The number of stubs if known and I guess it is the same as in phase one. There is probably a slight difference in nodes between phase 1 and 2 (since bugs were discovered in the phase-1-code IIRC) but that shouldn't change all too much. I had assumed that knowing how many nodes there were in phase 1 gives a *very good* approximation of how many nodes there are in phase 2.

On the other hand, if we really are faster near the end of the project, we should finish even earlier than in one year's time.

[alpha]

I was unsure about some of this so I had a chat with TheJet and he cleared it up for me.

The stubs in Phase 1 have nothing to do with the stubs in Phase 2. They are not the same work, and so we do NOT know how many nodes need to be crunched in Phase 2. Phase 2 percentage completion is calculated based on the number of stubs, since we know how many stubs there are.

OGR-25 Phase 2 is broken down into 9 subphases including:
221,636,328 6-diff stubs
76,468,258 5-diff stubs
6,515,212 4-diff stubs
237,944 3-diff stubs
304,857,742 stubs TOTALAlso, remember that each stub has to be crunched by "at least" two different computers (an effective minimum of 609,715,484 stubs).

Higher-diff stubs take less work to complete, but there are clearly lots more of them.

[the-mk]

Today's Percent Complete (Phase 2): ~53.01%

[Guilherme]

The shortest workunit I've ever seen:

[Aug 08 19:25:48 UTC] OGR-P2 #b: Completed 25/55-20-6-3-1-7 (0.01 stats units)
0.00:00:01.01 - [13,903,553 nodes/s]
[Aug 08 19:25:48 UTC] OGR-P2 #b: 25/55-20-6-3-1-7 [14,126,010 nodes]

[the-mk]

Indeed the shortest one :D

Today I got some "shorties" too, but I didn't make it below 1.00 stats units...

[wirthi]

I've been monitoring the "percentage complete" value for some time now. After some quick calculations: the project will likely be completed in slightly over 200 days.

Both when calculating with 97-day-old data (53,01% on May 13th) and 10-day-old-data (65,96% on August 8th) and using the current value (67,78% on August 18th) the project will be finished at roughly 212 days from now. I register a slightly increasing speed though (could be more crunchers, could be that the project really gets faster near the end to decreasing stub sizes). Yesterday it was 213 days, a week ago 220 days (based on the 97-day-data).

So, my current guess would be that OGR-25 is finished by the end of February 2008.

[alpha]

I don't think you can really make much of an accurate prediction because the end of the project will be unpredictable with the super long stubs that we haven't even started.

The dnet team could probably give us a good estimate. Last time I spoke to someone there, they were convinced it would end this year.

[wirthi]

You'r right of course. We just don't know how dramatic this increase in speed will be. I guess the officials know.

Yesterday we did a fantastic 0.31% of the whole stubspace - while doing an below-average of 166 Gnodes/sec (avg: 175). At yesterday's speed, the project will be finished in 100 days (end of November 2007).

[Guilherme]

The shortest workunit I've ever seen:

[Aug 08 19:25:48 UTC] OGR-P2 #b: Completed 25/55-20-6-3-1-7 (0.01 stats units)
0.00:00:01.01 - [13,903,553 nodes/s]
[Aug 08 19:25:48 UTC] OGR-P2 #b: 25/55-20-6-3-1-7 [14,126,010 nodes]

A WU even shorter:

[Aug 22 19:48:15 UTC] OGR-P2 #b: Completed 25/36-44-6-5-2-1 (0.01 stats units)
0.00:00:00.64 - [19,749,881 nodes/s]
[Aug 22 19:48:15 UTC] OGR-P2 #b: 25/36-44-6-5-2-1 [12,659,674 nodes]

[wirthi]

Yesterday we did an amazing 0.67% of the stubspace. At this speed we'r down to 46 days. 15-days average is at 146 days (mid-january 2008), further dropping down every day.

[the-mk]

Do you have an excel sheet for calculating this infos or did you write a program? :D
:cheers:

[wirthi]

An Excel Sheet is sufficient :-)

[the-mk]

we didn't do much of the stubspace, did we?

[wirthi]

Yes, only big stubs yesterday. We only did 0.04%. The now 19-day-average is at 150 days (January 23.).

[the-mk]

WinXP P4 client
[Aug 26 12:36:00 UTC] OGR-P2: 25/5-13-8-6-30 [698,856,989,983 nodes]

Win XP AMD X2 4800+ client
[Aug 25 23:05:59 UTC] OGR-P2 #a: 25/5-10-3-16-35 [642,047,096,169 nodes]
got a bigger one here!!
[Aug 26 16:13:23 UTC] OGR-P2 #a: 25/5-10-3-19-24 [721,883,405,088 nodes]


Win XP AMD X2 3800+ client
[Aug 26 13:59:02 UTC] OGR-P2 #b: 25/5-17-6-38-21 [306,563,831,628 nodes]

Linux P4 client
[Aug 26 14:22:52 UTC] OGR-P2: 25/5-15-27-12-14 [299,657,595,860 nodes]

[wirthi]

Things are slowing down a bit. Yesterday we did only 0.03%. The 21-day-average is down to 0,183 % per day, giving a projected completion date of February 7th. 106-days-average is at 0,158, predicting March 4th.

To complete the project this year is still in reach as the stub-sizes will continue to drop again, at least that's what we were told. But guys, why did we drop to #4 in yesterday's team stats :bang: That's not the way to speed up things :)

[alpha]

To complete the project this year is still in reach as the stub-sizes will continue to drop again, at least that's what we were told.
Stub size will increase towards project end. We still have not started the 6,515,212 4-diff or 237,944 3-diff stubs and these are going to be the biggies.

A 21 day average is pretty useless for predicting a completion date because of the spikes caused as new stub lengths are introduced. We are not "slowing down", it's just that we must be crunching mostly longer stubs in comparison with previous days.

[wirthi]

Well, but if stub sizes increase, there is no chance that we will finish the project this year as you stated before. As I said, my 100-day-prediction is well into next year, and if the stub sizes increase we slow down additionally.

I remember having read somewhere that we will improve our speed near the end of the project. This won't be the case if stub sizes increase. I'm cowfused ...

[alpha]

Project progress will get quicker closer to the end of the project. That is because the longer stubs that I previously mentioned are few in number.

6,515,212 4-diff
237,944 3-diff
--
6,753,156 stubs total

That is only 2.2% of the total stubcount. So, whilst we still have some massive stubs to crunch through, relatively it isn't that many.

[the-mk]

Why don't we make a "when-does-OGR-end-prediction" website?
Collect the difference of completed stubs everyday and compute the possible end of it?

[paleseptember]

Why don't we make a "when-does-OGR-end-prediction" website?
Collect the difference of completed stubs everyday and compute the possible end of it?

:clap:

Great idea (though I think you might have been being sarcastic).

I'm trying to work out how much Gnodes each day I'll have to do to break the 3000th place by the project end. Alternately, assuming a reasonable rate of progress, just how far I can climb the ladder.

Competition, eh? :thumbs:

[the-mk]

Well, with the "mental" help of wirthi (he provided the Excel sheet and the formulas) I've made a little "possible end of OGR25 phase 2" page, which can be found at http://the-mk.dyndns.org/dnet/ :D

The dates might be a little different from what wirthi has in his excel sheet, that's because he is longer collecting data since I did, but he didn't collect them every day while my script does. I wanted "complete" data without gaps, so I didn't use all his numbers collected...

I hope you enjoy that page :cheers:

[wirthi]

Yea, your data is gives a far too high end-date. Mine is somewhere end of April 2008, which might be too late as well.

[the-mk]

As we say, it is only predicted (if we would have the complete-% from much earlier dates, it would look much more different I think...). And I see that complete data makes not much sense... so I've added all dates from your sheet except the one from May...

[the-mk]

I'm getting a lot of small stubs at the moment... seems to be sooner end date tomorrow :D

[wirthi]

We've taken up more speed again but are still doing only about 0.12% a day. My current prediction is still 27th April. Hope the smaller stubs are coming soon, I want this project to finish this year :-)

[the-mk]

didn't you say that there will be larger stubs again? I'm against finishing this year, but it would be ok finishing after winter has ended :D so I can warm up my rooms with DC power :cheers:

[wirthi]

Yes, there will be some *huge* stubs, but in general the trend should continue towards smaller stubs (from what I have read; I could be wrong).

[the-mk]

We did more than 1% of the stubspace since you last posted :cheers:

[the-mk]

That last few days my "days-of-finishing" increased :(
bigger stubs are sent out again... hopefully they finish soon :D

[yoyo]

Saw also that the wus now getting bigger. Is there any rule when ogr's are bigger and when smaller?
yoyo

[wirthi]

You can mathematically explain why they get bigger and smaller but you never know that stubs the server sends out next. There is some information about the open and finished stub-spaces available on the distributed.net homepage somewhere but that isn't really accurate either.

[wirthi]

Update on my prediction: after gathering 170 days of data now, the average speed is now at 0.122 percent of the project's workspace completed every day. This figure has a still a slight downward trend though.

I would currently predict the date of 100% now at June 1st, 2008. Still hoping for significantly smaller packets near the end of the project, but to complete the project in 2007 as predicted earlier seems quite impossible.

[alpha]

A different way of doing this would be to use your data (% per day) to calculate how many stubs are being completed daily. This is quite significant, because we know how many stubs are left, and you can account for stub length adjustments (like when we started 5-diff stubs, project-wide %/day probably dropped slightly). Using that information, you can predict how much our %/day will drop when 4-diff and 3-diff stubs are introduced and use that figure to calculate an ETA.

Remember, the sum of 4-diff and 3-diff stubs is just under 6.8 million. There were over 76 million 5-diff and well over 221 million 6-diff stubs. We actually don't have that much further to go.

Edit: we shouldn't be getting short stubs near the end, unless the team find a bunch of untested ones, or ones lacking second verification, or ones left over from 6-diff or 5-diff stubspace.

[Guilherme]

2008-01-29 Percent Complete (Phase 2): 79.71%
2008-01-30 Percent Complete (Phase 2): ~80.29%

0.58% in one day.

[the-mk]

Yeah, that's a lot, but my stubs are currently around 20-50 GNodes "big"...

but the date to completion went down fast to the end of September!

[IronBits]

So... pretty much for sure this year then... :D
3rd quarter maybe?

[the-mk]

as Wirthi says, OGR is a little bit unpredictable, since you don't know how many GNodes a stub takes...

[wirthi]

Yes, 3rd quarter should be accomplishable. I currently predict August 8th.

[the-mk]

we completed about 0.49% yesterday!
currently stopping at 81.05%
:cheers:

[wirthi]

Yes, but we did 965 Gnodes/sec yesterday. That is much, much more than usual (something like 250 would be average I guess). That is still the dump of [DPC] NoizyCows. Or this guy captured a data center on the weekend.

[the-mk]

OK, didn't have DPC's dump in mind... but capturing a datacenter on the weekend? That would be a very huge one! Anybody knows where NoizyCows is working?

[[NGS]Cpt00Kirk]

NoizyCows is a sub team of DPC and 1 of the guys in that team has found a big bump in power...

after the 155.000.000 dump 2 days ago, this will be our daily! :rock:

From this day forward untill the end he will do 5.700.000 - 6.500.000 A DAY!!! :scared::scared:

:thumbs:

so be quick with the end-of-OGR page Bok.... we calculated that round june its over and out with OGR!