Thanks alpha for the Perl script. I use it now here.
yoyo
pppla - Perl proxyper log analyser
Now new and improved with OGR-26 and RC5-72 analysis.
v0.6 - too many changes to list. I had loads of help from Darkness Productions and the RC5-72 code is based on Lani's, so thanks to you guys. I still require the stub-length or diff-length (or whatever) constraints for the stubspaces if anyone can help with that. Any problems let me know.
The formatting gets a bit broken when you post output here, but this is a rough idea of the new layout:
OGR-26 output:
RC5-72 output:Overall OGR-26 PProxy stats based on 16 days:
Smallest: 3.66 Gnodes, submitted on 06/11/2008
Largest: 338.36 Gnodes, submitted on 11/11/2008
Average: 62.80 Gnodes
Total: 58,091 Gnodes
Rate: 3,631 Gnodes/day, 0.04 Gnodes/sec, 58 stubs/day
Best days:
6,612 Gnodes, on 10/11/2008
131 stubs, on 05/11/2008
Stubs done: 925 (0.00464% of project)
Give it a try and post your output. Your feedback is welcome!Overall RC5-72 PProxy stats based on 25 days:
First block: 25/03/2007
Last block: 11/11/2008
Best day: 01/11/2008 with 864 blocks
Rate: 116 blocks/day, 5.76 Mkeys/sec
Keys done: 12,433,930,321,920
Blocks done: 2,895 (0.000000263% of project)
Thanks alpha for the Perl script. I use it now here.
yoyo
yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.
v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.
Hi alpha,
one minor Problem. I think the counting of working days ist not correct
for ogrng because there no different logfiles (like for ogrp2); all ogr26 and
ogr27 logging goes into files like "pproxyogrngYYYYMMDD". But pppla counts
all logfiles with "ogrng" in it.
For example after 2 days for my little proxy
Sure, the % of Project is also wrong until we get numbers from dnet.Code:Overall OGR-27 PProxy stats based on 121 days: Smallest: 55.80 Gnodes, submitted on 26/02/2009 Largest: 615.77 Gnodes, submitted on 25/02/2009 Average: 279.05 Gnodes Total: 17,580 Gnodes Rate: 145 Gnodes/day, 0.00 Gnodes/sec, 1 stubs/day Best days: 11,952 Gnodes, on 26/02/2009 49 stubs, on 26/02/2009 Stubs done: 63 (0.00032% of project)
Bye,
clzupp
It's because I haven't released a new version which supports OGR-27 yet. You must've modified the script to have gotten output like you pasted. pppla v0.7 only supports RC5-72 and OGR-26 so if you specified OGR-27 on the command line it would have told you the supported projects in that version and bailed. pppla v0.8 will support RC5-72, OGR-25 (added back in), OGR-26 and OGR-27.
I could easily write the script to support any future OGR project (it's practically one line of code per project), but I would need to know the total stubs for each project. Unfortunately, without this information there will have to be a new version of the script every time we start a new project. However, the script is already (mostly) future-proofed for this, so all I have to add is a couple of lines. The log filenames doesn't matter, the script looks at the stub information on each line to pick out the appropriate stubs for the project queried.
As soon as I know the stubspace size of OGR-27 I'll post the new version.
clzupp, I misread what you said initially and can now see your point. I've made the necessary changes to fix the day (log) counting and that will be in the next version, fully working across all OGR projects (past and present).
IB: it's only a ~300 line script, <10 kilobytes.
OK, dnet have updated their stubspace status page for OGR-27 so I have updated the script.
v0.8 - Added support for OGRp2-25 back in. Added support for OGR-27. Minor cosmetic tweaks.
Post any problems or oddities here. I'm on holiday for a week from tomorrow, but I'll keep an eye on things and reply if and when I can.
I edited the source to fix a problem with string and numbering also did adjust the output format.
Source has detailed stubspace stats you might edit the script in order to disable it.
good work Alpha
Lani, thanks for taking interest in the script again. Unfortunately, the stubspace stats have different constraints for each OGR project, so it isn't such a simple fix. The dnet guys aren't always very forthcoming with the required information (for instance, it wasn't posted for OGR-26 and only recently appeared for OGR-27) and it isn't that interesting anyway, so I preferred to leave it out altogether. Also, for the OGR-27 stubspaces to be numbered correctly, lines 140-143 of your modified script should be:
However, this still won't be right for OGRp2-25 or OGR-26, as I mentioned earlier.PHP Code:
case 3 { $stubspace[1]++ }
case 4 { $stubspace[2]++ }
case 5 { $stubspace[3]++ }
case 6 { $stubspace[4]++ }
What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.
I've had a new version of pppla sitting around for a while now with revamped output formatting (and an additional stat) but I was holding back on posting it until some more major changes were made. If nothing comes up in the next few weeks I'll post it anyway.
formating was ok. I edited it, because the dots weren't inline. so i modified it that all dots are in one column for each output. That's the way I like it best.What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.
anyway, it's still a great tool, in order to quickly get a good overview on the logfiles.
v0.9 - tweaked the output to try and get things aligned slightly better (thanks for the inspiration Lani). Also the stats have been beefed up a bit.
The spacing is somewhat dynamic, so if you have super large (or small) numbers it should still output nicely. I'd be interested to know how this works out for everyone.PHP Code:
Overall OGR-27 PProxy stats based on 90 active days:
Smallest: 7.37 Gnodes, submitted on 18/08/2009
Largest: 1,079.32 Gnodes, submitted on 08/09/2009
Avg. stub: 158.87 Gnodes
Avg. rate: 2,065 G/day, 0.02 Gnodes/sec, 13 stubs/day
Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009
Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009
Total: 185,884 Gnodes, 1,170 stubs (0.00019% of project)
I have also made some significant changes to the RC5-72 portion of the script, but I'll detail those in the other thread.
I think I like it
something like "what's the name of the smallest/largest stub" would be coolPHP Code:
Overall OGR-27 PProxy stats based on 2 active days:
Smallest: 3.52 Gnodes, submitted on 27/12/2009
Largest: 885.33 Gnodes, submitted on 26/12/2009
Avg. stub: 87.10 Gnodes
Avg. rate: 39,240 G/day, 0.45 Gnodes/sec, 451 stubs/day
Most Gnodes: 73,271 Gnodes, 0.85 Gnodes/sec, on 27/12/2009
Most stubs: 838 stubs, 35 stubs/hour, on 27/12/2009
Total: 78,480 Gnodes, 901 stubs (0.00015% of project)
something similar like
and I like itPHP Code:
...
Smallest: 3.52 Gnodes, submitted on 27/12/2009 --> 27/x-y-z
Largest: 885.33 Gnodes, submitted on 26/12/2009 --> 27/z-x-y
...
the-mk
Thanks for the feedback! I should be able to add the stub marks like you mentioned, although that will make the output of those two lines quite long. I'll have a think about the best way to do it.
depends on the length of the output... you could also consider something like this:
but that is just a suggestion...PHP Code:
...
Smallest OGR-27 stub:
---------------------
3.52 Gnodes
submitted on 27/12/2009
27/x-y-z
Largest OGR-27 stub:
--------------------
885.33 Gnodes
submitted on 26/12/2009
27/y-z-x
...
the-mk
How does this look?
orPHP Code:
Overall OGR-27 PProxy stats based on 95 active days:
Smallest: 7.37 Gnodes (27/1-20-7-60-3-2), submitted on 18/08/2009
Largest: 1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009
Avg. stub: 158.92 Gnodes
Avg. rate: 1,966 G/day, 0.02 Gnodes/sec, 12 stubs/day
Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009
Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009
Total: 186,726 Gnodes, 1,175 stubs (0.00019% of project)
I'm not completely convinced yet that it looks OK. I tried your suggested format but it looked a bit strange, especially with the new dynamic spacing code that I introduced in the last release.PHP Code:
Overall OGR-27 PProxy stats based on 95 active days:
Smallest: 7.37 Gnodes, submitted on 18/08/2009 --> 27/1-20-7-60-3-2
Largest: 1,079.32 Gnodes, submitted on 08/09/2009 --> 27/1-20-9-2-24-38
Avg. stub: 158.92 Gnodes
Avg. rate: 1,966 G/day, 0.02 Gnodes/sec, 12 stubs/day
Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009
Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009
Total: 186,726 Gnodes, 1,175 stubs (0.00019% of project)
I've toyed with the idea of making it an optional component, enabled by a command line flag. Maybe I'm putting too much thought into it.
I like the first one, but the second is also an option
the-mk
v0.91 - Added more stub details for smallest/largest stubs, as per the-mk's suggestion. There have also been lots of code cleanups and tweaks. Significant changes have been made to the RC5-72 code, but again, I'll go into more details in the appropriate thread.
Please post problems, feedback, or even your own output just so I can see that it works as it should for different length outputs (mostly to check that the dynamic alignment works properly).
PHP Code:
Overall OGR-27 PProxy stats based on 165 active days:
Smallest: 0.3958 Gnodes (27/1-66-16-3-4-2), submitted on 15/02/2010
Largest: 1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009
Avg. stub: 112.69 Gnodes
Avg. rate: 2,996 G/day, 0.03 Gnodes/sec, 27 stubs/day
Most Gnodes: 10,273 Gnodes, 0.12 Gnodes/sec, on 16/02/2010
Most stubs: 231 stubs, 10 stubs/hour, on 14/02/2010
Totals: 494,393 Gnodes, 4,387 stubs (0.00072% of stubspace)
output from new version 091:
I think it looks good!PHP Code:
Overall OGR-27 PProxy stats based on 109 active days:
Smallest: 0.2060 Gnodes (27/5-68-1-3-6-2), submitted on 25/04/2010
Largest: 1,230.03 Gnodes (27/4-16-5-14-17-38), submitted on 14/02/2010
Avg. stub: 77.00 Gnodes
Avg. rate: 25,622 G/day, 0.30 Gnodes/sec, 333 stubs/day
Most Gnodes: 127,960 Gnodes, 1.48 Gnodes/sec, on 01/01/2010
Most stubs: 4,553 stubs, 190 stubs/hour, on 24/04/2010
Totals: 2,792,813 Gnodes, 36,270 stubs (0.00599% of stubspace)
the-mk
It's been a while folks, are you ready for a new version bulging with features?
v0.93:
- New command line option "-d" allows you to specify a specific year, month or day that you want stats for. The format is YYYYMMDD.
- New command line option "-p" is mandatory for project selection.
- New command line option "-b" shows your 10 smallest stubs.
- New command line option "-t" shows your 10 largest stubs.
- New command line option "-u" uses dynamic units instead of Gnodes for everything.
- The "most nodes" day now shows how many stubs were completed.
- The "most stubs" day now shows how many nodes were completed.
- Code cleanups.
- Possibly some other stuff! This has been developed on and off over the last year and a half.
I believe most options should be compatible with each other. For example, it is possible to choose to see your 10 largest stubs so far for this month, using dynamic units for project OGR-27. The command line option for this would be: "-p OGR-27 -t -u -d 201111". From this you can see that to choose a month you simply omit the day number.
Please report any brokenness here, if anybody still uses this thing.
There's a bug in v0.93 which applies to the new -b and -t options. If you haven't submitted at least 10 stubs for the date specified, it will produce error messages. This was my fault due to a lack of testing, sorry!
I won't bother releasing a bug fix for this, rather, I'll try to release a new version with a couple more improvements in the not so distant future.
Pro tip: With the new "-d" option, you can get daily stats automatically sent to you by putting something like this in your crontab:
That works as-is on FreeBSD, but might need tweaking for Linux or other operating systems.Code:59 23 * * * cd /path/to/script/ ; /usr/bin/perl pppla_093.pl -p OGR-27 -d `/bin/date +\%Y\%m\%d` -u
New version coming soon with bug fix and a few tiny new bits.
v0.94:
- Fixed a bug affecting "-b" and "-t" when there were not at least 10 stubs worth of data.
- Corresponding stub marks have been added to the output of "-b" and "-t".
- Corresponding submission dates have been added to the output of "-b" and "-t".
- New command line option "-s" shows an additional stubspace breakdown for OGR projects, including contribution as a percentage of each stubspace.
Example of "-b":
Example of "-t":Code:Smallest stubs for OGR-27: #1 - 0.40 Gnodes (27/1-66-16-3-4-2) on 15/02/2010 #2 - 0.43 Gnodes (27/1-66-16-3-5-2) on 15/02/2010 #3 - 0.61 Gnodes (27/23-45-4-9-1-2) on 20/05/2011 #4 - 0.82 Gnodes (27/23-45-4-9-1-5) on 20/05/2011 #5 - 0.95 Gnodes (27/23-45-4-9-1-7) on 20/05/2011 #6 - 0.95 Gnodes (27/20-49-4-12-2-1) on 01/05/2011 #7 - 0.99 Gnodes (27/23-45-4-9-1-6) on 20/05/2011 #8 - 1.03 Gnodes (27/17-16-50-5-1-3) on 23/03/2011 #9 - 1.04 Gnodes (27/20-49-4-12-3-2) on 01/05/2011 #10 - 1.06 Gnodes (27/25-13-41-9-1-2) on 29/05/2011
Example of "-s" (with -u enabled):Code:Largest stubs for OGR-27: #1 - 1,186.69 Gnodes (27/3-18-2-11-19-41) on 15/11/2011 #2 - 1,157.12 Gnodes (27/4-3-17-12-14-44) on 06/05/2010 #3 - 1,098.91 Gnodes (27/3-20-1-14-17-39) on 13/11/2011 #4 - 1,079.32 Gnodes (27/1-20-9-2-24-38) on 08/09/2009 #5 - 1,059.07 Gnodes (27/2-3-18-4-31-36) on 25/02/2010 #6 - 1,049.72 Gnodes (27/1-20-9-3-24-37) on 14/09/2009 #7 - 1,043.85 Gnodes (27/1-20-9-2-25-37) on 09/09/2009 #8 - 1,017.24 Gnodes (27/2-3-18-4-30-37) on 25/02/2010 #9 - 1,013.64 Gnodes (27/3-12-13-14-4-48) on 12/11/2011 #10 - 1,006.35 Gnodes (27/5-6-14-4-27-38) on 17/05/2010
Feedback and script output is very welcome, and encouraged.Code:Overall OGR-27 PProxy stats based on 464 active days: Smallest: 395.76 Mnodes (27/1-66-16-3-4-2), submitted on 15/02/2010 Largest: 1.19 Tnodes (27/3-18-2-11-19-41), submitted on 15/11/2011 Avg. stub: 90.33 Gnodes Avg. rate: 2.31 T/day, 26.76 Mnodes/sec, 26 stubs/day Most nodes: 10.79 Tnodes (119 stubs), 124.87 Mnodes/sec, on 16/04/2011 Most stubs: 231 stubs (8.08 Tnodes), 10 stubs/hour, on 14/02/2010 Stubspace 2: 436 (0.24341%) Stubspace 3: 221 (0.00342%) Stubspace 4: 11,218 (0.00379%) Totals: 1.07 Pnodes, 11,875 stubs (0.00196% of stubspace)