pppla - your shiny new pproxy log analyser!

[alpha]

pppla - Perl proxyper log analyser

Now new and improved with OGR-26 and RC5-72 analysis.

v0.6 - too many changes to list. I had loads of help from Darkness Productions and the RC5-72 code is based on Lani's, so thanks to you guys. I still require the stub-length or diff-length (or whatever) constraints for the stubspaces if anyone can help with that. Any problems let me know.

The formatting gets a bit broken when you post output here, but this is a rough idea of the new layout:

OGR-26 output:

Overall OGR-26 PProxy stats based on 16 days:

Smallest: 3.66 Gnodes, submitted on 06/11/2008
Largest: 338.36 Gnodes, submitted on 11/11/2008
Average: 62.80 Gnodes
Total: 58,091 Gnodes
Rate: 3,631 Gnodes/day, 0.04 Gnodes/sec, 58 stubs/day

Best days:
6,612 Gnodes, on 10/11/2008
131 stubs, on 05/11/2008

Stubs done: 925 (0.00464% of project)

RC5-72 output:

Overall RC5-72 PProxy stats based on 25 days:

First block: 25/03/2007
Last block: 11/11/2008
Best day: 01/11/2008 with 864 blocks
Rate: 116 blocks/day, 5.76 Mkeys/sec

Keys done: 12,433,930,321,920
Blocks done: 2,895 (0.000000263% of project)

Give it a try and post your output. Your feedback is welcome!

[yoyo]

Thanks alpha for the Perl script. I use it now here (http://www.rechenkraft.net/yoyo/download/download/stats/).
yoyo

[alpha]

yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.

[yoyo]

yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.

Ok, I'll wait for the update ;)
thx
yoyo

[alpha]

v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.

[yoyo]

Thanks, I updated my server with your new script.
yoyo

[clzupp]

v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.

Hi alpha,

one minor Problem. I think the counting of working days ist not correct
for ogrng because there no different logfiles (like for ogrp2); all ogr26 and
ogr27 logging goes into files like "pproxyogrngYYYYMMDD". But pppla counts
all logfiles with "ogrng" in it.

For example after 2 days for my little proxy


Overall OGR-27 PProxy stats based on 121 days:

Smallest: 55.80 Gnodes, submitted on 26/02/2009
Largest: 615.77 Gnodes, submitted on 25/02/2009
Average: 279.05 Gnodes
Total: 17,580 Gnodes
Rate: 145 Gnodes/day, 0.00 Gnodes/sec, 1 stubs/day

Best days: 11,952 Gnodes, on 26/02/2009
49 stubs, on 26/02/2009

Stubs done: 63 (0.00032% of project)

Sure, the % of Project is also wrong until we get numbers from dnet.

Bye,
clzupp

[alpha]

It's because I haven't released a new version which supports OGR-27 yet. You must've modified the script to have gotten output like you pasted. pppla v0.7 only supports RC5-72 and OGR-26 so if you specified OGR-27 on the command line it would have told you the supported projects in that version and bailed. pppla v0.8 will support RC5-72, OGR-25 (added back in), OGR-26 and OGR-27.

I could easily write the script to support any future OGR project (it's practically one line of code per project), but I would need to know the total stubs for each project. Unfortunately, without this information there will have to be a new version of the script every time we start a new project. However, the script is already (mostly) future-proofed for this, so all I have to add is a couple of lines. The log filenames doesn't matter, the script looks at the stub information on each line to pick out the appropriate stubs for the project queried.

As soon as I know the stubspace size of OGR-27 I'll post the new version. :)

[IronBits]

You should submit this work at sourceforge :D

[alpha]

clzupp, I misread what you said initially and can now see your point. I've made the necessary changes to fix the day (log) counting and that will be in the next version, fully working across all OGR projects (past and present).

IB: it's only a ~300 line script, <10 kilobytes. :)

[alpha]

OK, dnet have updated their stubspace status page for OGR-27 so I have updated the script.

v0.8 - Added support for OGRp2-25 back in. Added support for OGR-27. Minor cosmetic tweaks.

Post any problems or oddities here. I'm on holiday for a week from tomorrow, but I'll keep an eye on things and reply if and when I can.

[yoyo]

Thanks for the script, alpha.
yoyo

[Lani]

I edited the source to fix a problem with string and numbering also did adjust the output format.

Source has detailed stubspace stats you might edit the script in order to disable it.

good work Alpha :thumbs:

[alpha]

I edited the source to fix a problem with string and numbering also did adjust the output format.

Source has detailed stubspace stats you might edit the script in order to disable it.

good work Alpha :thumbs:

Lani, thanks for taking interest in the script again. Unfortunately, the stubspace stats have different constraints for each OGR project, so it isn't such a simple fix. The dnet guys aren't always very forthcoming with the required information (for instance, it wasn't posted for OGR-26 and only recently appeared for OGR-27) and it isn't that interesting anyway, so I preferred to leave it out altogether. Also, for the OGR-27 stubspaces to be numbered correctly, lines 140-143 of your modified script should be:

case 3 { $stubspace[1]++ }
case 4 { $stubspace[2]++ }
case 5 { $stubspace[3]++ }
case 6 { $stubspace[4]++ }However, this still won't be right for OGRp2-25 or OGR-26, as I mentioned earlier.

What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.

I've had a new version of pppla sitting around for a while now with revamped output formatting (and an additional stat) but I was holding back on posting it until some more major changes were made. If nothing comes up in the next few weeks I'll post it anyway.

[Lani]

What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.

formating was ok. I edited it, because the dots weren't inline. so i modified it that all dots are in one column for each output. That's the way I like it best. :rolleyes:

anyway, it's still a great tool, in order to quickly get a good overview on the logfiles. :thumbs:

[alpha]

v0.9 - tweaked the output to try and get things aligned slightly better (thanks for the inspiration Lani). Also the stats have been beefed up a bit.


Overall OGR-27 PProxy stats based on 90 active days:

Smallest: 7.37 Gnodes, submitted on 18/08/2009
Largest: 1,079.32 Gnodes, submitted on 08/09/2009
Avg. stub: 158.87 Gnodes
Avg. rate: 2,065 G/day, 0.02 Gnodes/sec, 13 stubs/day
Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009
Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009

Total: 185,884 Gnodes, 1,170 stubs (0.00019% of project)The spacing is somewhat dynamic, so if you have super large (or small) numbers it should still output nicely. I'd be interested to know how this works out for everyone.

I have also made some significant changes to the RC5-72 portion of the script, but I'll detail those in the other thread.

[the-mk]

I think I like it ;)

Overall OGR-27 PProxy stats based on 2 active days:

Smallest: 3.52 Gnodes, submitted on 27/12/2009
Largest: 885.33 Gnodes, submitted on 26/12/2009
Avg. stub: 87.10 Gnodes
Avg. rate: 39,240 G/day, 0.45 Gnodes/sec, 451 stubs/day
Most Gnodes: 73,271 Gnodes, 0.85 Gnodes/sec, on 27/12/2009
Most stubs: 838 stubs, 35 stubs/hour, on 27/12/2009

Total: 78,480 Gnodes, 901 stubs (0.00015% of project)
something like "what's the name of the smallest/largest stub" would be cool
something similar like

...
Smallest: 3.52 Gnodes, submitted on 27/12/2009 --> 27/x-y-z
Largest: 885.33 Gnodes, submitted on 26/12/2009 --> 27/z-x-y
...

and I like it :thumbs: ;)

[alpha]

Thanks for the feedback! :) I should be able to add the stub marks like you mentioned, although that will make the output of those two lines quite long. I'll have a think about the best way to do it.

[the-mk]

depends on the length of the output... you could also consider something like this:

...
Smallest OGR-27 stub:
---------------------
3.52 Gnodes
submitted on 27/12/2009
27/x-y-z

Largest OGR-27 stub:
--------------------
885.33 Gnodes
submitted on 26/12/2009
27/y-z-x
...
but that is just a suggestion...

[alpha]

How does this look?



Overall OGR-27 PProxy stats based on 95 active days:

Smallest: 7.37 Gnodes (27/1-20-7-60-3-2), submitted on 18/08/2009
Largest: 1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009
Avg. stub: 158.92 Gnodes
Avg. rate: 1,966 G/day, 0.02 Gnodes/sec, 12 stubs/day
Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009
Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009

Total: 186,726 Gnodes, 1,175 stubs (0.00019% of project)


or



Overall OGR-27 PProxy stats based on 95 active days:

Smallest: 7.37 Gnodes, submitted on 18/08/2009 --> 27/1-20-7-60-3-2
Largest: 1,079.32 Gnodes, submitted on 08/09/2009 --> 27/1-20-9-2-24-38
Avg. stub: 158.92 Gnodes
Avg. rate: 1,966 G/day, 0.02 Gnodes/sec, 12 stubs/day
Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009
Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009

Total: 186,726 Gnodes, 1,175 stubs (0.00019% of project)
I'm not completely convinced yet that it looks OK. I tried your suggested format but it looked a bit strange, especially with the new dynamic spacing code that I introduced in the last release.

I've toyed with the idea of making it an optional component, enabled by a command line flag. Maybe I'm putting too much thought into it. :)

[the-mk]

I like the first one, but the second is also an option ;)
:thumbs:

[alpha]

v0.91 - Added more stub details for smallest/largest stubs, as per the-mk's suggestion. There have also been lots of code cleanups and tweaks. Significant changes have been made to the RC5-72 code, but again, I'll go into more details in the appropriate thread.

Please post problems, feedback, or even your own output just so I can see that it works as it should for different length outputs (mostly to check that the dynamic alignment works properly).


Overall OGR-27 PProxy stats based on 165 active days:

Smallest: 0.3958 Gnodes (27/1-66-16-3-4-2), submitted on 15/02/2010
Largest: 1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009
Avg. stub: 112.69 Gnodes
Avg. rate: 2,996 G/day, 0.03 Gnodes/sec, 27 stubs/day
Most Gnodes: 10,273 Gnodes, 0.12 Gnodes/sec, on 16/02/2010
Most stubs: 231 stubs, 10 stubs/hour, on 14/02/2010

Totals: 494,393 Gnodes, 4,387 stubs (0.00072% of stubspace)

[the-mk]

output from new version 091:

Overall OGR-27 PProxy stats based on 109 active days:

Smallest: 0.2060 Gnodes (27/5-68-1-3-6-2), submitted on 25/04/2010
Largest: 1,230.03 Gnodes (27/4-16-5-14-17-38), submitted on 14/02/2010
Avg. stub: 77.00 Gnodes
Avg. rate: 25,622 G/day, 0.30 Gnodes/sec, 333 stubs/day
Most Gnodes: 127,960 Gnodes, 1.48 Gnodes/sec, on 01/01/2010
Most stubs: 4,553 stubs, 190 stubs/hour, on 24/04/2010

Totals: 2,792,813 Gnodes, 36,270 stubs (0.00599% of stubspace)
I think it looks good!

[alpha]

Thanks! :thumbs:

[alpha]

It's been a while folks, are you ready for a new version bulging with features?

v0.93:


New command line option "-d" allows you to specify a specific year, month or day that you want stats for. The format is YYYYMMDD.
New command line option "-p" is mandatory for project selection.
New command line option "-b" shows your 10 smallest stubs.
New command line option "-t" shows your 10 largest stubs.
New command line option "-u" uses dynamic units instead of Gnodes for everything.
The "most nodes" day now shows how many stubs were completed.
The "most stubs" day now shows how many nodes were completed.
Code cleanups.
Possibly some other stuff! This has been developed on and off over the last year and a half.


I believe most options should be compatible with each other. For example, it is possible to choose to see your 10 largest stubs so far for this month, using dynamic units for project OGR-27. The command line option for this would be: "-p OGR-27 -t -u -d 201111". From this you can see that to choose a month you simply omit the day number.

Please report any brokenness here, if anybody still uses this thing.

[alpha]

There's a bug in v0.93 which applies to the new -b and -t options. If you haven't submitted at least 10 stubs for the date specified, it will produce error messages. This was my fault due to a lack of testing, sorry! :)

I won't bother releasing a bug fix for this, rather, I'll try to release a new version with a couple more improvements in the not so distant future.

[alpha]

Pro tip: With the new "-d" option, you can get daily stats automatically sent to you by putting something like this in your crontab:


59 23 * * * cd /path/to/script/ ; /usr/bin/perl pppla_093.pl -p OGR-27 -d `/bin/date +\%Y\%m\%d` -u

That works as-is on FreeBSD, but might need tweaking for Linux or other operating systems.

New version coming soon with bug fix and a few tiny new bits.

[alpha]

v0.94:


Fixed a bug affecting "-b" and "-t" when there were not at least 10 stubs worth of data.
Corresponding stub marks have been added to the output of "-b" and "-t".
Corresponding submission dates have been added to the output of "-b" and "-t".
New command line option "-s" shows an additional stubspace breakdown for OGR projects, including contribution as a percentage of each stubspace.


Example of "-b":



Smallest stubs for OGR-27:

#1 - 0.40 Gnodes (27/1-66-16-3-4-2) on 15/02/2010
#2 - 0.43 Gnodes (27/1-66-16-3-5-2) on 15/02/2010
#3 - 0.61 Gnodes (27/23-45-4-9-1-2) on 20/05/2011
#4 - 0.82 Gnodes (27/23-45-4-9-1-5) on 20/05/2011
#5 - 0.95 Gnodes (27/23-45-4-9-1-7) on 20/05/2011
#6 - 0.95 Gnodes (27/20-49-4-12-2-1) on 01/05/2011
#7 - 0.99 Gnodes (27/23-45-4-9-1-6) on 20/05/2011
#8 - 1.03 Gnodes (27/17-16-50-5-1-3) on 23/03/2011
#9 - 1.04 Gnodes (27/20-49-4-12-3-2) on 01/05/2011
#10 - 1.06 Gnodes (27/25-13-41-9-1-2) on 29/05/2011


Example of "-t":



Largest stubs for OGR-27:

#1 - 1,186.69 Gnodes (27/3-18-2-11-19-41) on 15/11/2011
#2 - 1,157.12 Gnodes (27/4-3-17-12-14-44) on 06/05/2010
#3 - 1,098.91 Gnodes (27/3-20-1-14-17-39) on 13/11/2011
#4 - 1,079.32 Gnodes (27/1-20-9-2-24-38) on 08/09/2009
#5 - 1,059.07 Gnodes (27/2-3-18-4-31-36) on 25/02/2010
#6 - 1,049.72 Gnodes (27/1-20-9-3-24-37) on 14/09/2009
#7 - 1,043.85 Gnodes (27/1-20-9-2-25-37) on 09/09/2009
#8 - 1,017.24 Gnodes (27/2-3-18-4-30-37) on 25/02/2010
#9 - 1,013.64 Gnodes (27/3-12-13-14-4-48) on 12/11/2011
#10 - 1,006.35 Gnodes (27/5-6-14-4-27-38) on 17/05/2010


Example of "-s" (with -u enabled):



Overall OGR-27 PProxy stats based on 464 active days:

Smallest: 395.76 Mnodes (27/1-66-16-3-4-2), submitted on 15/02/2010
Largest: 1.19 Tnodes (27/3-18-2-11-19-41), submitted on 15/11/2011
Avg. stub: 90.33 Gnodes
Avg. rate: 2.31 T/day, 26.76 Mnodes/sec, 26 stubs/day
Most nodes: 10.79 Tnodes (119 stubs), 124.87 Mnodes/sec, on 16/04/2011
Most stubs: 231 stubs (8.08 Tnodes), 10 stubs/hour, on 14/02/2010

Stubspace 2: 436 (0.24341%)
Stubspace 3: 221 (0.00342%)
Stubspace 4: 11,218 (0.00379%)

Totals: 1.07 Pnodes, 11,875 stubs (0.00196% of stubspace)


Feedback and script output is very welcome, and encouraged. :)