pppla - your shiny new pproxy log analyser!

Printable View

11-11-2008, 03:19 PM
alpha

1 Attachment(s)

pppla - your shiny new pproxy log analyser!

pppla - Perl proxyper log analyser

Now new and improved with OGR-26 and RC5-72 analysis.

v0.6 - too many changes to list. I had loads of help from Darkness Productions and the RC5-72 code is based on Lani's, so thanks to you guys. I still require the stub-length or diff-length (or whatever) constraints for the stubspaces if anyone can help with that. Any problems let me know.

The formatting gets a bit broken when you post output here, but this is a rough idea of the new layout:

OGR-26 output:

Quote:

Overall OGR-26 PProxy stats based on 16 days:

Smallest: 3.66 Gnodes, submitted on 06/11/2008
Largest: 338.36 Gnodes, submitted on 11/11/2008
Average: 62.80 Gnodes
Total: 58,091 Gnodes
Rate: 3,631 Gnodes/day, 0.04 Gnodes/sec, 58 stubs/day

Best days:
6,612 Gnodes, on 10/11/2008
131 stubs, on 05/11/2008

Stubs done: 925 (0.00464% of project)

RC5-72 output:

Quote:

Overall RC5-72 PProxy stats based on 25 days:

First block: 25/03/2007
Last block: 11/11/2008
Best day: 01/11/2008 with 864 blocks
Rate: 116 blocks/day, 5.76 Mkeys/sec

Keys done: 12,433,930,321,920
Blocks done: 2,895 (0.000000263% of project)

Give it a try and post your output. Your feedback is welcome!
11-12-2008, 01:12 PM
yoyo

Thanks alpha for the Perl script. I use it now here.
yoyo
11-21-2008, 09:41 AM
alpha

yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.
11-21-2008, 01:34 PM
yoyo

Quote:

Originally Posted by alpha

yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.

Ok, I'll wait for the update ;)
thx
yoyo
12-02-2008, 07:31 AM
alpha

1 Attachment(s)

v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.
12-02-2008, 04:28 PM
yoyo

Thanks, I updated my server with your new script.
yoyo
02-26-2009, 05:08 AM
clzupp

Quote:

Originally Posted by alpha

v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.

Hi alpha,

one minor Problem. I think the counting of working days ist not correct
for ogrng because there no different logfiles (like for ogrp2); all ogr26 and
ogr27 logging goes into files like "pproxyogrngYYYYMMDD". But pppla counts
all logfiles with "ogrng" in it.

For example after 2 days for my little proxy

Code:

Overall OGR-27 PProxy stats based on 121 days: Smallest: 55.80 Gnodes, submitted on 26/02/2009 Largest: 615.77 Gnodes, submitted on 25/02/2009 Average: 279.05 Gnodes Total: 17,580 Gnodes Rate: 145 Gnodes/day, 0.00 Gnodes/sec, 1 stubs/day Best days: 11,952 Gnodes, on 26/02/2009 49 stubs, on 26/02/2009 Stubs done: 63 (0.00032% of project)

Sure, the % of Project is also wrong until we get numbers from dnet.

Bye,
clzupp
02-26-2009, 06:22 AM
alpha

It's because I haven't released a new version which supports OGR-27 yet. You must've modified the script to have gotten output like you pasted. pppla v0.7 only supports RC5-72 and OGR-26 so if you specified OGR-27 on the command line it would have told you the supported projects in that version and bailed. pppla v0.8 will support RC5-72, OGR-25 (added back in), OGR-26 and OGR-27.

I could easily write the script to support any future OGR project (it's practically one line of code per project), but I would need to know the total stubs for each project. Unfortunately, without this information there will have to be a new version of the script every time we start a new project. However, the script is already (mostly) future-proofed for this, so all I have to add is a couple of lines. The log filenames doesn't matter, the script looks at the stub information on each line to pick out the appropriate stubs for the project queried.

As soon as I know the stubspace size of OGR-27 I'll post the new version. :)
02-26-2009, 06:37 AM
IronBits

You should submit this work at sourceforge :D
02-26-2009, 09:37 AM
alpha

clzupp, I misread what you said initially and can now see your point. I've made the necessary changes to fix the day (log) counting and that will be in the next version, fully working across all OGR projects (past and present).

IB: it's only a ~300 line script, <10 kilobytes. :)
02-27-2009, 02:16 AM
alpha

1 Attachment(s)

OK, dnet have updated their stubspace status page for OGR-27 so I have updated the script.

v0.8 - Added support for OGRp2-25 back in. Added support for OGR-27. Minor cosmetic tweaks.

Post any problems or oddities here. I'm on holiday for a week from tomorrow, but I'll keep an eye on things and reply if and when I can.
03-01-2009, 09:27 AM
yoyo

Thanks for the script, alpha.
yoyo
07-08-2009, 02:38 AM
Lani

1 Attachment(s)

I edited the source to fix a problem with string and numbering also did adjust the output format.

Source has detailed stubspace stats you might edit the script in order to disable it.

good work Alpha :thumbs:
07-14-2009, 03:36 AM
alpha

Quote:

Originally Posted by Lani

I edited the source to fix a problem with string and numbering also did adjust the output format.

Source has detailed stubspace stats you might edit the script in order to disable it.

good work Alpha :thumbs:

Lani, thanks for taking interest in the script again. Unfortunately, the stubspace stats have different constraints for each OGR project, so it isn't such a simple fix. The dnet guys aren't always very forthcoming with the required information (for instance, it wasn't posted for OGR-26 and only recently appeared for OGR-27) and it isn't that interesting anyway, so I preferred to leave it out altogether. Also, for the OGR-27 stubspaces to be numbered correctly, lines 140-143 of your modified script should be:

PHP Code:

case 3 { $stubspace[1]++ } case 4 { $stubspace[2]++ } case 5 { $stubspace[3]++ } case 6 { $stubspace[4]++ }

However, this still won't be right for OGRp2-25 or OGR-26, as I mentioned earlier.

What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.

I've had a new version of pppla sitting around for a while now with revamped output formatting (and an additional stat) but I was holding back on posting it until some more major changes were made. If nothing comes up in the next few weeks I'll post it anyway.
07-14-2009, 02:41 PM
Lani

Quote:

What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.

formating was ok. I edited it, because the dots weren't inline. so i modified it that all dots are in one column for each output. That's the way I like it best. :rolleyes:

anyway, it's still a great tool, in order to quickly get a good overview on the logfiles. :thumbs:
09-17-2009, 12:02 PM
alpha

1 Attachment(s)

v0.9 - tweaked the output to try and get things aligned slightly better (thanks for the inspiration Lani). Also the stats have been beefed up a bit.

PHP Code:

Overall OGR-27 PProxy stats based on 90 active days: Smallest: 7.37 Gnodes, submitted on 18/08/2009 Largest: 1,079.32 Gnodes, submitted on 08/09/2009 Avg. stub: 158.87 Gnodes Avg. rate: 2,065 G/day, 0.02 Gnodes/sec, 13 stubs/day Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009 Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009 Total: 185,884 Gnodes, 1,170 stubs (0.00019% of project)

The spacing is somewhat dynamic, so if you have super large (or small) numbers it should still output nicely. I'd be interested to know how this works out for everyone.

I have also made some significant changes to the RC5-72 portion of the script, but I'll detail those in the other thread.
12-27-2009, 04:52 PM
the-mk

I think I like it ;)

PHP Code:

Overall OGR-27 PProxy stats based on 2 active days: Smallest: 3.52 Gnodes, submitted on 27/12/2009 Largest: 885.33 Gnodes, submitted on 26/12/2009 Avg. stub: 87.10 Gnodes Avg. rate: 39,240 G/day, 0.45 Gnodes/sec, 451 stubs/day Most Gnodes: 73,271 Gnodes, 0.85 Gnodes/sec, on 27/12/2009 Most stubs: 838 stubs, 35 stubs/hour, on 27/12/2009 Total: 78,480 Gnodes, 901 stubs (0.00015% of project)

something like "what's the name of the smallest/largest stub" would be cool
something similar like

PHP Code:

... Smallest: 3.52 Gnodes, submitted on 27/12/2009 --> 27/x-y-z Largest: 885.33 Gnodes, submitted on 26/12/2009 --> 27/z-x-y ...

and I like it :thumbs: ;)
12-28-2009, 04:33 AM
alpha

Thanks for the feedback! :) I should be able to add the stub marks like you mentioned, although that will make the output of those two lines quite long. I'll have a think about the best way to do it.
12-28-2009, 05:22 AM
the-mk

depends on the length of the output... you could also consider something like this:

PHP Code:

... Smallest OGR-27 stub: --------------------- 3.52 Gnodes submitted on 27/12/2009 27/x-y-z Largest OGR-27 stub: -------------------- 885.33 Gnodes submitted on 26/12/2009 27/y-z-x ...

but that is just a suggestion...
01-19-2010, 06:50 AM
alpha

How does this look?

PHP Code:

Overall OGR-27 PProxy stats based on 95 active days: Smallest: 7.37 Gnodes (27/1-20-7-60-3-2), submitted on 18/08/2009 Largest: 1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009 Avg. stub: 158.92 Gnodes Avg. rate: 1,966 G/day, 0.02 Gnodes/sec, 12 stubs/day Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009 Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009 Total: 186,726 Gnodes, 1,175 stubs (0.00019% of project)

or

PHP Code:

Overall OGR-27 PProxy stats based on 95 active days: Smallest: 7.37 Gnodes, submitted on 18/08/2009 --> 27/1-20-7-60-3-2 Largest: 1,079.32 Gnodes, submitted on 08/09/2009 --> 27/1-20-9-2-24-38 Avg. stub: 158.92 Gnodes Avg. rate: 1,966 G/day, 0.02 Gnodes/sec, 12 stubs/day Most Gnodes: 8,102 Gnodes, 0.09 Gnodes/sec, on 20/08/2009 Most stubs: 135 stubs, 6 stubs/hour, on 18/08/2009 Total: 186,726 Gnodes, 1,175 stubs (0.00019% of project)

I'm not completely convinced yet that it looks OK. I tried your suggested format but it looked a bit strange, especially with the new dynamic spacing code that I introduced in the last release.

I've toyed with the idea of making it an optional component, enabled by a command line flag. Maybe I'm putting too much thought into it. :)
01-19-2010, 03:29 PM
the-mk

I like the first one, but the second is also an option ;)
:thumbs:
04-27-2010, 02:29 PM
alpha

1 Attachment(s)

v0.91 - Added more stub details for smallest/largest stubs, as per the-mk's suggestion. There have also been lots of code cleanups and tweaks. Significant changes have been made to the RC5-72 code, but again, I'll go into more details in the appropriate thread.

Please post problems, feedback, or even your own output just so I can see that it works as it should for different length outputs (mostly to check that the dynamic alignment works properly).

PHP Code:

Overall OGR-27 PProxy stats based on 165 active days: Smallest: 0.3958 Gnodes (27/1-66-16-3-4-2), submitted on 15/02/2010 Largest: 1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009 Avg. stub: 112.69 Gnodes Avg. rate: 2,996 G/day, 0.03 Gnodes/sec, 27 stubs/day Most Gnodes: 10,273 Gnodes, 0.12 Gnodes/sec, on 16/02/2010 Most stubs: 231 stubs, 10 stubs/hour, on 14/02/2010 Totals: 494,393 Gnodes, 4,387 stubs (0.00072% of stubspace)
04-27-2010, 04:12 PM
the-mk

output from new version 091:

PHP Code:

Overall OGR-27 PProxy stats based on 109 active days: Smallest: 0.2060 Gnodes (27/5-68-1-3-6-2), submitted on 25/04/2010 Largest: 1,230.03 Gnodes (27/4-16-5-14-17-38), submitted on 14/02/2010 Avg. stub: 77.00 Gnodes Avg. rate: 25,622 G/day, 0.30 Gnodes/sec, 333 stubs/day Most Gnodes: 127,960 Gnodes, 1.48 Gnodes/sec, on 01/01/2010 Most stubs: 4,553 stubs, 190 stubs/hour, on 24/04/2010 Totals: 2,792,813 Gnodes, 36,270 stubs (0.00599% of stubspace)

I think it looks good!
04-27-2010, 04:31 PM
alpha

Thanks! :thumbs:
11-01-2011, 08:42 AM
alpha

1 Attachment(s)
It's been a while folks, are you ready for a new version bulging with features?

v0.93:
- New command line option "-d" allows you to specify a specific year, month or day that you want stats for. The format is YYYYMMDD.
- New command line option "-p" is mandatory for project selection.
- New command line option "-b" shows your 10 smallest stubs.
- New command line option "-t" shows your 10 largest stubs.
- New command line option "-u" uses dynamic units instead of Gnodes for everything.
- The "most nodes" day now shows how many stubs were completed.
- The "most stubs" day now shows how many nodes were completed.
- Code cleanups.
- Possibly some other stuff! This has been developed on and off over the last year and a half.
I believe most options should be compatible with each other. For example, it is possible to choose to see your 10 largest stubs so far for this month, using dynamic units for project OGR-27. The command line option for this would be: "-p OGR-27 -t -u -d 201111". From this you can see that to choose a month you simply omit the day number.

Please report any brokenness here, if anybody still uses this thing.
11-01-2011, 11:08 AM
alpha

There's a bug in v0.93 which applies to the new -b and -t options. If you haven't submitted at least 10 stubs for the date specified, it will produce error messages. This was my fault due to a lack of testing, sorry! :)

I won't bother releasing a bug fix for this, rather, I'll try to release a new version with a couple more improvements in the not so distant future.
11-16-2011, 02:32 AM
alpha

Pro tip: With the new "-d" option, you can get daily stats automatically sent to you by putting something like this in your crontab:

Code:

59 23 * * * cd /path/to/script/ ; /usr/bin/perl pppla_093.pl -p OGR-27 -d `/bin/date +\%Y\%m\%d` -u

That works as-is on FreeBSD, but might need tweaking for Linux or other operating systems.

New version coming soon with bug fix and a few tiny new bits.
12-01-2011, 08:43 AM
alpha

1 Attachment(s)
v0.94:
- Fixed a bug affecting "-b" and "-t" when there were not at least 10 stubs worth of data.
- Corresponding stub marks have been added to the output of "-b" and "-t".
- Corresponding submission dates have been added to the output of "-b" and "-t".
- New command line option "-s" shows an additional stubspace breakdown for OGR projects, including contribution as a percentage of each stubspace.
Example of "-b":

Code:

Smallest stubs for OGR-27: #1 - 0.40 Gnodes (27/1-66-16-3-4-2) on 15/02/2010 #2 - 0.43 Gnodes (27/1-66-16-3-5-2) on 15/02/2010 #3 - 0.61 Gnodes (27/23-45-4-9-1-2) on 20/05/2011 #4 - 0.82 Gnodes (27/23-45-4-9-1-5) on 20/05/2011 #5 - 0.95 Gnodes (27/23-45-4-9-1-7) on 20/05/2011 #6 - 0.95 Gnodes (27/20-49-4-12-2-1) on 01/05/2011 #7 - 0.99 Gnodes (27/23-45-4-9-1-6) on 20/05/2011 #8 - 1.03 Gnodes (27/17-16-50-5-1-3) on 23/03/2011 #9 - 1.04 Gnodes (27/20-49-4-12-3-2) on 01/05/2011 #10 - 1.06 Gnodes (27/25-13-41-9-1-2) on 29/05/2011

Example of "-t":

Code:

Largest stubs for OGR-27: #1 - 1,186.69 Gnodes (27/3-18-2-11-19-41) on 15/11/2011 #2 - 1,157.12 Gnodes (27/4-3-17-12-14-44) on 06/05/2010 #3 - 1,098.91 Gnodes (27/3-20-1-14-17-39) on 13/11/2011 #4 - 1,079.32 Gnodes (27/1-20-9-2-24-38) on 08/09/2009 #5 - 1,059.07 Gnodes (27/2-3-18-4-31-36) on 25/02/2010 #6 - 1,049.72 Gnodes (27/1-20-9-3-24-37) on 14/09/2009 #7 - 1,043.85 Gnodes (27/1-20-9-2-25-37) on 09/09/2009 #8 - 1,017.24 Gnodes (27/2-3-18-4-30-37) on 25/02/2010 #9 - 1,013.64 Gnodes (27/3-12-13-14-4-48) on 12/11/2011 #10 - 1,006.35 Gnodes (27/5-6-14-4-27-38) on 17/05/2010

Example of "-s" (with -u enabled):

Code:

Overall OGR-27 PProxy stats based on 464 active days: Smallest: 395.76 Mnodes (27/1-66-16-3-4-2), submitted on 15/02/2010 Largest: 1.19 Tnodes (27/3-18-2-11-19-41), submitted on 15/11/2011 Avg. stub: 90.33 Gnodes Avg. rate: 2.31 T/day, 26.76 Mnodes/sec, 26 stubs/day Most nodes: 10.79 Tnodes (119 stubs), 124.87 Mnodes/sec, on 16/04/2011 Most stubs: 231 stubs (8.08 Tnodes), 10 stubs/hour, on 14/02/2010 Stubspace 2: 436 (0.24341%) Stubspace 3: 221 (0.00342%) Stubspace 4: 11,218 (0.00379%) Totals: 1.07 Pnodes, 11,875 stubs (0.00196% of stubspace)

Feedback and script output is very welcome, and encouraged. :)