Results 1 to 28 of 28

Thread: pppla - your shiny new pproxy log analyser!

  1. #1
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700

    Exclamation pppla - your shiny new pproxy log analyser!

    pppla - Perl proxyper log analyser

    Now new and improved with OGR-26 and RC5-72 analysis.

    v0.6 - too many changes to list. I had loads of help from Darkness Productions and the RC5-72 code is based on Lani's, so thanks to you guys. I still require the stub-length or diff-length (or whatever) constraints for the stubspaces if anyone can help with that. Any problems let me know.

    The formatting gets a bit broken when you post output here, but this is a rough idea of the new layout:

    OGR-26 output:
    Overall OGR-26 PProxy stats based on 16 days:

    Smallest: 3.66 Gnodes, submitted on 06/11/2008
    Largest: 338.36 Gnodes, submitted on 11/11/2008
    Average: 62.80 Gnodes
    Total: 58,091 Gnodes
    Rate: 3,631 Gnodes/day, 0.04 Gnodes/sec, 58 stubs/day

    Best days:
    6,612 Gnodes, on 10/11/2008
    131 stubs, on 05/11/2008

    Stubs done: 925 (0.00464% of project)
    RC5-72 output:
    Overall RC5-72 PProxy stats based on 25 days:

    First block: 25/03/2007
    Last block: 11/11/2008
    Best day: 01/11/2008 with 864 blocks
    Rate: 116 blocks/day, 5.76 Mkeys/sec

    Keys done: 12,433,930,321,920
    Blocks done: 2,895 (0.000000263% of project)
    Give it a try and post your output. Your feedback is welcome!
    Attached Files Attached Files

  2. #2
    Thanks alpha for the Perl script. I use it now here.
    yoyo

  3. #3
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.

  4. #4
    Quote Originally Posted by alpha View Post
    yoyo, it looks like the script is adding commas after decimal points which is why your smallest stub size looks strange. Hopefully there will be a fix for this soon and I have quite a few under-the-hood changes (future-proofing) for the next release which should be soon anyway.
    Ok, I'll wait for the update
    thx
    yoyo

  5. #5
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.
    Attached Files Attached Files

  6. #6
    Thanks, I updated my server with your new script.
    yoyo

  7. #7
    Junior Member
    Join Date
    Aug 2008
    Location
    Germany
    Posts
    3
    Quote Originally Posted by alpha View Post
    v0.7 - Fixed the comma problem and made quite a lot of changes which will make adding new OGR or RC5 projects trivial. Also some other small miscellaneous changes.
    Hi alpha,

    one minor Problem. I think the counting of working days ist not correct
    for ogrng because there no different logfiles (like for ogrp2); all ogr26 and
    ogr27 logging goes into files like "pproxyogrngYYYYMMDD". But pppla counts
    all logfiles with "ogrng" in it.

    For example after 2 days for my little proxy

    Code:
    Overall OGR-27 PProxy stats based on 121 days:
    
      Smallest: 55.80 Gnodes, submitted on 26/02/2009
       Largest: 615.77 Gnodes, submitted on 25/02/2009
       Average: 279.05 Gnodes
         Total: 17,580 Gnodes
          Rate: 145 Gnodes/day, 0.00 Gnodes/sec, 1 stubs/day
    
     Best days: 11,952 Gnodes, on 26/02/2009
                49 stubs, on 26/02/2009
    
    Stubs done: 63 (0.00032% of project)
    Sure, the % of Project is also wrong until we get numbers from dnet.

    Bye,
    clzupp

  8. #8
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    It's because I haven't released a new version which supports OGR-27 yet. You must've modified the script to have gotten output like you pasted. pppla v0.7 only supports RC5-72 and OGR-26 so if you specified OGR-27 on the command line it would have told you the supported projects in that version and bailed. pppla v0.8 will support RC5-72, OGR-25 (added back in), OGR-26 and OGR-27.

    I could easily write the script to support any future OGR project (it's practically one line of code per project), but I would need to know the total stubs for each project. Unfortunately, without this information there will have to be a new version of the script every time we start a new project. However, the script is already (mostly) future-proofed for this, so all I have to add is a couple of lines. The log filenames doesn't matter, the script looks at the stub information on each line to pick out the appropriate stubs for the project queried.

    As soon as I know the stubspace size of OGR-27 I'll post the new version.

  9. #9
    Target Butt IronBits's Avatar
    Join Date
    Dec 2001
    Location
    Morrisville, NC
    Posts
    8,619
    You should submit this work at sourceforge

  10. #10
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    clzupp, I misread what you said initially and can now see your point. I've made the necessary changes to fix the day (log) counting and that will be in the next version, fully working across all OGR projects (past and present).

    IB: it's only a ~300 line script, <10 kilobytes.

  11. #11
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    OK, dnet have updated their stubspace status page for OGR-27 so I have updated the script.

    v0.8 - Added support for OGRp2-25 back in. Added support for OGR-27. Minor cosmetic tweaks.

    Post any problems or oddities here. I'm on holiday for a week from tomorrow, but I'll keep an eye on things and reply if and when I can.
    Attached Files Attached Files

  12. #12
    Thanks for the script, alpha.
    yoyo

  13. #13
    Junior Member
    Join Date
    Aug 2008
    Location
    Berlin
    Posts
    6
    I edited the source to fix a problem with string and numbering also did adjust the output format.

    Source has detailed stubspace stats you might edit the script in order to disable it.

    good work Alpha
    Attached Files Attached Files

  14. #14
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    Quote Originally Posted by Lani View Post
    I edited the source to fix a problem with string and numbering also did adjust the output format.

    Source has detailed stubspace stats you might edit the script in order to disable it.

    good work Alpha
    Lani, thanks for taking interest in the script again. Unfortunately, the stubspace stats have different constraints for each OGR project, so it isn't such a simple fix. The dnet guys aren't always very forthcoming with the required information (for instance, it wasn't posted for OGR-26 and only recently appeared for OGR-27) and it isn't that interesting anyway, so I preferred to leave it out altogether. Also, for the OGR-27 stubspaces to be numbered correctly, lines 140-143 of your modified script should be:
    PHP Code:
    case $stubspace[1]++ }
    case 
    $stubspace[2]++ }
    case 
    $stubspace[3]++ }
    case 
    $stubspace[4]++ } 
    However, this still won't be right for OGRp2-25 or OGR-26, as I mentioned earlier.

    What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.

    I've had a new version of pppla sitting around for a while now with revamped output formatting (and an additional stat) but I was holding back on posting it until some more major changes were made. If nothing comes up in the next few weeks I'll post it anyway.

  15. #15
    Junior Member
    Join Date
    Aug 2008
    Location
    Berlin
    Posts
    6
    What was the problem with formatting? I realise I'm using arbitrary lengths for some stuff but it seems to have been appropriate thus far.
    formating was ok. I edited it, because the dots weren't inline. so i modified it that all dots are in one column for each output. That's the way I like it best.

    anyway, it's still a great tool, in order to quickly get a good overview on the logfiles.

  16. #16
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    v0.9 - tweaked the output to try and get things aligned slightly better (thanks for the inspiration Lani). Also the stats have been beefed up a bit.

    PHP Code:
    Overall OGR-27 PProxy stats based on 90 active days:

        
    Smallest:     7.37 Gnodessubmitted on 18/08/2009
         Largest
    1,079.32 Gnodessubmitted on 08/09/2009
       Avg
    stub:   158.87 Gnodes
       Avg
    rate:    2,065  G/day0.02 Gnodes/sec13 stubs/day
     Most Gnodes
    :    8,102 Gnodes0.09 Gnodes/secon 20/08/2009
      Most stubs
    :      135  stubs6 stubs/houron 18/08/2009

           Total
    :  185,884 Gnodes1,170 stubs (0.00019of project
    The spacing is somewhat dynamic, so if you have super large (or small) numbers it should still output nicely. I'd be interested to know how this works out for everyone.

    I have also made some significant changes to the RC5-72 portion of the script, but I'll detail those in the other thread.
    Attached Files Attached Files

  17. #17
    almost retired the-mk's Avatar
    Join Date
    Jan 2003
    Location
    KI/OOE/Austria
    Posts
    1,921
    Blog Entries
    6
    I think I like it
    PHP Code:
    Overall OGR-27 PProxy stats based on 2 active days:

        
    Smallest:     3.52 Gnodessubmitted on 27/12/2009
         Largest
    :   885.33 Gnodessubmitted on 26/12/2009
       Avg
    stub:    87.10 Gnodes
       Avg
    rate:   39,240  G/day0.45 Gnodes/sec451 stubs/day
     Most Gnodes
    :   73,271 Gnodes0.85 Gnodes/secon 27/12/2009
      Most stubs
    :      838  stubs35 stubs/houron 27/12/2009

           Total
    :   78,480 Gnodes901 stubs (0.00015of project
    something like "what's the name of the smallest/largest stub" would be cool
    something similar like
    PHP Code:
    ...
        
    Smallest:     3.52 Gnodessubmitted on 27/12/2009 --> 27/x-y-z
         Largest
    :   885.33 Gnodessubmitted on 26/12/2009 --> 27/z-x-y
    ... 
    and I like it
    the-mk

  18. #18
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    Thanks for the feedback! I should be able to add the stub marks like you mentioned, although that will make the output of those two lines quite long. I'll have a think about the best way to do it.

  19. #19
    almost retired the-mk's Avatar
    Join Date
    Jan 2003
    Location
    KI/OOE/Austria
    Posts
    1,921
    Blog Entries
    6
    depends on the length of the output... you could also consider something like this:
    PHP Code:
    ...
    Smallest OGR-27 stub:
    ---------------------
    3.52 Gnodes
    submitted on 27
    /12/2009
    27
    /x-y-z

    Largest OGR
    -27 stub:
    --------------------
    885.33 Gnodes
    submitted on 26
    /12/2009
    27
    /y-z-x
    ... 
    but that is just a suggestion...
    the-mk

  20. #20
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    How does this look?

    PHP Code:
    Overall OGR-27 PProxy stats based on 95 active days:

        
    Smallest:     7.37 Gnodes (27/1-20-7-60-3-2), submitted on 18/08/2009
         Largest
    1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009
       Avg
    stub:   158.92 Gnodes
       Avg
    rate:    1,966  G/day0.02 Gnodes/sec12 stubs/day
     Most Gnodes
    :    8,102 Gnodes0.09 Gnodes/secon 20/08/2009
      Most stubs
    :      135  stubs6 stubs/houron 18/08/2009

           Total
    :  186,726 Gnodes1,175 stubs (0.00019of project
    or

    PHP Code:
    Overall OGR-27 PProxy stats based on 95 active days:

        
    Smallest:     7.37 Gnodessubmitted on 18/08/2009 --> 27/1-20-7-60-3-2
         Largest
    1,079.32 Gnodessubmitted on 08/09/2009 --> 27/1-20-9-2-24-38
       Avg
    stub:   158.92 Gnodes
       Avg
    rate:    1,966  G/day0.02 Gnodes/sec12 stubs/day
     Most Gnodes
    :    8,102 Gnodes0.09 Gnodes/secon 20/08/2009
      Most stubs
    :      135  stubs6 stubs/houron 18/08/2009

           Total
    :  186,726 Gnodes1,175 stubs (0.00019of project
    I'm not completely convinced yet that it looks OK. I tried your suggested format but it looked a bit strange, especially with the new dynamic spacing code that I introduced in the last release.

    I've toyed with the idea of making it an optional component, enabled by a command line flag. Maybe I'm putting too much thought into it.

  21. #21
    almost retired the-mk's Avatar
    Join Date
    Jan 2003
    Location
    KI/OOE/Austria
    Posts
    1,921
    Blog Entries
    6
    I like the first one, but the second is also an option
    the-mk

  22. #22
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    v0.91 - Added more stub details for smallest/largest stubs, as per the-mk's suggestion. There have also been lots of code cleanups and tweaks. Significant changes have been made to the RC5-72 code, but again, I'll go into more details in the appropriate thread.

    Please post problems, feedback, or even your own output just so I can see that it works as it should for different length outputs (mostly to check that the dynamic alignment works properly).

    PHP Code:
    Overall OGR-27 PProxy stats based on 165 active days:

        
    Smallest:   0.3958 Gnodes (27/1-66-16-3-4-2), submitted on 15/02/2010
         Largest
    1,079.32 Gnodes (27/1-20-9-2-24-38), submitted on 08/09/2009
       Avg
    stub:   112.69 Gnodes
       Avg
    rate:    2,996  G/day0.03 Gnodes/sec27 stubs/day
     Most Gnodes
    :   10,273 Gnodes0.12 Gnodes/secon 16/02/2010
      Most stubs
    :      231  stubs10 stubs/houron 14/02/2010

          Totals
    :  494,393 Gnodes4,387 stubs (0.00072of stubspace
    Attached Files Attached Files

  23. #23
    almost retired the-mk's Avatar
    Join Date
    Jan 2003
    Location
    KI/OOE/Austria
    Posts
    1,921
    Blog Entries
    6
    output from new version 091:
    PHP Code:
    Overall OGR-27 PProxy stats based on 109 active days:

        
    Smallest:    0.2060 Gnodes (27/5-68-1-3-6-2), submitted on 25/04/2010
         Largest
    :  1,230.03 Gnodes (27/4-16-5-14-17-38), submitted on 14/02/2010
       Avg
    stub:     77.00 Gnodes
       Avg
    rate:    25,622  G/day0.30 Gnodes/sec333 stubs/day
     Most Gnodes
    :   127,960 Gnodes1.48 Gnodes/secon 01/01/2010
      Most stubs
    :     4,553  stubs190 stubs/houron 24/04/2010

          Totals
    2,792,813 Gnodes36,270 stubs (0.00599of stubspace
    I think it looks good!
    the-mk

  24. #24
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    Thanks!

  25. #25
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    It's been a while folks, are you ready for a new version bulging with features?

    v0.93:

    • New command line option "-d" allows you to specify a specific year, month or day that you want stats for. The format is YYYYMMDD.
    • New command line option "-p" is mandatory for project selection.
    • New command line option "-b" shows your 10 smallest stubs.
    • New command line option "-t" shows your 10 largest stubs.
    • New command line option "-u" uses dynamic units instead of Gnodes for everything.
    • The "most nodes" day now shows how many stubs were completed.
    • The "most stubs" day now shows how many nodes were completed.
    • Code cleanups.
    • Possibly some other stuff! This has been developed on and off over the last year and a half.


    I believe most options should be compatible with each other. For example, it is possible to choose to see your 10 largest stubs so far for this month, using dynamic units for project OGR-27. The command line option for this would be: "-p OGR-27 -t -u -d 201111". From this you can see that to choose a month you simply omit the day number.

    Please report any brokenness here, if anybody still uses this thing.
    Attached Files Attached Files

  26. #26
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    There's a bug in v0.93 which applies to the new -b and -t options. If you haven't submitted at least 10 stubs for the date specified, it will produce error messages. This was my fault due to a lack of testing, sorry!

    I won't bother releasing a bug fix for this, rather, I'll try to release a new version with a couple more improvements in the not so distant future.

  27. #27
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    Pro tip: With the new "-d" option, you can get daily stats automatically sent to you by putting something like this in your crontab:

    Code:
    59      23      *       *       *       cd /path/to/script/ ; /usr/bin/perl pppla_093.pl -p OGR-27 -d `/bin/date +\%Y\%m\%d` -u
    That works as-is on FreeBSD, but might need tweaking for Linux or other operating systems.

    New version coming soon with bug fix and a few tiny new bits.

  28. #28
    Dungeon Master alpha's Avatar
    Join Date
    Mar 2002
    Location
    Norfolk, UK
    Posts
    1,700
    v0.94:

    • Fixed a bug affecting "-b" and "-t" when there were not at least 10 stubs worth of data.
    • Corresponding stub marks have been added to the output of "-b" and "-t".
    • Corresponding submission dates have been added to the output of "-b" and "-t".
    • New command line option "-s" shows an additional stubspace breakdown for OGR projects, including contribution as a percentage of each stubspace.


    Example of "-b":

    Code:
    Smallest stubs for OGR-27:
    
    #1       - 0.40 Gnodes (27/1-66-16-3-4-2) on 15/02/2010
    #2       - 0.43 Gnodes (27/1-66-16-3-5-2) on 15/02/2010
    #3       - 0.61 Gnodes (27/23-45-4-9-1-2) on 20/05/2011
    #4       - 0.82 Gnodes (27/23-45-4-9-1-5) on 20/05/2011
    #5       - 0.95 Gnodes (27/23-45-4-9-1-7) on 20/05/2011
    #6       - 0.95 Gnodes (27/20-49-4-12-2-1) on 01/05/2011
    #7       - 0.99 Gnodes (27/23-45-4-9-1-6) on 20/05/2011
    #8       - 1.03 Gnodes (27/17-16-50-5-1-3) on 23/03/2011
    #9       - 1.04 Gnodes (27/20-49-4-12-3-2) on 01/05/2011
    #10      - 1.06 Gnodes (27/25-13-41-9-1-2) on 29/05/2011
    Example of "-t":

    Code:
    Largest stubs for OGR-27:
    
    #1       - 1,186.69 Gnodes (27/3-18-2-11-19-41) on 15/11/2011
    #2       - 1,157.12 Gnodes (27/4-3-17-12-14-44) on 06/05/2010
    #3       - 1,098.91 Gnodes (27/3-20-1-14-17-39) on 13/11/2011
    #4       - 1,079.32 Gnodes (27/1-20-9-2-24-38) on 08/09/2009
    #5       - 1,059.07 Gnodes (27/2-3-18-4-31-36) on 25/02/2010
    #6       - 1,049.72 Gnodes (27/1-20-9-3-24-37) on 14/09/2009
    #7       - 1,043.85 Gnodes (27/1-20-9-2-25-37) on 09/09/2009
    #8       - 1,017.24 Gnodes (27/2-3-18-4-30-37) on 25/02/2010
    #9       - 1,013.64 Gnodes (27/3-12-13-14-4-48) on 12/11/2011
    #10      - 1,006.35 Gnodes (27/5-6-14-4-27-38) on 17/05/2010
    Example of "-s" (with -u enabled):

    Code:
    Overall OGR-27 PProxy stats based on 464 active days:
    
        Smallest: 395.76 Mnodes (27/1-66-16-3-4-2), submitted on 15/02/2010
         Largest:   1.19 Tnodes (27/3-18-2-11-19-41), submitted on 15/11/2011
       Avg. stub:  90.33 Gnodes
       Avg. rate:   2.31  T/day, 26.76 Mnodes/sec, 26 stubs/day
      Most nodes:  10.79 Tnodes (119 stubs), 124.87 Mnodes/sec, on 16/04/2011
      Most stubs:    231  stubs (8.08 Tnodes), 10 stubs/hour, on 14/02/2010
    
     Stubspace 2:    436 (0.24341%)
     Stubspace 3:    221 (0.00342%)
     Stubspace 4: 11,218 (0.00379%)
    
          Totals:   1.07 Pnodes, 11,875 stubs (0.00196% of stubspace)
    Feedback and script output is very welcome, and encouraged.
    Attached Files Attached Files

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •