Stats Coding

Last few days I've been experimenting with the parser somewhat on a home machine.

Instead of writing directly to mysql, I'm writing out the queries to a file on the fly. When it hits 50,000 entries I fork off a job to pipe this through the commandline mysql using Parallel::Jobs in perl. Once finished parsing, I check for success on all running jobs and then continue on with the ranking and movement and the like...

Initial tests are very promising..

For instance, lets take Rieselsieve Boinc.

On the old server it takes roughly 8 mins to parse the whole teams/users/stats.

The new server it takes about the same even though the script is much more refined (it was all in memory anyway as there isn't too many entries)

With the new script this has gone down to around 20 seconds..

Seti stats

old server - won't even run (very memory intensive)
new server - 70mins or so for teams and users only..

new script - 40mins for teams,users and hosts...if it was only teams and users it would be in the 3min range

15 minutes of that last run was for ranking the hosts (1.7million of them), so that can possibly be improved still.

What does this all mean? Well, with that not taking up too much cpu on the server / DB I'm free to add in much more capability on the frontend, like country stats, combined boinc stats etc