stats dumps for third parties?

[Bananeweizen]

Hello !

Would it be possible to have a daily dump of the user and team statistics to a comma separated text file (or other easily parsable format) for use in third party statistics (e.g. on team sites) ?
I guess that should be better for the webserver than having people parse lots of html pages to extract the necessary data.

Ciao, Michael.

[kugano]

Mike,

Yeah, I had planned on making an easily-parsable text version of the stats available on the website. This was on my TODO list but somewhere down the line it got pushed down in priority. I'll see what I can do for you tonight.

Do you have any requests on what format or what types of stats should be available?

(Hooray, I can finally post on this forum now! Thanks to the free-dc people for 'fixing' my account!)

[Bananeweizen]

Hi kugano !

For me two simple text files holding a line for each user (respectively team in the second file) would be ok. Each line could hold comma separated values or the like.
If you prefer anything else, that's ok for me as my statsengine can be fed anything. :)

As data fields the following would be nice (in no particular order):
* some unique id
* name
* (teamid for users)
* current cEMs

Any other interesting data (rankings, active participants in teams, overtaking, speed or the like) can be calculated from that so it is not really necessary.

As update intervals of those files 24 hours are ok for me but there may be people who prefer shorter periods of time.

Ciao and thanks, Michael.

[kugano]

Okay, the moment you've all been waiting for -- raw data is now available. I've posted a thorough description of the format of the data, and where and how to get it, on our website:

http://www.seventeenorbust.com/help/textStats.mhtml

Please read this document before asking questions here, although I'll be more than happy to answer questions that I didn't cover or if you need more information.

If anybody isn't satisfied with the data or format available, let me know now so that we can make any needed changes before everyone starts pounding out code to parse it. :)

If anyone comes up with a cool interpretation of these stats on their sites, I'd like to see it! Just post a link here or email it to me, I'm curious to see what you guys will come up with. :)

Enjoy!

[kugano]

One afterthought I forgot to mention in the format description:

The "total work" and "last day's work" fields CAN AND PROBABLY DO EXCEED the maximum value of a 32 bit integer!! This means that you'll either have to interpret this value as a string, or use large-integer arithmetic on it.

If you're using a crippled language that doesn't have large-integer capability, my suggestion is to truncate the number by chopping off the last 3 (or 6) digits and then adding a suffix like "kilo" or "mega", etc. to your final output.

[shifted]

Originally posted by kugano
One afterthought I forgot to mention in the format description:

The "total work" and "last day's work" fields CAN AND PROBABLY DO EXCEED the maximum value of a 32 bit integer!! This means that you'll either have to interpret this value as a string, or use large-integer arithmetic on it.

In C/C++ try using 'long long' for the datatype (many compilers support it). In java, a 'long' is 64 bits long. In php, to get arbitrary-precision integers, check out http://www.php.net/manual/en/ref.gmp.php (non-windows only).

[kugano]

And of course, in Perl (the language of the Gods) it Just Works (tm). :-)

*kugano disclaims any resposibility for Perl-vs-XXX arguments erupting as a result of this post*
*kugano doesn't care if XXX is better than Perl*

[shifted]

Originally posted by kugano
And of course, in Perl (the language of the Gods) it Just Works (tm). :-)

*kugano disclaims any resposibility for Perl-vs-XXX arguments erupting as a result of this post*

<sarcasm>
I'll use perl when its interpreter doesn't use several megs to run.

*happily uses java*
</sarcasm>

[Bananeweizen]

Wow !

That was fast. I'm really happy with your dump. I will include it in my statistics engine today but I can only give you a link to that site in January as it is still in the last stages of development and will go live in January.
I'm fully satisfied with the given data and format of it.

Ciao, Michael.

BTW: To the language discussion people: Use the right tool for the right job. My stats engine consists of components in perl, php, C++ and Delphi. :)

[soni]

Well, I for one would love to have the stats in XML..

It mean I dont have to parse any of the data, just use them..

Why make all people write their own parser, when XML can do it for you??

[Auritania]

Am I doing something wrong? When I load the stats data page (http://www.seventeenorbust.com/stats/textStats.mhtml) I get nothing but line after line of stuff like

·ëéfk÷„\4(ÃQ„ÙR�yó8nw·‡mÜr¬'=Í„ÐËàÒi9

Do I need to change something somewhere to be able to read it?

[shifted]

Originally posted by Auritania
Am I doing something wrong? When I load the stats data page (http://www.seventeenorbust.com/stats/textStats.mhtml) I get nothing but line after line of stuff like

·ëéfk÷„\4(ÃQ„ÙR�yó8nw·‡mÜr¬'=Í„ÐËàÒi9

Do I need to change something somewhere to be able to read it?

Have you tried a different web-browser? The content is pure text, and the webserver sends a MIME-type of "text/plain" (should be ASCII).

[Auritania]

Originally posted by shifted
Have you tried a different web-browser? The content is pure text, and the webserver sends a MIME-type of "text/plain" (should be ASCII).

That must be it, apparently. IE6 cares not for the format. The component that will actualy grab the page will read it. So I guess it will work where I need it to work.

[kugano]

I'm not sure, but I'd bet solid money that this is what's happening:

The SB web server uses mod_gzip, a plugin for Apache that (if the browser claims to support it) will compress data before sending it out over the network. This results in a LOT less data being transferred, but also means that the browser has to be smart enough to decompress the data before displaying it. I'll bet that for one reason or another, IE6 is saying "yes, I'll accept gzip'd data" but then not really decompressing it. Do you have the latest service packs installed for IE6? I do and mine handles it fine...

If anyone else runs into this problem, let me know. I could always provide a separate link to the text dump and configure mod_gzip to ignore and never compress that file... but this sort of large text file is precisely what mod_gzip is there to improve... ;)

[MAD-ness]

IE (various flavors) throws out weird mod_gzip errors (I have seen it with various stats sites and the people with problems eventually found a way around them, but I can't remember a specific fix).

Thanks for a .txt file stats dump, it was on my list of things to ask for.

I think that you will be pleasantly surprised. As great as the SB stats are, the stats gurus can do some interesting and amazing things given a bit of time and a friendly stats system.

[Roman]

I'd prefer Teams and Users stats to be in separate files, so that it would be not needed to pull information about all 3000+ users to just get the team entry from the file.
While it's possible to fetch the file line by line checking whether the needed line was found, for me that would mean an unnecessary complication of the stats fetching engine...