PDA

View Full Version : STATS files



IronBits
12-19-2004, 12:09 PM
Are there static files that can be downloaded and/or parsed to support 3rd party stats?
Sorry if it's been asked before.
We would like to track all teams and all users...
Thanks

Matt
12-19-2004, 12:20 PM
For my stats sig I just capture the page and parse the data out of the HTML. If there is a better way I wouldn't mind using it myself.

Bok
12-19-2004, 12:25 PM
There are text files/csv files, or at least there used to be, just can't remember the location and can't seem to find it now :\

Bok

Joe O
12-19-2004, 02:04 PM
Originally posted by Louie:

SEVENTEEN OR BUST
PROJECT STATISTICS
RAW TEXT DUMP INFO

URL of text dump:
http://www.seventeenorbust.com/stats/textStats.mhtml

Yes, it has an "mhtml" extension. No, it is not an HTML
file. Yes, it is in plain text ("text/plain").

GENERAL
-------

The statistics for the Seventeen or Bust project, including system-
wide statistics and team- and user-based statistics, are dumped
into a raw text file that can be parsed by external sites for use
on their web pages.

The data in this text dump can be used on any site with only one
condition: that site must, somewhere, have a link to the actual
SB project site at http://www.seventeenorbust.com/ so that a
viewer can see our actual web-based statistics.

The text dump is available at:

http://www.seventeenorbust.com/stats/textStats.mhtml

Yes, I KNOW this link has an "mhtml" extension, but it is a
PLAIN TEXT file (NO HTML!), and the web server's "Content-Type"
response header will be set to "text/plain".

The contents of the text file are cached by our web server and
are recomputed EVERY 15 MINUTES.


NOTE READ THIS NOTE READ THIS NOTE READ THIS NOTE
-------------------------------------------------

PLEASE, PLEASE do not try to access this text dump every time
a user visits your page! Come up with some method of reading
our file every half hour or something and storing the parsed
results, however you want them, on your site. Accessing our
text dump for every hit not only puts unnecessary load on our
web server, but it will also make the data on your site appear
slower! PLEASE don't do this!


FORMAT
------

The text dump is a plain ASCII text file, where each line gives
a certain type of statistic. Each line contains a number of
fields, separated by whitespace (you can assume the separator
between fields will always be a single space, ASCII 0x20).

At the moment there are four types of lines: "Updated", giving
the time of the last recomputation of the stats, "System",
giving the aggregate system-wide statistics, "Team", giving
stats for a particular team, and "User", giving stats for a
particular user. In the future, more line types may be added,
so a parser should ignore lines that begin with unknown type
strings.

The different types of lines are explained below:



Updated <update_time>

update_time = time at which these statistics were last
recomputed, represented as the number of
seconds since 0000 GMT 1 January 1970
(suitable for use with POSIX date/time
functions like strftime)



System <total_cems> <last_day_cems> <total_tests> <last_day_tests>

total_cems = total number of cEMs crunched by the system

last_day_cems = number of cEMs crunched in the last 24 hours

total_tests = total number of Proth tests completed

last_day_tests = number of Proth tests completed last 24 hours



Team <id> <name> <create_time> <owner_ID> <total_tests>
<last_day_tests> <pending_tests> <total_work> <last_day_work>
<first_assign> <last_report>

id = our internal ID number for this team

name = name of the team (NOTE: spaces in the team
name are converted to UNDERSCORES "_")

create_time = time the team was created, expressed as the
number of seconds since 0000 GMT 1 January
1970 (suitable for use with POSIX date/time
functions like strftime)

owner_ID = ID number of the user who 'owns' this team

total_tests = total number of tests completed by the team

last_day_test = tests completed by team in the last 24 hours

pending_tests = number of tests currently assigned to
members of the team

total_work = total cEMs crunched by the team

last_day_work = cEMs crunched by the team in last 24 hours

first_assign = time at which the first block was assigned
to the team, OR the string "never" if the
team has never been assigned a block

last_report = time of the last progress report from any
member of the team, OR the string "never"
if there have never been any progress reports
from team members



User <id> <username> <signup_time> <team_ID> <total_tests>
<last_day_tests> <pending_tests> <total_work> <last_day_work>
<first_assign> <last_report>

id = our internal ID number for this user

name = the user's username (spaces not allowed)

signup_time = time the user's account was created, expressed
as the number of seconds since 0000 GMT 1
January 1970 (suitable for use with POSIX
date/time functions like strftime)

team_ID = ID number of the team to which this user
belongs, OR the string "none" if the user
does not belong to a team

total_tests = total number of tests completed by the user

last_day_test = tests completed by user in the last 24 hours

pending_tests = number of tests currently assigned to user

total_work = total cEMs crunched by the user

last_day_work = cEMs crunched by the user in last 24 hours

first_assign = time at which the first block was assigned
to the user, OR the string "never" if the
user has never been assigned a block

last_report = time of the last progress report from the
user, OR the string "never" if there have
been no progress reports from the user



(end)




Copyright © 2002, 2003, 2004 L. Helm ([email protected]) & D. Norris
([email protected])

Bok
12-19-2004, 03:33 PM
Thanks for the info. That is the one I'd seen before. Only problem I see with it is that it shows a users full score and notthe score they have for their current team.

For instance I have a total of 47.321T, but only 3.88T of that is with Free-DC, the rest was crunched on another team.

So anyone know how we would get that info out ?

:cheers:

Bok

IronBits
02-04-2005, 02:32 AM
Any chance of getting this changed, or to keep it from borking other 3rd part stats, kick out a 'rawstats.txt' ?

IronBits
02-11-2005, 07:07 AM
BUMP - Can you change the raw_stats output to include the information Bok requested for his and other 3rd party stats creators ???

Joe O
02-11-2005, 10:48 AM
As long as you're changing the stats files, could you include 2 new columns. The first would be the count of results where the residue has been proven wrong, and the second where the residue has been proven correct. The sum of these subtracted from the total results would tell us which ones either have no second residue, or a nonmatching residue that has not yet been proven to be either wrong or right. If I were greedy, I would ask for a third column that would be the count of residues that have no second residue (or the suspect ones if this is easier) Then we could track all four possibilities for each user. Thank you.

IronBits
02-12-2005, 11:28 AM
BUMP

Almost 2 months and no resolution/response ... :bonk:

IronBits
02-18-2005, 07:51 AM
^BUMP

Is there no interest from the project admins of supporting 3rd party stats ?
:Pokes:

maddog1
02-18-2005, 05:54 PM
Getting out a bug-free V3, including many of the improvements mentioned in this forum during the last months should be a higher priority and that's hopefully what the admins are currently working on ;)

Jkusuda
02-21-2005, 04:08 PM
Would appreciate seeing an admin at least respond and have the courtesy to say no to the requests made. I too would like to see more detalied and accurate 3rd party stats.

black_civic55
02-21-2005, 06:16 PM
me three

Matt
02-22-2005, 10:42 AM
me four! come on lads, it's not hard just a few simple SQL queries dumping output to a csv running on a crontab.

black_civic55
02-22-2005, 05:08 PM
haha wish i knew more about computers haha but what he said!!! :bonk:

wolfsraider
03-03-2005, 06:07 PM
Geoffs from TeAm Anandtech has a stats site as well that you might be intrested in using, it's url is http://sb.tastats.com/ .

I hope this helps :cheers:

Mike

Electrolyte
03-03-2005, 06:18 PM
Matt, I love your stat images so much I had to use one :D

Bok
03-03-2005, 06:37 PM
Originally posted by wolfsraider
Geoffs from TeAm Anandtech has a stats site as well that you might be intrested in using, it's url is http://sb.tastats.com/ .

I hope this helps :cheers:

Mike

Mike,

Geoff's suffers from the same problems I'm afraid. Take a look at the Free-DC stats in it and look at my total, then go to the seventeenorbust site and look at Free-DC's stats. It doesn't take into account that I did 40T or so over at US-D before moving to Free-DC.

Bok

IronBits
03-28-2005, 09:40 AM
^BUMP :confused: :help:

Bok
06-13-2005, 01:36 PM
Another BUMP....

surely all you'd need to do is split the users lines into multiple where users have moved teams....

A response yes or no would be nice :)

Bok

DukeBox
11-18-2006, 10:44 AM
Bumb.. it would be nice to have some raw stats concerning the work done for previous teams. Maybe just a list added of another file. Parsing all the html pages would stress your server and that should not be neccesary.

DukeBox
11-20-2006, 01:42 PM
bumb again..

DukeBox
11-22-2006, 04:53 AM
Sorry to bumb this again but it would be nice to get answer (fulfilling or not.. )

Sceptic
11-22-2006, 02:27 PM
Why don't you contact the authors directly?
Just go here (http://www.seventeenorbust.com/help/about.mhtml) and use the appropriate email adress and you should be fine.
I don't think the authors visits this forum very often (maybe Alien88 does, someone enlighten me!).

Sceptic

DukeBox
11-22-2006, 03:43 PM
Why don't you contact the authors directly?
Just go here (http://www.seventeenorbust.com/help/about.mhtml) and use the appropriate email adress and you should be fine.
I don't think the authors visits this forum very often (maybe Alien88 does, someone enlighten me!).

Sceptic
/me is stupid :P

I was checking everywhere for an address.. tryed severel but completely missed these one.. TNX (i'll post here the outcome offcouse)

Alien88
11-23-2006, 02:21 AM
Happy turkey day.

Please look at this:
http://www.seventeenorbust.com/stats/textStats2.mhtml

It's the same idea as textStats.mhtml, however, this is grouped per user/team.

EG:
User 4044 Bok 1056123856 60 1068 1 29 1.12492637691787e+15 3076534010668.27 1074325317 1164262456
User 4044 Bok 1056123856 201 432 0 0 43440648500256 0 1056124721 1075231400

Show's Bok's info broken into each team he's been on.. same order of stats as textStats.mhtml.

Does this work for you guys?
Alien88

DukeBox
11-23-2006, 03:54 AM
Happy turkey day.
Does this work for you guys?
Alien88
/me is checking it out now.. brb.

Looks great, must do some scripting/checking if my engine will eat this ;)

hmm.. see my post below

DukeBox
11-23-2006, 08:36 AM
Ok, there are some strange things..

On User Statistics (http://www.seventeenorbust.com/stats/users/) :

Total number of users 12058 users
Users participating on teams 5478 users

The textStats.mhtml

cat textStats.mhtml | cut -d" " -f1 | sort | uniq -c
1 System
481 Team
1 Updated
12060 User

Well, then checking the textStats2.mhtml

cat textStats2.mhtml | cut -d" " -f1 | sort | uniq -c
1 System
481 Team
1 Updated
11351 User
In the textStats2.mhtml are ofcouse duplicate users entry's so filtering those out..

cat textStats2.mhtml | grep "User " | cut -d" " -f1,2,3 | sort | uniq | wc -l
9718So this means 2340 (12058-9718) users gone ?

To go any deeper.. i took one example user thats not in the textStats2.mhtml

User Statistics for goiter (http://www.seventeenorbust.com/stats/users/user.mhtml?userID=9303)

This user is a member of the Dutch Power Cows team.
Team Statistics for Dutch Power Cows (http://www.seventeenorbust.com/stats/teams/team.mhtml?teamID=34)
User does not exist in this team.. so where did he go ?

This all concludes me

there are some manual edits for that user and it's filtered out of the team.
or
the stats on www.seventeenorbust.com are not correct.

umccullough
11-23-2006, 11:03 AM
In the textStats2.mhtml are ofcouse duplicate users entry's so filtering those out..
So this means 2340 (12058-9718) users gone ?

Maybe it doesn't include users who aren't on a team at all...?

Always possible that someone used an inner join instead of an outer join when building that query...

Also, while goiter's user info says he/she is in the Dutch Power Cows - I cannot find that user name in the Dutch Power Cows team roster. I have seen this happen on our team (TeamHaiku) also - where several "stagnant" users simply fell off the roster at one point. I don't think any of them had actually committed work to the team. In goiter's case, possible that he/she committed work PRIOR to joining Dutch Power Cows, and then stopped - and eventually fell off the roster as I have seen before.

I also have to raise questions about the username/teamname specification on the client - what happens if the user specifies a different teamname on the client? is it simply ignored by the server? was it always ignored by the server?

DukeBox
11-23-2006, 11:42 AM
Maybe it doesn't include users who aren't on a team at all...?

No, team 'none' and users with 0 (zero) point are also included in the textStats2.mhtml so that's not the problem.

I'm going into the 'user joined team but did not produce anymore' thingy, i'll let you know.

update:
I think you solved my puzzle..
This happens,
User X is teamless, he makes some points and after a while he stops running the client.
After a while he joins Team Y, but because hoe does not have any points for team Y he dissapeares from the statistics.

I wonder if it's save to assume that i can import all dissapeared users for team 'none', ill give it a try.. and let you know.

Many thanks for your idea's.. (!!!)

Alien88
11-23-2006, 12:12 PM
textStats2.mhtml does not include users who have never submitted any work.. hence the lower amount of users and smaller file size.

I don't see a reason to include those in textStats2

Alien88

DukeBox
11-23-2006, 01:58 PM
textStats2.mhtml does not include users who have never submitted any work.. hence the lower amount of users and smaller file size.
I don't see a reason to include those in textStats2

Ok, i understand that why bother mentioning zero's.. but there is one inconsistancy which confused me a bit:

User 574 santzi 1036502982 none 0 0 0 0 never never

I also understand that users that did work under an other team then it's currents dont show up like this one who is now on team 81:

User 4444 Brucifer 1061750791 81 48 0 0 18029911579056 0 1061751996 1144036704

He did only delever work under team 83 and 201:

User 4444 Brucifer 1061750791 83 1 0 0 1397299036160 0 1143411251 1144036704
User 4444 Brucifer 1061750791 201 47 0 0 16632612542896 0 1061751996 1111385384

What's confusing me more are users like:

User 3869 kamakazemike 1054737697 81 0 0 0 28499436864 0 1054737926 1054841352
Who is in team 81 BUT showing up in the textStats2.mhtml as teamless.

User 3869 kamakazemike 1054737697 none 0 0 0 28499436864 0 1054737926 1054841352
That I understand.. they did not do any work under there current teamid's

Eh... actually i think i got it (by typing this one, it became clear :P)

I'll go on and recreate the stats.. it seem's i have to think of something that fixes the history tnx for your time and comments !:thumbs:

DukeBox
11-23-2006, 03:31 PM
Oops.. found a few users that are still confusing me,

textStats.mhtml < User 6968 Adabiviak none 8263477504464
textStats2.mhtml > User 6968 Adabiviak none 999892976
textStats2.mhtml > User 6968 Adabiviak none 8262477611488Why is that user 2 times in team none ? I assume he switched team and went back without doing any point in it's temporarely team ?

It seems only to happen when turning back to teamless not within a team.
edit: this last scentence is correct, i diffed all users. B.t.w, user jeramy is the one with the biggest identity crisis ;P

Thanks again !

Alien88
11-24-2006, 06:40 PM
Oops.. found a few users that are still confusing me,
Why is that user 2 times in team none ? I assume he switched team and went back without doing any point in it's temporarely team ?

It seems only to happen when turning back to teamless not within a team.
edit: this last scentence is correct, i diffed all users. B.t.w, user jeramy is the one with the biggest identity crisis ;P

Thanks again !

The processing of the stats interprets a team of NULL or 0 as "none". In this users case, he at one point never had a team (NULL) and now is a member of team "0" or no team - hence the two "none" results.

Hope that helps...

DukeBox
11-29-2006, 06:04 PM
Okidoki.. another question, some users that have alle zero's, there is an extra space and total_work is missing. Should this not be an extra 0 ?

Alien88
12-01-2006, 05:36 PM
Okidoki.. another question, some users that have alle zero's, there is an extra space and total_work is missing. Should this not be an extra 0 ?

Can you give me an example of users like this so I don't have to hunt through the whole file and I know what you're looking at?

Thanks!
Mike

DukeBox
12-02-2006, 08:22 AM
examples:

( textStats2.mhtml 2006-12-02 14:15:07 )
team examples:

Team 538 CacheFlow 1150093426 11397 0 0 0 0 never never
Team 472 Enfield_Town 1124040725 9852 0 0 0 0 1124109980 1124109980
user examples:

User 8191 1001 1100203530 none 0 0 0 0 1100459588 1100459610
User 11373 25 1149500426 none 0 0 0 0 1149500438 1149757930
You can find them by grepping for '0 0' (2 spaces)

(Edit)
Oh.. one other question, the stats are genereated every 15 min.. any idea when they are updated ? Lets say, your query starts at 13:00:00, is it ready at 13:00:01 of 13:14:00 ?