Stats Changes

[Dyyryath]

First some great news:

Howard's come through with the unique IDs in the stats dump! :thumbs:

OK, now the other part:

The stats at Zeroth Element will be offline (not updating) for (hopefully) one day while I make some rather sweeping backend changes to take advantage of this new feature.

I'll post to let everyone know when they begin updating again.

At first, you probably won't notice any MAJOR differences, but I can assure you that things are about to get much, much better now that I don't have to worry about people changing usernames and whatnot. :D

[Dyyryath]

Scratch That!

I've decided to build the new stats system from scratch on a new server, which means that the old stats will stay as is (and updating) while I put the new system into operation.

A couple of notes about the new system:


Built on a newer, more powerful server
Expanded to track ALL project users and teams
New design with more functionality


Hopefully, I'll have the basic stuff in place and ready for some beta testing early next week. In the meantime, the current stats will continue on just as they have.

[KWSN_Millennium2001Guy]

:D :cheers: :|party|: :cool:

It's all good Dyyryath!

Keep up the great work!

Your stats are awesome!

Ni! :cheers:

[bwkaz]

:cheers:
Wonderful! If these new features are even half as good as you've made them sound, you deserve... well, I don't know. Something. Perhaps another :cheers: ? ;)

[starkebn]

You rock Dyyryath, I think many projects would be less popular without your stats.

Keep up the stellar work.

[pointwood]

:|party|: :|party|: :|party|:

That just sounds too cool :smoking: :thumbs:

And the best part of it: I should *never*, *ever* be pink :jester:

[IronBits]

Nice one Dyyryath! I love them horses! :D
I fixed the super simple plain stats page here for the new UID field.
www.free-dc.org/distfold/stats.shtml

[MAD-ness]

Even without improvements, your stats are wonderful.

It is great that the stats changes will enable you to tracks all users/teams without too much hassle, as this will hopefully be a boost for the project as a whole.

Everyone should get great stats. :)

[Paratima]

Yeah! What they said! :thumbs:

[Dyyryath]

I started design work on the new system last night and it's looking really good. During the design phase, I've (pretty much) settled on a feature list, and built the appropriate database schema to support it. I've also done some preliminary calculations on what kind of hardware will be required for good performance.

I can tell you, this one's gonna be HUGE, and so far, I'm just planning on the Distributed Folding module for it.

Wanna know what I mean by huge? The database will have (based on current user participation and team count for the whole project):

Approximately 7,000,000 user data entries once it's up to full load. These entries "roll" where as old ones drop off, new ones are added.

Approximately 250,000 team data entries. These also "roll".

That gives two weeks worth of data for every user in the project, and every team in the project. It also includes 'extended data support' with a much longer history for certain teams.

The teams that will start with extended data support enabled are currently:


Free-DC
Team Stir Fry
The Knights Who Say Ni!
Overclocker's Network
Team Ninja
Team Endeavor


These are the teams that I'm tracking now. I'll be able to add (or drop) teams from the extended data tracking list at any time.

Just to wet your appetite, one of the things I've been wanting to do for a long time is what I'm tentatively calling 'virtual teams'.

When I was at ars we were doing a lot of internal competitions with subteams. All the users stayed on the ars team in question, but we had a stats guy divide users into notional 'subteams' and keep track of their respective totals over a set period of time.

Virtual teams would allow anybody to create a team, and add users to it through a simple web interface. You wouldn't be changing which team anybody was actually a part of in the project, but rather grouping them together into a virtual team purely for stats purposes. You could then track them just like you do a real team.

What might this be used for?

For starters, it would make subteam competitions like ars used to have a snap. A team leader sets up a virtual team and adds each user on his 'team' to the roster.

Then, you create a 'view' (another feature coming with the new system) where you compare just the subteams. Voila! Instant gauntlet stats.

Also, since the users don't have to be from the same team, gauntlets could now borrow friendly users from other teams as well. ;)

What if you just wanted to have any easy way to track how you are doing compared to your friends (who may be on different teams than you)?

Easy, you create a virtual team, say "Dyyryath's Hit List", and then add those users to the team. Now you can check the team stats page and see precisely how each of you is doing without seeing anyone else. You can also see how you're performing as a group.

There are obviously a lot of ways that this can work, but that should get the basic idea across.

As I said, I've been wanting to do this for a long time, and things have finally come together (Howard's addition of a UID and some free time for me between work projects and between wakeboarding and snowboarding seasons ;) ), where I can do it. Heh, and this is just the beginning.

I'm also looking at dynamic, 'design your own' style graphs for every stat imaginable and a whole new look for easier readability. To be honest I've got a much longer list of ideas than I'll possibly have time for. :D

I think you guys are gonna like the new stuff...

[Darkness Productions]

/me drools in anticipation.

Man, I gotta tell you Dyy, you're the one that got me into stats as I know them today. The best stuff I could have come up with is already blown away by your DF stats, but this just takes the cake. Seems I may need a "demotion" ;)

[pointwood]

I'll never get any work done :swear: :p :D

That just sounds extremely cool :smoking: :thumbs:

[Jetta]

SWEET!!! thanks for all the work Dyyryath! sounds amazing! :notworthy

[Dyyryath]

Thought I'd post a quick update before I go to the office for the day...

Work on the new stats package is proceeding nicely. I'm spending quite a bit of time on performance issues right now to ensure that the vastly increased size and complexity of both the data and the system which accesses it don't impact the speed of the system too much.

I've already got a couple of days worth of data running, and so far, things look pretty good. Over the next week, while more data accumulates, I'll be working on the new web interface and page displays.

A 'demo' could be available as early as the middle of next week...

[KWSN_Millennium2001Guy]

I can't wait to see what you have cooked up. :p :cool:

[pointwood]

/me jumps around like crazy in pure excitement :D :cool: :thumbs:

[MAD-ness]

Be careful, jumping around in that pink dress might result in you exposing yourself. :(

[IronBits]

:rotfl:

[pointwood]

IronBits - you really shouldn't be laughing - your dress is even uglier ;) :D

[bwkaz]

BAD MENTAL PICTURES!! BAD!!

Oh, I wish you people would stop bringing that thread back up.... :(

:jester:

[Alpha_7]

Are the stats in a state of transition at the moment ?

Is that why they have been reset ?

[cygnussphere]

Dyyryath
Most Honorable StatMaster!

Code On!!!:thumbs:

:notworthy :notworthy :notworthy

[Dyyryath]

Things are continuing along pretty well. The spider (the part that compiles the information into the database) is completed. I've got the performance about where it should be and it's generating data for the database (at about 20 megs worth per day).

I've spent the last couple of days experimenting with the data to see just how much information I can pull out of it and it's looking pretty good. I've got a whole bunch of things that I'll show, but I'm interested in hearing if anyone has something in particular they'd like to see.

For the the teams, I'll be starting with the following:


Basic table updated hourly with name,total users, active users, movement, last update production, last day production, last week production, total production. Each of the production numbers will also display it's relation to the number 1 team's and the 2 teams immediately above and below it's equivalent by means of percentage (i.e. 28% of team x's production)
Each team will have a page similar to the current 'Personal Stats Page' for invidual users showing an adjustable number of recent updates, graphs of production by update and by day, threats & conquests, and defections. The defections table will show people coming to the team and leaving from the team as well as where they came from/went to, and how much production they moved. There will also be options to dynamically graph that team's output in any one of the time periods against any other combination of teams' output.
Each team will also have a user's page like the ones that I do for Free-DC, Ars, etc now, complete with new 'Personal Stats Pages' for the individual users with additional graphing and customization features.
I'm also working on a 'Project Overview' kind of page. It'll have the top 25 teams ranked by total production, last update's production, last day's production, last week's production, total users, active users, and some more unusual things like last day/week's production per active user (which I find REALLY interesting). This page will also have a 'movement' table showing user movement to and from all teams listed in the last 24/48 hours.


This is by no means the complete list. You'll notice that I've left 'Virtual Teams' and such out of this. That's because right now I'm just looking for ways to display/combine data to make interesting stats. There are quite a few other features that will make use of these "combinations".

So, let's hear it. Do you have something you'd like to see? I'll give you an example....

The piece above about production per active user came from my curiosity about which teams had the most 'farmers'. In other words, which teams had the highest average output per day per user?

Want to see other things? How about the top five fastest growing teams? Or the top five fastest dying teams? How about teams with the most number of small crunchers? What about teams with the most number of jumping members (could this be anyone other than KWSN? ;) ).

You'd be surprised what can be gleaned from putting various pieces of information together. I'd like to offer stats that are going to be useful and interesting to people other than myself, so if anyone has any ideas, speak now!

[IronBits]

WOW!
Umm, lemme see...
Show who left what team and where they went... for 24 hours...
:D
I can hardly wait!!! :cry:

[MAD-ness]

Dyy: I knew immediately what you had in mind when you mentioned most structures/day/active member. :)

That is a great stat, as are the ones you list in the final paragraph.

Fastest growing teams is one I would love to see. There is something about watching a team rise to challenge the top that really interests me. There is so much more energy and excitement in a younger, fresher team than in the established teams (just naturally, no knock intended on the top 5 teams).

One thing that I saw requested on the official forums, but which I do not think Howard will implement is this:

Someone wanted production broken down by each protein. i.e. under this protein, this person and team produced x amount of structures.

There is no automatic way of doing this, but in the future, it doesn't seem like it would be hard to just take a shapshot on the day of changeovers for each user/team and store them somewhere. It wouldn't take a lot of data, you would just compare the stats at ...screw it, you can figure it out better than I can.

Anyways, if one already has the collection scripts and the the database, adding this for future usage seems like a simple and low cost thing.

[MAD-ness]

SOOOO much data. I am salivating just thinking about it.

Should we all make donations to some sort of addiction counseling fund in the name of the stats deity?

:)

[Dyyryath]

MAD-ness: That's one I've given some thought to. You'll probably see it after I've had a change over or two to experiment with it. This would also keep the 'Best RMS' numbers, too.

[pointwood]

:shocked: :shocked: :shocked: :smoking: :smoking: :smoking:
Suggestion: Deny access for me in my work hours so I get at least a little bit of work done :p :D

[MAD-ness]

Dyy: Howard is considering adding a page or changing the stats page to display per protein stats.

There is a discussion of it going on over in the technical forum.

Just a head's up.

[Dyyryath]

Thanks for the notice, MAD-ness. :thumbs: