PDA

View Full Version : Bok, I need you...



CaptainMooseInc
09-23-2005, 06:23 AM
Hey Bok, this is quoted from the forums at Riesel Sieve. I'm trying to get usable data output to you to make good stats.

If you have a username over there you can jump over to this thread:
http://www.rieselsieve.com/forum/viewtopic.php?t=413

and send bryan a PM maybe???

I'm hoping to get sieve stats and LLRNET stats for RieselSieve made into the Free-DC sig. stats thingy.

Hopefully you two could talk and get something going. I would really love to see stats appear for this.

Quote from bryan:

"So I'm not 100% sure what Bok (and others) would be looking for in a stats feed for outside-RS stats. However, I've made an alpha version of data output, and can easily add/remove things from it as needed. Should I try and find Bok over at free-dc's forums, or is he perhaps lurking here?

The link itself is http://stats.rieselsieve.com/data/stats.csv . Feel free to comment. The basic format is User, Teamid, Countryid, Tests (LLR)/Factors (Sieve), Score.

Bryan"

Thanks Bok!

-Jeff

CaptainMooseInc
09-23-2005, 06:25 AM
P.S. Total scores are fine...the Riesel Sieve sig that is already around had a lot of data in it if you're just a Riesel Siever...

For Free-DC purposes total Sieve Scores and total LLRNET Scores are fine. :)

Bok
09-23-2005, 07:46 AM
That csv looks fine. It's all that is needed for me to create stats.

Will do it as soon as I get a spare 30 mins.

Bok

Bok
09-23-2005, 03:55 PM
CM,

I need something to refer the team id's to actual team names too, same with country id's if applicable.

i.e.

team_id, team name
1,Free-DC

etc
etc

These can be seperate files (preferable) that I'll parse first.

Bok

CaptainMooseInc
09-23-2005, 05:46 PM
bryan is heading over to our forums here soon I believe to get in contact with you Bok. Instead of me being the intermediary with no coding experience he'll be able to contact you directly and you two can discuss everything. :)

bryan[RS]
09-23-2005, 06:25 PM
Bok,

I've PM'ed the CSV files for both teams and countries. A couple questions I have for you -

How often would you like the CSV data updated, and how should the updates be? (Grand total for the day, or only an incremental?)

Any other information you need in the CSV file, or as a start? What about previous totals?

Just let me know what you need - thanks for your help!

Bryan
RieselSieve Project

Bok
09-23-2005, 06:45 PM
oops, read and replied to you PM before looking at this :)

Update as often as you like, but no more than every hour I'd say.

I *much* prefer to have everything in an update whether it's changed or not, otherwise my current script would flag anything missing as deleted.

Other information, if you already hold project ranking information, that would be a bonus. I can calulcate it if need be though by just setting a flag in a config file. No big deal with this amount of data, it's only when there are 100,000 entries it takes a long time to re-rank :)


Previous totals ??? Not sure what you mean

Bok

bryan[RS]
09-23-2005, 07:01 PM
Bok,

I've set the script to auto-update the CSV every 2 hours, on the even hours.

The stats pages I have calculate rank on-the-fly, so I can't really include the information in the file (although it is sorted in order for today's rank, so I guess that helps.

When I say previous totals, I mean that the CSV file only has the data from today in it. Should I send you a CSV with everything up until today, or should I modify the file so it gives you the score & totals today, then the overall score & totals? That way you have the overall totals as well as the daily totals. Hopefully that's a bit clearer.

Thanks again!

Bryan

Bok
09-23-2005, 07:06 PM
I only need overall totals, my db will do the rest from this point onwards...

If it's sorted in order, then that's fine I'll just use a counter for project rank.

:cheers:

Bok

bryan[RS]
09-23-2005, 07:22 PM
Ok, thanks. I've updated the CSV to pull only the overall scores for both the LLR and Sieve scores. It is sorted by overall score. It added a few more rows, but still not very many.

:cheers: indeed.

Bryan

CaptainMooseInc
09-27-2005, 02:47 PM
hows that stats thing coming along???

just checkin up...

-jeff

Bok
09-27-2005, 03:37 PM
Done,

I just haven't announced it yet...

still need to code the dcr and signature bits anyway, perhaps tonight.. but don't count on it.

http://stats.free-dc.org/new/projpage.php?proj=rllr

http://stats.free-dc.org/new/projpage.php?proj=rsieve

Bok

Chuck
10-09-2005, 12:00 PM
Bok,
Please disregard... this post edited... RS made an error in recomputing stats.
My apologies.

Chuck

em99010pepe
10-09-2005, 12:07 PM
Chuck,

I think the problem is in the stats source:

http://stats.rieselsieve.com/sieve.php

Carlos

bryan[RS]
10-09-2005, 12:09 PM
Chuck,

The change is not on Bok's end. We had to do some DB editing to clear out duplicate factors that had been submitted (either by accidentally entering factexcl.txt and factrange.txt into the submission system, or just true dupes that had not been caught). I think the re-run of the stats did not catch some of the factors correctly, so it sent the numbers for some users plummeting. I'm checking all the totals right now, and hope to have this fixed in 1-2 hours. Don't panic....yet

:trash:


Bryan

Stats Administrator
RieselSieve Project

Chuck
10-09-2005, 12:13 PM
darn you are fast Bry..


I had just updated my post.. and asked to disregard.


lol


Chuck

em99010pepe
10-09-2005, 12:15 PM
Originally posted by Chuck
darn you are fast Bry..


I had just updated my post.. and asked to disregard.


lol


Chuck

And I am faster than Bryan!

I was the one who accidentally entered factexcl.txt and factrange.txt.:blush: I told b2 on Riesel forum.

Carlos

Bok
10-09-2005, 12:40 PM
Carlos is correct,

I can only work with what I'm given I'm afraid.

And yes, chartdirector does not work with -ve's.

Bok

AMDave
10-25-2005, 05:48 AM
Hi guys.

been trying out LLR and noticed that the overall LLR team scores on the RS LLR page are not the same as here.

Found this thread and was wondering if the problems described above have been fully played out or if there are still corrections in progress?

I suspect that the stats file provided to Bok is not reflecting the same as the figures that RS are displaying.

Ta

AMDave

bryan[RS]
10-26-2005, 07:22 PM
Dave,

I've confirmed the discrepancy you've noted. Looking at it, it appears that Bok/Free-DC's stats are actually the correct stats. I have to re-set the scoring algorithm for teams (right now, it bases each team's total each day off of the users on the team, stores it in a DB, and then calculates the total to find our point values. Bok's scores are based on the scores of each user on that team every day, and movements affect the scores immediately. I'll be working out that bug as we work on re-formatting the stats pages.

Bryan
Stats Administrator
RieselSieve Project

CaptainMooseInc
10-26-2005, 08:27 PM
Bok Rules...

:allhail:

Bok
10-26-2005, 08:54 PM
:woot:

:cool:

bryan[RS]
10-26-2005, 08:58 PM
LOL...Now remember, Bok's data is only as good as the data I send him. He has the best data because I CHOSE for him to have the best data....wait no, I just figured out a way to more easily export the user data than the pages I have for teams right now. Guess all I really did was make Bok look good and find another bug to add to my list...Oh Well :)

:gangpunch

b2uc
10-27-2005, 12:24 AM
And I just walk by and look at all the bugs building up in Bryan's corner.... I keep on walking. I guess I should go back to work so we can afford that new SQL server I keep promising....then we can do some crazy stats.


Lee Stephens
Riesel Sieve

CaptainMooseInc
10-27-2005, 05:09 AM
Buy one of Bok's (or two) Dual Xeon systems. I'm sure a Dual 1.6GHz Xeons would be a decent SQL server???

-Jeff

b2uc
10-27-2005, 11:10 AM
I have two Xeon's and 1GB of RAMBus...but alas...no motherboard...I've been looking for one for about 3 months now....well....lazily looking. If anyone can recommend something..I'd be happy to listen.

CPUS:
1800DP/512L2/400/1.5V
SL5Z8 COSTA RICA
3220A396-0302

A49925-01 NB5

so... 603-pin FC-BGA/mPGA

Any ideas on a nice...and affordable MoBo would be great:)
O...and I'm willing to scrap the Rambus if it means an equal or better solution with a MoBo with DDR.


Lee Stephens
Riesel Sieve

AMDave
12-03-2005, 07:07 AM
@ bryan[RS]

looks like the Team Totals are out again.
Can U fix ?
maybe change the team stats page to caculate from a sum of the individual scores aggregated on team ID#


cheers,

AMDave

bryan[RS]
12-03-2005, 11:58 AM
Dave,

The totals are, but aren't, off. The team pages on stats.rieselsieve.com pull from a caching table. That table calculates the team's score on the day it is done - and does not change it when users change teams. The export pages that feed the data to Free-DC, the DC-Vault, and individual teams, however, does exactly what you say - add the overall scores of a user on a team at that moment.

The reason for this is the debate is still on-going. With so few users so far, we really don't have a set-in-stone policy about what happens to a user's points when they change teams. I've set up a poll over at rieselsieve.com to try and get input, as I don't want to unilaterally impose a policy on our user base. I will say, though, that the new stats pages I'm working on do still use the "Team you're on each individual day" concept. Any input will be welcome.

Bryan

Stats Administrator
RieselSieve Project

bryan[RS]
12-04-2005, 10:59 AM
Bok,

Every now and then, I've noticed that the update for the scores here runs early (about 57 past the hour, not 14). The scores then go haywire, but return to normal after an hour or two. Here's the output of the latest log (09:55 CST), I don't know if it might help in debugging:


Starting script at Sun Dec 4 09:55:14 2005
Using config file -> rsieve_configs.pl
Debugging is off
Taking backup of table -->> rsieve_milestone
gzip: /home/backup/stats/rsieve_milestone.2005124955.sql.gz already exists; not overwritten
Taking backup of table -->> rsieve_movement
gzip: /home/backup/stats/rsieve_movement.2005124955.sql.gz already exists; not overwritten
Taking backup of table -->> rsieve_user
gzip: /home/backup/stats/rsieve_user.2005124955.sql.gz already exists; not overwritten
Taking backup of table -->> rsieve_team
gzip: /home/backup/stats/rsieve_team.2005124955.sql.gz already exists; not overwritten
Taking backup of table -->> rsieve_cntrl
gzip: /home/backup/stats/rsieve_cntrl.2005124955.sql.gz already exists; not overwritten
DBD::mysql::st execute failed: You have an error in your SQL syntax near 'teamrank = 7' at line 1 at ./gather.pl line 151.
1 seconds in total elapsed...
Calling local gathering file -> rsieve_gather.pl
Optimizing table --->> rsieve_team
Optimizing table --->> rsieve_user
1 seconds in total elapsed...
1 seconds in total elapsed...
About to parse data..
Parsed through 110 users...
Added 0 new users...
1 seconds in total elapsed...
Running Total is 2572907594.83
1 seconds in total elapsed...
Comparing results with prior totals
difference in user scores is -2571277463.21 proceeding with update...
No change in date........
Doing projranking.....
Doing projrank for users....
No.users = 110
1 seconds in total elapsed...
Getting all teams in to array...
1 seconds in total elapsed...
Number of teams = 18
About to update each team...
setting rank for No Team Selected to 0
high is rena of Ultimate Chaos with 174285.18
highest on Free-DC is with
Updated rsieve_cntrl with rundate and generate date
Updated rsieve_cntrl with top ranks..
2 seconds in total elapsed...
Ending script at Sun Dec 4 09:55:16 2005
Processed 710 sql statements...
...

Bok
12-04-2005, 12:30 PM
I guess I'm picking them up 'inflight' so to speak. Are you outputting the csv files as you calculate the scores ?

Might be better to output them to a variable, then copy to a file at the end.

I could alter the time at which I run the stats though, what would you recommend ?

Bok

bryan[RS]
12-04-2005, 01:08 PM
Bok,

It seems that the stats normally update at :14 past the hour, so I have the export done at :03 past the hour. Our stats run at :27 and :57, so adjusting for differences in times on each computer, it is possible that your stats are catching them as they are updating. However, this is a once in a while thing, as I pointed out. Do you have two different time schedules that you check on? This log output looks different than the normal one, when it runs at :14.

Bryan

Bok
12-04-2005, 01:22 PM
hmmm,

no, it's cron'd to run at 14 past. Can't see how it would start at 55. Just looked through the cron log and it doesn't show it :confused:

I'll see what I can find out.

Bok