Bug reporting...

[[H]Skillz]

I imagine you are working on the stats site probably a lot so these bugs might be on your to do list, but just in case you overlooked them I will point them out as I find them.

If I go to my overall stats page here:
https://stats.free-dc.org/userbycpid/cc2bdbcee7976d78486beeeb937bd376

Then click on the "Most Milestones" Rank T category I am taken to the overall milestone page instead of my teams milestones.

[Bok]

Fixed..

[[H]Skillz]

Quick service!

Found another bug. When you go to the Gerasim stats page https://stats.free-dc.org/proj/ger#select I get a drop down alert saying something about tables and the data doesn't ever show anything.

[Bok]

It's some foreign characters breaking the json returned to datatables, not sure why it's causing an issue though. Will try to debug.

[Bok]

ok, kind of fixed.... not sure it's the best solution

[[H]Skillz]

If I click on "Teams" under the project name I get the same tables error message and the team list doesn't ever load.

[Bok]

Fixed, same issue as before. I suspect it's in a few other places, so keep 'em coming if you find them.

Hosts I still have to convert so it isn't using datatables right now. Hope to work on that this weekend.

[[H]Skillz]

Sounds good man. You're awesome. Love the site as it's my favorite stats site for BOINC.

[vaughan]

bok I'm getting this message when I go to Gerasim stats:

DataTables warning (table id = 'users'): DataTables warning: JSON data from server could not be parsed. This is caused by a JSON formatting error.

[Bok]

Fixed, let me know if you find more. It's a json issue with a name that has utf8 characters where I have to use htmlentities prior to encoding. I'd fixed it prior in the top10 teams but this one was in the users part

[Ken_g6[TA]]

Hi again, Bok.

When I go to some pages, such as https://stats.free-dc.org/teams/eni and I try to sort by some fields, like "Yesterday", the sorting doesn't work.

[Bok]

hmm, it does a check to see how many teams there are before deciding whether to do serverside processing (only on certain rows where there are indexes) or pull everything in and allow local sorting (on anything).

In this case it's using serverside, so something is wrong on that logic, which is a lot more complicated.

I've upped the count to be 2500 before it goes serverside for now, so it's working for Enigma, will debug it on larger projects

[Bok]

hmm, it does a check to see how many teams there are before deciding whether to do serverside processing (only on certain rows where there are indexes) or pull everything in and allow local sorting (on anything).

In this case it's using serverside, so something is wrong on that logic, which is a lot more complicated.

I've upped the count to be 2500 from 1000 before it goes serverside for now, so it's working for Enigma (takes a while to pull all that data though) will debug it on larger projects

[Mildew]

Hello Bok.
I'll start off by saying I love the stats page, and I use it every day. So these are not complaints, just bug reports and humble suggestions ;)

When I look at for example my stats for MilkyWay here: https://stats.free-dc.org/user/mil/150943
The User Daily Scores and User Cumulative Scores for Teammates are empty.
The user Cumulative Scores for Project Members has the y-axis go from 0. Would it be possible to rescale the y-axis to make the diagram more readable?
The user Cumulative Scores for Project Members has the legend on the top right hand. This is really the most interesting part of the diagram, so putting the legend in any other position would be better I think. The best option may be to put it at the top as in the User Daily Scores - Project Members diagram where it doesn't overwrite any of the information in the actual diagram.
The old (stats3) page used to have a pop-up in the hosts list so you could tell which computer is doing work without having to click the link. I found this very useful. Would it be possible to get that feature in the new stats page?
The Milestones diagram is a bit tricky to read since the distance between points on the x-axis is arbitrary. When each point in the diagram doesn't have a readable value on the x-axis it gets very difficult to get an actual overview of your progress. I don't have a concrete suggestion on how to make the presentation better, but since the milestone information is already available in the top-right table I think I would find it more informative to have a "lifetime production" graph where the x-axis is a continuous time scale.

Thanks!

[Bok]

I intend to look at all the charts at some point. I didn't actually code these so when I took a quick look yesterday it was a struggle to figure out what was going on. I may go back to the older ChartDirector charts.

[MarkRBright]

Hi Bok, You are doing sterling work as ever on the stats, but in case you haven't spotted these I'll throw them into the pot.

. Clicking "User Milestones" on the main screen gives "The requested URL /userbycpidmilestones/fa1a611d7c2bb42d13617eb6b8903f8c was not found on this server." Don't know if it's just me, somewhat surprised nobody else appears to have mentioned it.
. The new RAC column on the Subprojects list appears to be a bit interesting. Most seem to have fairly random values such as my RAC of 375M for something i have 2M on, or an RAC of 75 for something I do around 100K a day on!
. User Timeline and Firsts! appears to no long er show anything for me at least - not sure if everything has been reinitialised or what. Personally I don't use this page but I just thought I'd point it out.

[Update]
. And of course it would be remiss of me not to request that the User Milestones be enhanced hugely but adding columns showing for each MM, the number required for a Maxwell, for a TeraMiRB and for a MiRB. THis would be teh same data for every single user, so it shouldn't be too difficult to collect - he says in his ignorance :-)
[EndUpdate]

As to the new format - I like it, particularly the clickable/sortable column heads.
Cheers and thanks

[Bok]

Hi Bok, You are doing sterling work as ever on the stats, but in case you haven't spotted these I'll throw them into the pot.

. Clicking "User Milestones" on the main screen gives "The requested URL /userbycpidmilestones/fa1a611d7c2bb42d13617eb6b8903f8c was not found on this server." Don't know if it's just me, somewhat surprised nobody else appears to have mentioned it.

Ah, it's not in the rewrite rules. Fixed, though that page needs converting to datatables, hence it's css and formatting is all off. I've made a note.



. The new RAC column on the Subprojects list appears to be a bit interesting. Most seem to have fairly random values such as my RAC of 375M for something i have 2M on, or an RAC of 75 for something I do around 100K a day on!

That column should not be visible. Switched it off..


. User Timeline and Firsts! appears to no long er show anything for me at least - not sure if everything has been reinitialised or what. Personally I don't use this page but I just thought I'd point it out.

Mine does show some things. Probably does not for too many people to be honest.




[Update]
. And of course it would be remiss of me not to request that the User Milestones be enhanced hugely but adding columns showing for each MM, the number required for a Maxwell, for a TeraMiRB and for a MiRB. THis would be teh same data for every single user, so it shouldn't be too difficult to collect - he says in his ignorance :-)
[EndUpdate]

As to the new format - I like it, particularly the clickable/sortable column heads.
Cheers and thanks

The MM's one is more difficult as we have discussed before. Once I get everythign fixed up and the graphs sorted too, I'll take another look.

[Bok]

Hi Bok, You are doing sterling work as ever on the stats, but in case you haven't spotted these I'll throw them into the pot.

. Clicking "User Milestones" on the main screen gives "The requested URL /userbycpidmilestones/fa1a611d7c2bb42d13617eb6b8903f8c was not found on this server." Don't know if it's just me, somewhat surprised nobody else appears to have mentioned it.

Ah, it's not in the rewrite rules. Fixed, though that page needs converting to datatables, hence it's css and formatting is all off. I've made a note.



. The new RAC column on the Subprojects list appears to be a bit interesting. Most seem to have fairly random values such as my RAC of 375M for something i have 2M on, or an RAC of 75 for something I do around 100K a day on!

That column should not be visible. Switched it off..


. User Timeline and Firsts! appears to no long er show anything for me at least - not sure if everything has been reinitialised or what. Personally I don't use this page but I just thought I'd point it out.

Mine does show some things. Probably does not for too many people to be honest.




[Update]
. And of course it would be remiss of me not to request that the User Milestones be enhanced hugely but adding columns showing for each MM, the number required for a Maxwell, for a TeraMiRB and for a MiRB. THis would be teh same data for every single user, so it shouldn't be too difficult to collect - he says in his ignorance :-)
[EndUpdate]

As to the new format - I like it, particularly the clickable/sortable column heads.
Cheers and thanks

The MM's one is more difficult as we have discussed before :) . Once I get everything fixed up and the graphs sorted too, I'll take another look...

[MarkRBright]

All good. Thanks Bok

[Dirk Broer]

Somewhere a table has crashed, one of the two stats servers keeps reporting the same value for 'combined score' over the day.

[mmonnin]

Both of the Goodyxgrid projects, NCI and CPU, reverted to some stats value that are really old. The NCI one is probably several years old. The sites have been down for some time.

Edit: I also have negative 18m for the day yet only -1mil for Goofy NCI for the day. I have the same credit for both. I might have had that at Goofy CPU for awhile. 56,521 is an impossible credit value at NCI since every task has been 10 credits.

[Cesium_133*]

Same here, Bok, as with mmonnin's issue. Got docked 3M+ points. If they don't come back... I might disengage from the Goofy project. It's a bit goofy in itself lol. I'll keep a record of the # of points I got hit for. If I do get rid of it, I won't come back, and I suppose I could petition you, good Bok, to add them back in? :)

[mmonnin]

Looks like the site came back up although there is no work yet. Maybe on the stats pull today Free-DC will correct itself.

[Bok]

I'm hoping Goofy will fix the exports on his side and it will correct here.

I'll take some copies of the backups from a few days ago just in case I need to restore them at some point.

[Bok]

I'm hoping Goofy will fix the exports on his side and it will correct here.

I'll take some copies of the backups from a few days ago just in case I need to restore them at some point.

[MarkRBright]

I noticed a big 3M+ hit today on Goofyxgrid too but also lost 30+ Squarepants, which when I looked at it, was down to the four Goofyxgrid subprojects disappearing from Free-DC stats. Is that expected? Will they come back when/if they start exporting again? They didn't go the last time the Goofyxgrid NCI stones value dropped to match the Goofyxgrid CPU stones value. Just curious becasue I haven't noticed inactive projects disappearing before.

[Bok]

Part of the GDPR changes is to delete a user and data if it's not in an export any longer.

they would return if and when the data is exported again.

[Dirk Broer]

Part of the GDPR changes is to delete a user and data if it's not in an export any longer.

If executed to the letter, it would create hell with the stats. Half of the projects I worked for no longer exists and their servers have since gone offline.

[JohnMD]

An export without users - or users with zero cobblestones - is "obviously" an error, and should be disregarded. Ask Boincstats...

[Dirk Broer]

An export without users - or users with zero cobblestones - is "obviously" an error, and should be disregarded. Ask Boincstats...

GoofyxGrid is exporting April 2018 stats files at the moment

[Dirk Broer]

GoofyxGrid is exporting April 2018 stats files at the moment

BOINCStats is starting to give scores with the new GoofyxGrid export stats, which seem up-to-date and have all the credit we've scored. Now hoping for correct sub-project values...

Update: GoofyxGrid credits are back for both projects, sub-project wise we're still waiting for Goofyx to solve this hurdle (and when he gets his thing done, let him publish what the problem was, then we can inform LHC@Home and LHC Dev@Home what it is they're doing wrong -they blame the BOINC software, which Goofyx may prove right).

[Dirk Broer]

GoofyxGrid NCI is exporting amongst other the following files
user.gz, user_work.gz and badge_user.gz

user.gz looks like


<user>
<id>user id</id>
<name>user name</name>
<country>country name</country>
<create_time>value</create_time>
<total_credit>value</total_credit>
<expavg_credit>value</expavg_credit>
<expavg_time>value</expavg_time>
<cpid>cpid</cpid>
<teamid>teamId</teamid>
</user>


user_work looks like


user_work.gz
<user>
<id>user id</id>
<subproject name="Monkeys_v1">
<workunits>value</workunits>
<credit>value</credit>
<expavg_credit>value</expavg_credit>
</subproject>
<subproject name="Monkeys_v2">
<workunits>value</workunits>
<credit>value</credit>
<expavg_credit>value</expavg_credit>
</subproject>
<subproject name="Monkeys_v3">
<workunits>value</workunits>
<credit>value</credit>
<expavg_credit>value</expavg_credit>
</subproject>
<subproject name="Monkeys_v4">
<workunits>value</workunits>
<credit>value</credit>
<expavg_credit>value</expavg_credit>
</subproject>
</user>


badge.gz is perhaps a problem due to the repetition of the same user id:


<badge_user>
<user_id>user id</user_id>
<badge_id>bad id</badge_id>
<create_time>value</create_time>
</badge_user>
<badge_user>
<user_id>user id</user_id>
<badge_id>bad id</badge_id>
<create_time>value</create_time>
</badge_user>
<badge_user>
<user_id>user id</user_id>
<badge_id>bad id</badge_id>
<create_time>value</create_time>
</badge_user>
<badge_user>
<user_id>user id</user_id>
<badge_id>bad id</badge_id>
<create_time>value</create_time>
</badge_user>
<badge_user>
<user_id>user id</user_id>
<badge_id>bad id</badge_id>
<create_time>value</create_time>
</badge_user>
<badge_user>
<user_id>user id</user_id>
<badge_id>bad id</badge_id>
<create_time>value</create_time>
</badge_user>


That's six badges: total, subprojects and RAC

[[H]Skillz]

I've noticed that the team page for a lot of projects do not show the last 7 or last 28 for the team. For example here:

https://stats.free-dc.org/team/gne/80

Last 7 and last 28 next to the team is blank.

[Bok]

I see that, though I'm unsure why. The sql certainly brings back the values ok. I will try to debug..

[Bok]

Fixed this.

[[H]Skillz]

Awesome work man!

[Dirk Broer]

Enigma@Home is suddenly gone from our scores. Their server is still up (http://www.enigmaathome.net/server_status ), but I can't reach their stats page (duh)

[Bok]

dang it, that's a bug for sure. It got a 'not found' page when downloading the tables.xml but still carried on processing - end result is it thought there were zero entries and with the new GDPR rules in effect it deleted everything out.

It should be restored soon and I've disabled enigma for now.

[MarkRBright]

Any progress on Enigma yet? I seem to be down a good million stones compared to BoincStats and Enigma for that matter. I am guessing the data is being correctly exported by Enigma otherwise BoincStats would be wrong, so I just wondered if you needed to re-enable Enigma.

[Bok]

I re-enabled enigma last night. showing now ?

[MarkRBright]

Sorry, I only just got notified that you responded! Yes it's fine thanks.

[Dirk Broer]

According to my mega milestones I have 93 projects, but as of today I only have credit for 92 (make that 91 now) -and a negative last update.
LHC@Home Dev is missing
Albert@Home missing too now

Edit: LHC Dev@Home is back

[Bok]

yes, I just restored user records from both of those projects from last nights backup. Combined totals will fix in next run.

I've switched off the projects from update for now as the stats have completely gone... and with the new changes for GDPR they get auto deleted.

[Bok]

yes, I just restored user records from both of those projects from last nights backup. Combined totals will fix in next run.

I've switched off the projects from update for now as the stats have completely gone... and with the new changes for GDPR they get auto deleted.

[Dirk Broer]

For some reason our totals for Citizen Science Grid keep growing, but our totals for the connected sub-projects -only one active at the moment, Wildlife- do not.
And though the project does not export badges_user.gz or user_work.gz Sebastien, of https://signature.statseb.fr manages to get that information on this page (https://signature.statseb.fr/next_badges.py). How does he do that?
How did FreeDC used to get the sub-project and badges information in the past?
The difference in reported credits for Wildlife between FreeDC (5,599,895) and Citizen Science Grid's account page for me (6,082,302) has grown to almost 500k by now.

[Bok]

I get the data from the user_work.gz file but it's not available any longer...

https://csgrid.org/csg/stats/

[Dirk Broer]

I get the data from the user_work.gz file but it's not available any longer...

https://csgrid.org/csg/stats/

I know, i've written them about it in the Citizen Science Grid forum (https://csgrid.org/csg/forum_thread.php?id=2551&postid=7646#7646). They seem deaf however.

[Marmot]

I get the data from the user_work.gz file but it's not available any longer...

https://csgrid.org/csg/stats/

Is this why there are no stats at all on CSG?

It's possible to get the stats, as another stats reporting site shows my RAC and totals over the last week; but I don't like using that site as much.

[Dirk Broer]

Is this why there are no stats at all on CSG?

It's possible to get the stats, as another stats reporting site shows my RAC and totals over the last week; but I don't like using that site as much.

It is not the totals or the RAC that is needed for subproject stats https://csgrid.org/csg/stats needs more data than it is showing now.

[Bok]

yes, that is actually causing it to fail to run, I hadn't realized it had not been updating at all.

I've took out the code for the subproject data for that and it should catch up.

[MarkRBright]

Just noticed that your original badges signature shows Goofyxgrid CPU but not Goofyxgrid NCI, and your new Badges test screen does the reverse. Thought it might be worth pointing out. Keep up the excellent work.

[SAHJ@H]

Hi Bok,
On the page User by CPID Stats, the link JoinDate Rank of the line Combined (https://stats.free-dc.org/userdatecomb/12130 for me) always leads to the same test table Userdatecomb Stats for Users joining on Wed, 31 December 1969. This date seems to be the calendar origin and the table does not change when you modify the link to https://stats.free-dc.org/userdatecomb/10000 for instance.

[Bok]

ah. missing entry in my rewrite rules. Should be fixed.

Though I need to convert that page to the new format

[SAHJ@H]

Fixed! Thanks for your fast answer.

[Dirk Broer]

The WCG Subprojects do not seem to update in the subprojects page

[Bok]

My bad, was adding a new id yesterday and ran it on a single user.

It's running now, should be updated momentarily

[Nflight]

Woke up to marbled confusion, I hope it is not just my CPID that is missing ?