Database problem?

[[H]Skillz]

Seems none of the stats are showing for me. It just shows a page without any of the stats numbers when I try to view my stats.

[Nflight]

You are, not the only one seeing nothing on the view screen. Seems the DB administrator is high overhead in an Airplane and will not change or reboot the system while the plane is in flight. It may be till Friday midday till the DB is returned to normal.
Please stand By

[vaughan]

I'm suffering from bok's stats withdrawal symptoms.

[Bok]

Yes, tried some remote debugging via FaceTime with my wife as I’m in Buffalo but it looks to be hardware related. Still trying.

[Nflight]

Speaking with Bok this morning here on the East Coast, it seems this is not a simple reboot. It may be a hardware problem, and thus till he returns and gets to look at the problem we are out of Stats.

[[H]Skillz]

I'm going to need to go to the Doctor to get a prescription for my withdrawals.

[step2000]

I emailed him and he has tried with aid of his bride to repair things but still no soap. Seems the site of late has been having more and more issues. Hope to see it return functional soon as BOK does have a nice site!

[Bok]

Finally got some time with my wife to look more into this. Unfortunately the database server has crashed hard and while it appears to boot I can’t connect so it must be sitting at a bios prompt. Not going to try and walk my wife through hooking up keyboard and monitor so it will need to wait till I’m home :(

[[H]Skillz]

Dang, I could drive over there and look at it. lol

On a more serious note. Any idea when you'll be able to physically look at it? My stats withdraws are at an all time high right now. LOL

[Nflight]

I picked up a part-time job to get my mindset off the statistics fandango, It didn't work all through work I thought of where I stood with everyone near me, how far I was away from people I am attempting to catch and then decided to scream as loud as I could at a brick. Since the brick did not listen and did not respond, I put the brick down and walked into the Bar nearby and ordered a double Crown Royal Vanilla on the rocks. After one I ordered a second and forgot why I came in there. On the steps coming down from the bar, I tripped over the brick and took two stitches, plus scraped my knee and elbow. I removed the bandages Saturday night, the wounds will continue to heal and the scrapes will be there for a long time. If my Stats worked this would not have happened. The End

[[H]Skillz]

OMG the stats being down almost killed you too???

[vaughan]

I have nervous anticipation for when Bok can fix the stats. Are we there yet?

[Bok]

OS drive is hosed. Database drives are all ok though. I have to rebuild it somehow which is going to take time.

[vaughan]

Not found error now when trying to reach the stats page. Bok must still be re-building the site. Do you have an ETA?

[Bok]

Was that on stats3? I'd forgotten to alter that one, I usually do a redirect on everything to a single page. I'd prefer to get rid of stats3 altogether but I know some people still use it. Anyway it was still trying to connect into the DB server which was causing some contention while I fix up things.

I've got about 50% of the perl scripts converted and working, including the main ones that download the data and the ones that push the data into MariaDB and do all the aggregation and such. Times seem pretty similar to before so I think I've got the DB parameters set correctly. I can tweak those as I go along.

At this point I *think* I'll be able to switch things back on again tomorrow. Of course the drives are hanging loose in the DB server right now, so I'll have to tidy that up probably over the weekend.

[[H]Skillz]

You're doing a great job, Bok.

[vaughan]

Bok, yes it was stats3 - I mainly use that as it shows more fine details than the new version. eg hover over computer and it shows info, bar highlights of my username in lists etc; much better.

[Bok]

All switched on again. There may still be some lingering issues.

Already had one issue, where it was updating one database, but the webside was still pointing to same database leading to some contention. That is fixed now.

I wasn't backing up some files like

1) the custom db_dump.xml files I use for projects that have subproject data - fixed I think.
2) my globalconfigs file!! aarghh, not much in it though so no harm
4) my cron entries - there may be some scripts not running due to this, hopefully I got them all (wcg badges and subproject data, wuprop hours data, generic badge updates)
5) mariadb conf file. I think this is recreated now, 2Gb tmpfs, plus all the sort and index settings I use. db is using 26Gb of Ram of the 32Gb on the server (it's max)

[[H]Skillz]

Are the stats currently "frozen" on the time it went down and will update as they normally would now? I am showing 0 for stats today when most projects would have already updated by now. I'm guessing when those scripts run at that time they'll update the information at that time.

[Bok]

Skillz;187536']Are the stats currently "frozen" on the time it went down and will update as they normally would now? I am showing 0 for stats today when most projects would have already updated by now. I'm guessing when those scripts run at that time they'll update the information at that time.

they'll work themselves out.

The way it works is I updated one database, while the website points to the other, then flip it. But because of one of the bugs above it was only ever updating one of the databases and never flipping. I didn't notice till I switched on the site and got all the contention problems.

You should start seeing some numbers soon.

[[H]Skillz]

You rock man. Thanks for all your hardwork.

[Dirk Broer]

I do not know if this is an issue, but the numbers haven't only started coming, I've seen them going too. I started with 34 projects at 1M+, and WEP M2 was at 999,244 at the next update giving me just 33 projects at 1M+. Looks like the two stats servers (let's call them flip and flop) are out of sync witch each other (and one of the two has a serious gap with the projects as well).

[Bok]

I do not know if this is an issue, but the numbers haven't only started coming, I've seen them going too. I started with 34 projects at 1M+, and WEP M2 was at 999,244 at the next update giving me just 33 projects at 1M+. Looks like the two stats servers (let's call them flip and flop) are out of sync witch each other (and one of the two has a serious gap with the projects as well).
So WEP is missing for you?

[Dirk Broer]

So WEP is missing for you?

Not missing, WEP was flipping to and fro round 1,000,000 credits yesterday -both servers not picking up DHEP's last update, which would give me yet another 1,000,000+ credits project.

[Bok]

Well that sounds exactly like the issue I mentioned above where one database was not being updated. Is it ok today?

[Dirk Broer]

It is much better, only DHEP (Distr. Hardware Evolution) still refuses to update

[Bok]

the db_dump.xml file is giving a 403 error

-bash-4.2$ wget http://dhep.ga/boinc/stats/db_dump.xml
--2018-12-07 14:04:34-- http://dhep.ga/boinc/stats/db_dump.xml
Resolving dhep.ga (dhep.ga)... 139.184.49.229
Connecting to dhep.ga (dhep.ga)|139.184.49.229|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.dhep.ga/boinc/stats/db_dump.xml [following]
--2018-12-07 14:04:34-- https://www.dhep.ga/boinc/stats/db_dump.xml
Resolving www.dhep.ga (www.dhep.ga)... 139.184.49.229
Connecting to www.dhep.ga (www.dhep.ga)|139.184.49.229|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-12-07 14:04:35 ERROR 403: Forbidden.


I've asked Michael to take a look. I can work around that if need be as it doesn't really change.

[Bok]

the db_dump.xml file is giving a 403 error

-bash-4.2$ wget http://dhep.ga/boinc/stats/db_dump.xml
--2018-12-07 14:04:34-- http://dhep.ga/boinc/stats/db_dump.xml
Resolving dhep.ga (dhep.ga)... 139.184.49.229
Connecting to dhep.ga (dhep.ga)|139.184.49.229|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.dhep.ga/boinc/stats/db_dump.xml [following]
--2018-12-07 14:04:34-- https://www.dhep.ga/boinc/stats/db_dump.xml
Resolving www.dhep.ga (www.dhep.ga)... 139.184.49.229
Connecting to www.dhep.ga (www.dhep.ga)|139.184.49.229|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-12-07 14:04:35 ERROR 403: Forbidden.


I've asked Michael to take a look. I can work around that if need be as it doesn't really change.

[mmonnin]

Not missing, WEP was flipping to and fro round 1,000,000 credits yesterday -both servers not picking up DHEP's last update, which would give me yet another 1,000,000+ credits project.

NFS is doing this too. An update with values for the day, then later everyone has zeros.

[mmonnin]

Multiple people with the same rank on E@H. Team Overclock.net. Team credit is out of order as well but its by the default sort.
https://imgur.com/a/bX03KSL

Other teams are also showing similar things.

[Bok]

I had a feeling this would happen...it's due to the new option at Einstein@Home where users have to opt-in to have their data exported for the 3rd party stats, they have just enabled it by the looks of it and I'm showing the #users in the xml has dropped from ~469,000 to just 2619!!!!


MariaDB [dcfree]> select active_this_run,count(*) from boinc_user where proj = 'eah' group by active_this_run;
+-----------------+----------+
| active_this_run | count(*) |
+-----------------+----------+
| 7269 | 1 |
| 7270 | 466126 |
| 7272 | 2619 |
+-----------------+----------+
3 rows in set (1 min 11.18 sec)

Luckily I've deferred the automatic deletion of users which we are supposed to do, which is why the projranks and such are messed up but this is a total mess....

[mmonnin]

Ah yeah. Makes sense. I had just went in and enabled stats export at LHC for this as well so I'm guessing stats will be garbled for that project at some point.