Yesterday evening I was trying to find the cause of a problem that was
reported by animal on behalf of gistech1978.
http://www.chessbrain.net/bbs/viewtopic.php?t=541
Gistech1978 was climbing steadily in the rankings until the 19th when
his stats slipped from 273 to 2. I've seen this problem before on
rare occasions with other members.
Before continuing let me first explain how stats are logged. While
the SuperNode server is running it records all PeerNode client
transactions in a memory hashmap. Every five minutes the contents of
the hashmap is flushed to a disk file. This approach is considerably
faster than a database access and has the advantage that the resulting
disk files can be easily scanned and ultimately imported into a
database. This was important when the SuperNode was running on our
old PIII-1GHz machine with only 256MB. Now that we're using a P4
3GHz sever with 768MB its clearly less of an issue! However we're
still in the process of making the transition to a new stats system.
With our current approach, stats data is recorded based on game
sessions. This allows us to see exactly what happens during a single
game. In fact this information is stored in a game database.
A problem arises when the hashmap is being flushed to disk and the
server crashes for some other reason. The SuperNode server is a
heavily multithreaded application and so any number of things may be
happening while stats are being flushed. So it's not the flushing
of stats that causes a crash - its any other thing that may happen
on the server. Keep in mind that this is all relatively rare. The
SuperNode is quite stable at this point.
During the past six months, the times when the server has crashed, the
stats file has been incompletely written. This is a problem because
the same file is used by the server when it restarts. When the
SuperNode restarts it loads an incomplete user list into memory. This
causes users who connect and are not in the hashmap to receive new
records.
IMPORTANT: This is a reset in their stats for that session only!
When the stat session file is later imported into the database there
is a discrepancy for the users that were affected (that is the users
data which was not written into the incomplete stats files).
This happened often during the start of stage two, and less often
over time. What I did was rebuild the database by scanning all prior
stat data files and compensating for when the server crashed by adding
the total stats for that session to the users last recorded data.
The data is real and represents actual recorded transactions - the
problem was entirely in the program that scans the stats files and
imports into the database! I fixed the program so that it correctly
tallies user stats and so some users
The important thing to keep in mind is that the stats generated for
each user is stored in files so we can recreate a database in any form
and track problems over the cause of a project phase.
So I realized that if I "fixed" the import problem all users
will get their fair share of corrections and so this was the honest
thing to do. However, there was no way to avoid the disappointment of
contributors who thought that they were a bit further along and
slipped when another competitor moved up the ranks because of the
correction. It's a difficult decision to make however; I felt that
it's best to choose honesty - no matter what the consequences.
I suspect that other (larger) DC projects would probably hide this
sort of thing for fear of losing contributors. So I wonder how
contributors really know when their stats results are real..?
Perhaps the answer is open source stats J certainly open source in
general. That was part of my reason for openly releasing stats2.txt,
stats3.txt and the new rtstats.txt - so that problems could be
quickly identified and the results open to public scrutiny.
Reply With Quote