Website still acting funny...

[runestar]

Hey Brian,

The website is still acting wierd. For one thing, even though the stats are supposedly generated at ever half hour, its taking more than 10 minutes for the website to update itself, even on multiple refreshes. (For example: update stamp says 04:34. but website will not show new stats until 04:46.

Before, I could come in at the time, and within less than a minute according to my computer clock, the stats would be updated.

I went into the directory with all the links to the teams, and the timestamps would actually fall BACK when I refreshed the page, particularly for Parent Directory and the Team 0 links. I've been noticing this throughout the day yesterday.

Have you considered large mallet or a swift kick? Yeah I know, but it would at least make you feel a little better.

TTFN,

RuneStar½
PBB

[Brian the Fist]

The stats, for now, are computed every 1/2 hour on the hour, and only go 'online' 15 minutes later so that behaviour is intentional (i.e. you won't se ethe 1:30 stats until 1:45, etc).

As for the second part, if you suspect some pages are 'out-of-date' and change when you refresh the page, please post the EXACT URL of the page and I can fix it up. Thanks.

[runestar]

I'll see if I can get some screenshots next time it does that.

Best,

RS½

[runestar]

Okay, in this first shot, the directory listing says all the files have been updated (based on the timestamp)...

[runestar]

But here, you can see the page has not yet updated even though according to the directory all the files have been updated... (They ALL said 04:31 on the timestamp). NOTE, I didn't try clicking on the link, but hit the back button to get to the team page again and REFRESHED the page. I tried the refresh a few times and it wasn't until a few more minutes it gave the proper timestamp.

[Brian the Fist]

Right, well as I explained there's multiple web servers now. Every time the stats are computed they must re-sync. Thus you obviously viewed the pages during the sync-ing process so some servers were updated and some were not. They will never be off by more than 1/2 hour (or whatever the stats interval is) and as you say, only for a minute or so until they have time to copy the entire web site from the 'master'.

If anyone has any idea how they could all be made to sync simultaneously let me know but their clocks are already synchronized so I think the discrepancy is only due to the actual time required to copy over the web site.

If you DON'T refresh your browser, you wouldn't even notice except that the stats could be at worst case 45 minutes old (which will be less when I rewrite it and make it faster and more frequent again).

[MAD-ness]

Good work as always Howard.

I know it seems like we only come to with problems, but if we give you the wrong impression that would be a shame.

Over on the Ars forum and the Free-DC forum, I see it mentioned by members and visitors from other forums how refreshing the management and your attentiveness are.

Keep up the good work.

[runestar]

I started replying earlier but the browser locked up on me and I had to get going out the door.

Brian, so the directory listing is from the data server and not the web servers then?

Also, I was curious why the delay between the data server and the web server updating? I'm sure you have a good reason, I just haven't been enlightened yet. I guess we're just wondering about some of the changes with the new backend.

And we do appreciate the interaction. =) I think it gives the project that personal touch. Its the yeah, the sales system has its problems and quarks, but the salespeople are really great so we keep coming back. ;-)

RS½

[runestar]

Okay, the servers did that wierd time thing to me again tonight around 3:40EST. I was waiting for the individual stats to update. About 3:40 it listed a 3:30EST time. I went to our team page and that said 3:00EST so I clicked back to the stats which then read 3:00EST. I refreshed several times but I could not get it to come up with the 3:30 time again...

So like I said before, your servers are acting wierd...

BTW, you never answered what the 15 minute delay is for? If the stats are ready, why do you need a delay to post them??

Best,

RuneStar½
PBB

[runestar]

Either one of the servers is off or you got the data server in there with the web servers.

I went to our team page and I was ABLE TO DUPLICATE this wierd bug on your website. It involved some power refreshing. But in a span about 30-40 seconds I was able to have it do it about 3-4 times with the newer data (and the data was newer than the supposed current and older one). I'm not sure if this is more prominent the closer to your 15 minute delay time or not though. The previous time it was only 10mins along since the supposed stats update on the data server.

RS½

[Brian the Fist]

please read my explanation above carefully.

[bubbadog]

Originally posted by MAD-ness
Good work as always Howard.

I know it seems like we only come to with problems, but if we give you the wrong impression that would be a shame.

Over on the Ars forum and the Free-DC forum, I see it mentioned by members and visitors from other forums how refreshing the management and your attentiveness are.

Keep up the good work.

I can only echo what MAD-ness has said. The same things are said in the Team 3DHQ forum at Ninjamicros, and I suspect, in the forum of just about every team involved in this project.

[runestar]

Originally posted by Brian the Fist
please read my explanation above carefully.

Well, if the servers are all synced by clock then they should all copy at the exact same time? Now unless the database is locked while copying, although I'm not exactly sure why since that database wouldn't be updated till another half hour, why would there be that much difference between the various web servers in general?

<Thinks> are the web servers pulling the data to them, or is the data server pushing the information to them? If the data server is the scheduler, then its possible instead of simultaneously uploading to all the web servers its doing it one at a time.


Also, as far as I can tell, you never answered the question about the 15 minute delay, which I did ask about twice. If you answered it or answered it elsewhere, please point it out and I'll stand corrected. I just don't see what's different between the old servers and the new servers that there needs to be a delay that long. It would seem it would actually be the opposite...

Best,

RS½

[Brian the Fist]

The 15 minute delay is to compute the stats. It is slower now because I have to rewrite it. I mentioned this elsewhere (search for a message with the text 'SQL' in it)

Contrary to popular belief, data transfers are not instant. Copying a 100MB web site to a bunch of machines over Ethernet takes a minute or two.
The majority of the bulk is in the user progress GIFs (on your login page) which were previously generated on the fly but now must be precomputed to ensure they get to all the servers.

If there's no major objections, I can just replace these (the GIFs showing your personal progress on the login page) with plain text and the web sites will sync a lot faster, and with less load overall. Let me know if anyone has a problem with such a change.

[wirthi]

Hi,

if we'r talking about the same pictures ("your best scoring structure", "overall best scoring structure", ...), then you could exchange all the generated pics with this method:

make one image (size 1x1 pixel) and scale it to the desired size using the html img tag. So, if the best protein of a user has an energy level of 60, make an <img ... width="60">.

You can put that image in the background of a table (i believe), so you could still write the energy level in front of the pic (as it is done right now). Then you only need two jpg/gif files, each 1x1 pixel: one for the red color, one for the cyan ("number of strucures you have contributed", ...)

Compare: Work submission history at distributed.net http://stats.distributed.net/partici...id=5&id=314897

[Halon50]

Text would be great. Loading the stats webpage would run a lot faster, and I'm sure things would be a lot speedier on the server end as well.

I always wondered why you bothered with such an obviously slow method of showing stats on the webpage. It looks nice, I suppose, but a text bar would work just as well.

Thanks for your hard work, Howard/BtF!

[runestar]

Originally posted by Brian the Fist
The 15 minute delay is to compute the stats. It is slower now because I have to rewrite it. I mentioned this elsewhere (search for a message with the text 'SQL' in it)

Contrary to popular belief, data transfers are not instant. Copying a 100MB web site to a bunch of machines over Ethernet takes a minute or two.
The majority of the bulk is in the user progress GIFs (on your login page) which were previously generated on the fly but now must be precomputed to ensure they get to all the servers.

If there's no major objections, I can just replace these (the GIFs showing your personal progress on the login page) with plain text and the web sites will sync a lot faster, and with less load overall. Let me know if anyone has a problem with such a change.

I didn't even think about the GIFs needing generation, DUH! How many web servers do you got going anyways?

The GIFs are nice, but I'd rather have the text/HTML in place personally. Halon makes a good point that would speed things up not having to worry about the graphics. Once the stats are cooked for the current period, all you have to is have the web servers pick it up/recieve it. Plus text compresses better than a GIF...heh

Have you considered compressing (such as ZIP) and then having the web servers pick it up and decompress it on themselves? This would reduce the time it takes to transfer. It would seem this would give you a better compression level than any network protocal, plus, there's no ramping up to speed for the compression to take full effect. Now you're not necessarily looking for the BEST compression, but what is speedy. For example WinRAR gets better compression than WinZip, but WinZip is a heck of a lot faster than WinRAR.

It might be worthwhile to test on some internal computers to simulate the actual process and see how it works out with different archivers and compression options.


ALSO BTW, would it be possible to have a link generated for the team you are on so you don't have to go up to the menu or your favroites/bookmarks to see your team stats? In fact, right where it says the:
Number of structures
your team has contributed
(as of a few minutes ago)

Either the word "team" as a link (maybe written as "TEAM"), or the whole line as a link. IMHO, "TEAM" as a link would look better. This would make in more in line like the other D.C. projects in convenience. =) I always wondered why you didn't include a link when you mention the # of structures for the user's team...

TTFN,

RuneStar½

[dtsang]

Originally posted by runestar1/2

Have you considered compressing (such as ZIP) and then having the web servers pick it up and decompress it on themselves? This would reduce the time it takes to transfer. It would seem this would give you a better compression level than any network protocal, plus, there's no ramping up to speed for the compression to take full effect. Now you're not necessarily looking for the BEST compression, but what is speedy. For example WinRAR gets better compression than WinZip, but WinZip is a heck of a lot faster than WinRAR.

The time it takes to compress a GIF and decompress it will result in a loss of performance, simply because it is faster to transfer 100 MB of data over Ethernet than to compress it to 50 MB (which takes more than a few seconds) then transfer it. Besides, GIFs can hardly be compressed further; they are quite compressed already!

I would fully support using text instead of GIFs to represent our numbers.

[bwkaz]

Yeah, text is fine with me, too. Plus, there are issues (in the U.S. anyway, not sure about Canada) with GIFs and Unisys and their LZW compression, so I'm sort of against using them anyway (I figure it's probably better to get biases out in the open... ).

The only thing GIFs can do that PNGs can't (well, that is, assuming full browser alpha support) is animation -- or is that even correct? Can PNGs do animation?

The problem, of course, is that some of the more popular browsers (*cough* IE *cough*) don't support PNG alpha.

But this is all a moot point anyway, because using PNGs would probably take just as long. Just use text.

[Brian the Fist]

I will do something about this in the near future then. I think I like wirthi's idea (which we've actually already used elsewhere, so I feel dumb now not having thought of it) so Ill try that out and it should be much faster.
May be a few days til I get around to it though.

[runestar]

Well, if you simply copy the GIF into the compression file with no compression on it... remember that this would be all automated, so everything that is .txt and .html would be compressed and we would simply store the GIF.


What, no JPEG? We can crank up the compression to 75%... at that point, everybody's graphics should look the same. =)

TTFN,

RS½

[runestar]

I was just thinking, brian the fist, why not just make the page text information (i.e. no stat GIFs), and then if the person prefers the graphics, they could could click on a link for the graphical version, and it would be dynamically generated like before?

RS½

[pointwood]

I think Howard should concentrate on the important stuff == the project and making sure the client and the server backend runs smoothly. The stats are fine IMHO. There is a pure text version of the stats if you like to see that instead of graphics. The only thing that would be nice, is to be able to uniquely identify each user. That isn't possible right now and it 's a problem for the other stats engines. I believe the reason Howard doesn't make that possible, is privacy issues and I respect and can live with that. I do believe it should be possible to create something that made it possible, but it's not that big a problem.

[runestar]

Actually there's no such identifier for SETI either. I don't remember about F@H though whatever they are doing with the stats system these days...

I suppose it would be possible to brute force a hack into the various DF individual accounts... they wouldn't be able to change passwords, although e-mail addresses could be changed... although as long as the owner has the true name for the account, that wouldn't necessarily a problem. They could cause some chaos by redirecting accounts to other teams...

Actually, there is no way to see your OWN stats in text. You can see the Top 10, and if you are on a team, you can see your name within the team... but there is no text for just your own stats.

RS½