PDA

View Full Version : Forum and stats server move



Bok
10-05-2010, 11:10 AM
I'm going to start the last part of the process of moving the stats and forum over to their new homes today/tomorrow. Hopefully it will be fairly smooth and not much noticeable difference but it's possible things might get missed so let me know.

I'll have to switch off the dnet proxy temporarily whilst I move it's contents. Correspondingly, ppstats will switch off and will be the last thing to switch back on.

The stats have been running parallel on the new server for a few months now. It's quite a bit faster given it's a core i7 and the database tables themselves are housed on solid state drives. The combined BOINC stats are therefore calculated in real time rather than on a slight delay as they currently are. What is missing is the weekly/monthly data. I'll be copying that over in the coming days given it's an 80Gb transfer..

The bandwidth up is lower but we'll just have to see if it handles the load ok, I can't tell right now. If it doesn't I'll look into getting that increased on a business line which will also allow for a static IP, but the costs are fairly high there so I'm trying not to do so unless really necessary..

I'll be switching DNS on the stats site prior to the forum which will mean it will not think you are 'logged in' on the stats until the forum is switched too. Therefore statstool should not really be used until completed...

Bok

vaughan
10-05-2010, 08:43 PM
I'm going to start the last part of the process of moving the stats and forum over to their new homes today/tomorrow. Hopefully it will be fairly smooth and not much noticeable difference but it's possible things might get missed so let me know.

...

I'll be switching DNS on the stats site prior to the forum which will mean it will not think you are 'logged in' on the stats until the forum is switched too. Therefore statstool should not really be used until completed...

Bok
Yes that is the case - logged in automatically on the Forum but the Stats pages show I'm not logged in. Is there a way to log in on the Stats pages separately from the Forum?

Bok
10-05-2010, 09:53 PM
It's something I'm going to have to investigate. Likely a 'clear the cookie' issue. Will sort it out tomorrow though. Spent way too much time on all of this today.. :)

Bok
10-05-2010, 10:36 PM
Should be fixed now...

Now only thing I have a problem with is :

my firewall does not support putting through custom ports to NAT, so I can't forward port 2064 for the dnetproxy.. it's running on port 23 for now though as it always has in addition.

vaughan
10-07-2010, 07:24 AM
bok are you still working on the stats server? I ask because I'm finding the stats pages extremely slow to load, lots of "waiting for stats.free-dc.org" messages

Bok
10-07-2010, 03:09 PM
Connection is not as good as it used to be. IB was paying quite a lot for the old one and I haven't got that kind of spare change to throw around right now. $100-150+ per month extra likely..

Just updated the firmware on my firewall to see if it makes a difference but it hasn't. It's just the amount of traffic coming in when I only have 768K up (15Mbps down). I'd ideally need to upgrade to a business connection and get the 5Mbps up.

gopher_yarrowzoo
10-08-2010, 05:02 AM
Hmm I suppose it would depends on what uses the most bandwidth I'm suspecting the stats pity you couldn't have put that somewhere else with a better connection and made sure you could get to it if need be.

Bok
10-08-2010, 08:05 AM
The forum takes little to no resources at all really. Stats uses up 99% of the bandwidth and requires the most use of the hardware too....

LAURENU2
10-12-2010, 01:25 AM
Well all we have to do is fine a few more people like me
That are willing to put out some money every month like I do
It is Like DC-ing DC-ers share there resources to help the World
Well giving your resources to Free-DC so they can help the world does the same
I Pledge $50.00 a month I have set it up with my bank Auto-pay to pay once a month:thumbs:

So come on Gay/Gals Look into your Harts and your pockets
And make a pledge to help the Free-DC world:Pokes:
If you have the Bucks to DC You have the Buck to HELP

Bok
10-12-2010, 08:10 AM
Thanks Lauren, I do appreciate that.

I've been spending time trying to figure out where the problems are. Found and blocked one IP address which seems to have been trying to get the same file every second or so in bursts, was taking approximately 1/3 of available bandwidth usage. Next I may replace my standard cable modem and also the router I have in place, I just wish I had a good way to check where the problem is occuring. Running ntop on the webserver shows that it can handle the traffic with ease.

Johnny Cool
10-12-2010, 02:21 PM
Where is that PayPal link here? I saw it when I was not logged in, now I cannot see it. I don't have that much, but I guess that it is 'BTN' (Better Then Nothing). :o

Johnny Cool
10-12-2010, 02:32 PM
Where is that PayPal link here? I saw it when I was not logged in, now I cannot see it. I don't have that much, but I guess that it is 'BTN' (Better Then Nothing). :o

Sorry — your last action could not be completed

If you were making a purchase or sending money, we recommend that you check both your PayPal account and your email for a transaction confirmation after 30 minutes.

If you came to this page from another website, please return to that site (don't use your browser's Back button) and restart your activity.

If you came from PayPal's website, click the PayPal logo in the upper-left corner to return to our home page and restart your activity. You might have to log in again.

Hmmmm...

ChertseyAl
10-12-2010, 03:05 PM
So come on Gay/Gals Look into your Harts and your pockets

I just threw a few dollars in. Not many, as I don't know how big a dollar is, being from the colonies ;)

Still not entirely sure why I should look into a male deer (hart), but my *heart* suggested I pay a little ;)

Al.

Bok
10-12-2010, 03:29 PM
Not sure why the paypal link does not work. A few others have used it recently ok??

Thanks Al, got that.

Let me give you all an update

Anyways I'm almost stuck as to what the problems are right now. I do see a lot of traffic so it must be mostly working but there is obviously some bottleneck. I wish I knew someone with more networking experience than I have to help out so if anyone knows please get in touch.

I've just replaced my switch with a $100 netgear gigabit version and it's not made any difference at all that I can tell.

This is the setup right now.

I'm on a 15MBps down/768K up connection. Comes into to a splitter using RG6 which splits the signal into 4 components. One goes to my Motorola SB5100 cable modem, one for voip and the other 2 for cable tv.
From the cable modem it goes into the Netgear FVG318 firewall, from there to the gigabit switch and then to the servers.

From either side of the switch if I ping say 8.8.8.8 I can get anything from 23ms to 1500ms.

Three places I could attempt to alter.

1. Firewall - it should be capable of handling the bandwidth.

2. Cable Modem - I could buy a newer one capable of docsis 3 (mine is docsis 2 )

3. I could try to alter the initial splitter to split it into 2, once going to the cable modem and the second one then splitting out for cable tv and voip.

I mention 3 as I managed to log into the cable modem and the downstream power level seemed a little high

<center style="font-family: 'Times New Roman'; font-size: medium;">
<table align="center" border="1" cellpadding="8" cellspacing="0"><tbody><tr><td style="margin: 0px; font-family: arial,sans-serif;">Frequency</td><td style="margin: 0px; font-family: arial,sans-serif;">609000000 Hz Locked</td></tr><tr><td style="margin: 0px; font-family: arial,sans-serif;">Signal to Noise Ratio</td><td style="margin: 0px; font-family: arial,sans-serif;">35 dB</td></tr><tr><td style="margin: 0px; font-family: arial,sans-serif;">Power Level</td><td style="margin: 0px; font-family: arial,sans-serif;">-10 dBmV<table border="0" cellpadding="0" cellspacing="0" width="300"><tbody><tr><td style="margin: 0px; font-family: arial,sans-serif;" align="LEFT"><small>The Downstream Power Level reading is a snapshot taken at the time this page was requested. Please Reload/Refresh this Page for a new reading</small></td></tr></tbody></table></td></tr></tbody></table></center><center style="font-family: 'Times New Roman'; font-size: medium;"><table style="text-align: left;" align="center" border="1" cellpadding="8" cellspacing="0"><tbody><tr><th bgcolor="#736b08">Upstream</th><th bgcolor="#736b08">Value</th></tr><tr><td style="margin: 0px; font-family: arial,sans-serif;">Channel ID</td><td style="margin: 0px; font-family: arial,sans-serif;">7</td></tr><tr><td style="margin: 0px; font-family: arial,sans-serif;">Frequency</td><td style="margin: 0px; font-family: arial,sans-serif;">33000000 Hz Ranged</td></tr><tr></tr><tr><td style="margin: 0px; font-family: arial,sans-serif;">Power Level</td><td style="margin: 0px; font-family: arial,sans-serif;">42 dBmV</td></tr></tbody></table></center>
I have a nagging doubt in my mind that DNS requests are somehow causing the problems though. I've been messing with nameservers locally for the last few days and even if I have one on the same machine I'm 50% of the time getting timeouts.

I'm really not sure where to go next and would really appreciate any help people can think of. I did try getting in touch with Dyyryath (old free-dc guru) but he's not responding..

Bok

Johnny Cool
10-12-2010, 03:50 PM
Sorry — your last action could not be completed

If you were making a purchase or sending money, we recommend that you check both your PayPal account and your email for a transaction confirmation after 30 minutes.

If you came to this page from another website, please return to that site (don't use your browser's Back button) and restart your activity.

If you came from PayPal's website, click the PayPal logo in the upper-left corner to return to our home page and restart your activity. You might have to log in again.

Hmmmm...

It went through fine (Terry from Colorado Springs).

LAURENU2
10-13-2010, 09:12 AM
It went through fine (Terry from Colorado Springs).
Your the Man Mr COOL:thumbs:

Bok
10-13-2010, 02:39 PM
I've built a smoothwall firewall from an old P4 server I had, been trying to get it configured which is why the site was down but it looks like one of the nics is bad. I'm going to have to buy a new nic for it as I don't seem to have any others in my 'stash'.

I'll probably have to sever the connection again sporadically whilst I get it up and running. This will replace the Netgear Firewall I have which at the moment is at the top of my list for the case of the bandwidth saturation.

ChertseyAl
10-13-2010, 03:02 PM
Mucho snippage, but ...


I'm on a 15MBps down/768K up connection. Comes into to a splitter using RG6 which splits the signal into 4 components. One goes to my Motorola SB5100 cable modem, one for voip and the other 2 for cable tv. From either side of the switch if I ping say 8.8.8.8 I can get anything from 23ms to 1500ms. 2. Cable Modem - I could buy a newer one capable of docsis 3 (mine is docsis 2 ) 3. I could try to alter the initial splitter to split it into 2, once going to the cable modem and the second one then splitting out for cable tv and voip. I mention 3 as I managed to log into the cable modem and the downstream power level seemed a little high. I have a nagging doubt in my mind that DNS requests are somehow causing the problems though. I've been messing with nameservers locally for the last few days and even if I have one on the same machine I'm 50% of the time getting timeouts.

Taken me all day to get to post this. Site is V slow. Actually F slow ;) Mostly timeouts today :( So this is very terse as I've got a feeling I won't get to post it ...

A few random thoughts:

Only 768k upstream? Seems slow. My home use bargain basement cable connection is 1m for only 10m downstream.

From my days working with the UK cable supplier, a four way splitter was a no-no. Three ways max. And that need much tweaking in the street cab to get the levels right.

SB5100 - Isn't that kinda outdated now? I don't think we use them over here any more. Hasn't your ISP given you something better?

Pings to 8.8.8.8 taking 1.5s seems odd. Hardware problem?

Power level - Not familiar with your network levels (only ever done satellite for US clients). 4-way split is going to be 14dB down. Seems odd to have too high level.

DNS - I don't think that's the problem. Download my 'big' graphic is slower than dialup. It's a bandwidth problem.

My guess is failing hardware. But then if your upstream is overloaded I think of malware, but you're not likely to be daft enough to not notice that!

Al.

Bok
10-13-2010, 08:07 PM
Appreciate the comments!

I think the UK are ahead of the US in cable connections. I can't get better than 768K up without getting what's termed a business connection and those are not cheap..


7 Mbps down, 786 Kbps up - $97.95/mo.


10 Mbps down, 1 Mbps up - $167.95/mo.


12 Mbps down, 1.5 Mbps up - $279.95/mo.


15 Mbps down, 2 Mbps up - $389.95/mo.

I've got the two way splitter so I can try that part tomorrow I think.

I've had that SB5100 for quite some time so they may be giving out better ones these days. It's the next thing to try anyway even if I buy my own.

If I switch off the http redirect on the firewall pings are a lot better but still I don't think the connection is saturated as such. Get to that in a moment.

I've got the smoothwall server running now in place of the netgear firewall so we'll see if that makes a difference. I'm not going to tweak anything again till tomorrow. Let me know if it appears better or not. It does have much better diagnostics and is currently showing a snapshot of

malware - it's possible but pretty unlikely - I do keep on top of that kind of thing, but I'll certainly check it out. I did see a few kb going to eDonkey when I installed ntop on the webserver but I think that was a red herring. It's a linux box and I never use p2p clients.

I did tweak apache a little as I was very concerned about reverse dns lookups. Found out that the standard logging format includes %h which can cause hostname lookups regardless of the settings and switched that to %a but I don't think it made a difference.

192.1 Kbit/s / 296.4 Kbit/s (Out / In) which is not too bad.

Just been watching the logs and someone has been trying to do some sql injection attacks from China.. Sneaky but it won't work. Banned the ip - I'm going to get more aggressive on doing this kind of thing.

So : next steps if needed.

1. redo rg6 splitter
2. new cable modem
3. give up unless I win the lottery tonight

There is a very real possibility that I'm just not going to be able to host the stats any longer but we aren't there yet. Forum and the other websites I run are not an issue.

Bok

Bok
10-14-2010, 09:13 AM
Is it any better? Certainly seems a lot better from within my network. Smoothwall is running quite nicely. Still got some challenges but I'd like feedback first..

gopher_yarrowzoo
10-14-2010, 10:06 AM
Seemed quick enough from here when it loaded a bit faster than that last time I looked, that's for sure. I'll work on the Paypal link on the front page / stat server see what's up with it if you like.

LAURENU2
10-14-2010, 12:01 PM
Is it any better? Certainly seems a lot better from within my network. Smoothwall is running quite nicely. Still got some challenges but I'd like feedback first..
Yes Faster Stats are loading in 14 sec. much faster then the 208 sec it took before
forum is only 4 sec NOW down from the 18 sec it took
it looks a LOT better:thumbs:
Your The Man Bok
:allhail:

ChertseyAl
10-14-2010, 01:41 PM
A lot quicker today. Have been able to get to this thread easily each time I tried throughout the day. My big project stats graphic is still pretty slow to download, but quicker than yesterday.

Overall, things are much better :)

Al.

Bok
10-14-2010, 02:12 PM
I think it's going to be ok then. I am seeing a lot of bandwidth activity but it's still not maxing out. So, perhaps now it's time to stat some bandwidth saving issues instead. I think I've spent a little under $1000 in the recent weeks on hardware, that part can take a rest for now, donations have definitely helped :)

Taking a look at the w3compiler though, free trial, we'll see what it does..

Bok
10-17-2010, 01:24 PM
So, is everything still looking good ? No issues with speed at all?

As of about 30mins ago I've finally figured out why my cacheControl wasn't working so all of the javascript files and static images should now be cached locally. This will hopefully speed up the pages quite a bit more I think.. Try it and see.

I'm also considering making the default on the pages to NOT include the graphs but instead have a button to show them if required. This will just be a parameter in the url, so easily accessible still. The graphs take up about 80% of the size of some of the pages so it will drop the footprint of most pages to only a few kb.

Any thoughts?

LAURENU2
10-17-2010, 09:53 PM
Graphs off to the side is OK with me
Some of the graphs/charts were hard for me to read
But That maybe just Me :bang:
Speed is vary GOOD now:thumbs:
_____Thank You_____
>>>Bok<<<
:allhail:

gopher_yarrowzoo
10-18-2010, 09:19 AM
Graphs off to the side is OK with me
Some of the graphs/charts were hard for me to read
But That maybe just Me :bang:
Speed is vary GOOD now:thumbs:
_____Thank You_____
>>>Bok<<<
:allhail:

Yes good plan I guess I should maybe go warm up some nice swishy code to do a nice reveal without having to reload the page :D

Additional also making all the tags STATIC automagically would that not help with bandwidth so it's not on having to do a call for each tag as they are cached too?

Bok
10-18-2010, 12:00 PM
No, making them static would not change the bandwidth as the end result is still the same size. It would save on db calls and such, but that's not an issue. They are cached locally on the server anyway.

For the 'reveal' function you are thinking about, what we have to ensure is that the data is not built at all until the reveal is done.

Say the fullpage with graphs and all is 20k of html + 100k of graphs, we want initially to only send that 20k. If the user hits reveal then the extra data for the graphs is transmitted. I guess what I'm geting at is not to just hide it via javascript. We are aiming to get the pages to load much faster from the client side.

this (http://www.webpagetest.org) is actually a real nice resource for testing webpages with lots of ideas...

I'm actually not worried about bandwidth as such as long as it's not affecting speed, really I'm just trying to make the client experience that much better. With the caching of the javascript and css files and saving of graphs when not needed the page load time should be a few seconds at most.

gopher_yarrowzoo
10-18-2010, 05:20 PM
Nope was never thinking of having the graphs hidden but called upon when asked would need to grab the variables for them from someplace but that is doable :D

LAURENU2
10-23-2010, 09:35 PM
:rant:
Ok Who changed my 3 crowns in the stats from Gold to Pink and why:hair:
I have a hard enough time with the way my Mother spelled my name:wife:
With 2 Pink crowns I am sure people will think I am a Gal:sack:

:rant:

Bok
10-23-2010, 09:51 PM
Where is it showing as pink? They should not have changed at all..

LAURENU2
10-23-2010, 10:27 PM
hahaha I'm sorry they are gold Except when you High lite that row
And I had it highlighted ha ha ha :bang:
http://stats.free-dc.org/stats.php?page=users&proj=mil

gopher_yarrowzoo
10-24-2010, 03:21 PM
hahaha I'm sorry they are gold Except when you High lite that row
And I had it highlighted ha ha ha :bang:
http://stats.free-dc.org/stats.php?page=users&proj=mil
Lauren check your monitor, your seeing things, looks fine on my screen :D

LAURENU2
10-25-2010, 02:28 AM
Lauren check your monitor, your seeing things, looks fine on my screen :D
:umm:My monitor is just OK. I can see you just fine :trash:
:punpolice:Go comb your hair it looks like you had your head tucked under your pillow all night:harhar:

LAURENU2
10-25-2010, 08:30 PM
OK since the Move on the main stat page http://stats.free-dc.org/stats.php?page=index
we have 2 Eon's listed
#1 http://stats6.free-dc.org/stats.php?page=proj&proj=eon
#2 http://stats6.free-dc.org/stats.php?page=proj&proj=eon2
And Both are wrong. The right link is
http://stats.free-dc.org/stats.php?page=proj&proj=eon2

We Need to take the ( 6 ) out of the link to make it right

And The old Eon is dead and should be laid to rest

Paratima
10-25-2010, 10:12 PM
I agree with Lauren on this one. Graeme and his crew now have their own forum at U. Texas.
We no longer need to preserve the EON forum space. Guess we could put up a linky, but even that seems excessive. Anyone wants to find him should be able to. And the stats for EON[0] are truly only historical. It wouldn't break my heart to reclaim that space. Just my .02 worth.

Bok
10-25-2010, 10:54 PM
OK since the Move on the main stat page http://stats.free-dc.org/stats.php?page=index
we have 2 Eon's listed
#1 http://stats6.free-dc.org/stats.php?page=proj&proj=eon
#2 http://stats6.free-dc.org/stats.php?page=proj&proj=eon2
And Both are wrong. The right link is
http://stats.free-dc.org/stats.php?page=proj&proj=eon2

We Need to take the ( 6 ) out of the link to make it right

And The old Eon is dead and should be laid to rest

It doesn't link to stats6 for me? All of the stats pages have dynamic links to whichever versions they are on, basically using a variable of $root_page.

Can you check again at http://stats.free-dc.org/stats.php?page=index

for the two Eon projects and see where they link to?

We can put the old Eon into Archive then - Done.

I'll switch off the scripts which do the updates tomorrow. Sad day really, I remember writing those scripts whilst in a hotel room in Sao Paulo, Brazil...

LAURENU2
10-26-2010, 02:05 AM
OK I found where the problem lies
It is with the BOINC link to Free-DC main stat
that they list in all projects Statistics and leaderboards
http://milkyway.cs.rpi.edu/milkyway/stats.php

There link brings you here http://stats6.free-dc.org/stats.php?page=index

But the right link should be http://stats.free-dc.org/stats.php?page=index

Bok you need to contact Boinc and have them change there link
Or redirect all your file links
If you do not see what I am saying go to any Boinc project and click on there
Statistics link Click on the link to Free-DC stats
and you will see it leads you to a OLD stat page
http://stats6.free-dc.org/stats.php?page=index

Bok
10-26-2010, 07:13 AM
ah, no that's me. It links to http://stats.free-dc.org/ which then redirects to stats6. It's an apache vhost config issue. I'll fix.

gopher_yarrowzoo
10-26-2010, 10:21 AM
OK I found where the problem lies
It is with the BOINC link to Free-DC main stat
that they list in all projects Statistics and leaderboards
http://milkyway.cs.rpi.edu/milkyway/stats.php

There link brings you here http://stats6.free-dc.org/stats.php?page=index

But the right link should be http://stats.free-dc.org/stats.php?page=index

Bok you need to contact Boinc and have them change there link
Or redirect all your file links
If you do not see what I am saying go to any Boinc project and click on there
Statistics link Click on the link to Free-DC stats
and you will see it leads you to a OLD stat page
http://stats6.free-dc.org/stats.php?page=index
Hmm Lauren, I just looked at that page I it directly linked me to stats.free-dc.org which is correct and also what it shows on the mouse over on my status bar.

Bok
10-26-2010, 11:18 AM
I fixed it this morning..

LAURENU2
10-26-2010, 05:13 PM
Hmm Lauren, I just looked at that page I it directly linked me to stats.free-dc.org which is correct and also what it shows on the mouse over on my status bar.

Ha Ha you are a little slow gopher :couchsleep:
Bok Fixed the misdirect by the time you replied :hair:

gopher_yarrowzoo
10-27-2010, 06:10 AM
Yeah so I see, Oh come on 5 hrs time difference add to that I don't read the forum as often as I did... I know I'll just stick to fixing stats bugs.

outlnder
10-27-2010, 10:11 PM
Ah Goph, we still love you. No pouting.

LAURENU2
10-28-2010, 01:44 PM
OK gopher :hifi: I have one for you
In the maine forum page menue bar "quick links"
Click "view site Leaders " It go's Nowhere

Bok :punpolice: this one's for gopher

gopher_yarrowzoo
10-28-2010, 03:18 PM
Hmm i'll look into it and FYI outlandr I wasn't pouting :p

gopher_yarrowzoo
10-29-2010, 09:34 AM
Hey Mario, I a fixa thata pagea for you :D

LAURENU2
10-29-2010, 05:26 PM
Hey Mario, I a fixa thata pagea for you :D
You Da Man gopher_yarrowzoo
____:allhail:____

So thats what it show:thumbs:
Seems weird names are hidden until you mouse over tho :confused:
I will keep looking for more Bugs:guntotin:

gopher_yarrowzoo
11-16-2010, 08:33 AM
Seems weird names are hidden until you mouse over tho :confused:
I will keep looking for more Bugs:guntotin:
Yeah well weird names might wanna stay hidden so you find any new bugs :P