Forum and stats server move

[Bok]

I'm going to start the last part of the process of moving the stats and forum over to their new homes today/tomorrow. Hopefully it will be fairly smooth and not much noticeable difference but it's possible things might get missed so let me know.

I'll have to switch off the dnet proxy temporarily whilst I move it's contents. Correspondingly, ppstats will switch off and will be the last thing to switch back on.

The stats have been running parallel on the new server for a few months now. It's quite a bit faster given it's a core i7 and the database tables themselves are housed on solid state drives. The combined BOINC stats are therefore calculated in real time rather than on a slight delay as they currently are. What is missing is the weekly/monthly data. I'll be copying that over in the coming days given it's an 80Gb transfer..

The bandwidth up is lower but we'll just have to see if it handles the load ok, I can't tell right now. If it doesn't I'll look into getting that increased on a business line which will also allow for a static IP, but the costs are fairly high there so I'm trying not to do so unless really necessary..

I'll be switching DNS on the stats site prior to the forum which will mean it will not think you are 'logged in' on the stats until the forum is switched too. Therefore statstool should not really be used until completed...

Bok

[vaughan]

I'm going to start the last part of the process of moving the stats and forum over to their new homes today/tomorrow. Hopefully it will be fairly smooth and not much noticeable difference but it's possible things might get missed so let me know.

...

I'll be switching DNS on the stats site prior to the forum which will mean it will not think you are 'logged in' on the stats until the forum is switched too. Therefore statstool should not really be used until completed...

Bok

Yes that is the case - logged in automatically on the Forum but the Stats pages show I'm not logged in. Is there a way to log in on the Stats pages separately from the Forum?

[Bok]

It's something I'm going to have to investigate. Likely a 'clear the cookie' issue. Will sort it out tomorrow though. Spent way too much time on all of this today..

[Bok]

Should be fixed now...

Now only thing I have a problem with is :

my firewall does not support putting through custom ports to NAT, so I can't forward port 2064 for the dnetproxy.. it's running on port 23 for now though as it always has in addition.

[vaughan]

bok are you still working on the stats server? I ask because I'm finding the stats pages extremely slow to load, lots of "waiting for stats.free-dc.org" messages

[Bok]

Connection is not as good as it used to be. IB was paying quite a lot for the old one and I haven't got that kind of spare change to throw around right now. $100-150+ per month extra likely..

Just updated the firmware on my firewall to see if it makes a difference but it hasn't. It's just the amount of traffic coming in when I only have 768K up (15Mbps down). I'd ideally need to upgrade to a business connection and get the 5Mbps up.

[gopher_yarrowzoo]

Hmm I suppose it would depends on what uses the most bandwidth I'm suspecting the stats pity you couldn't have put that somewhere else with a better connection and made sure you could get to it if need be.

[Bok]

The forum takes little to no resources at all really. Stats uses up 99% of the bandwidth and requires the most use of the hardware too....

[LAURENU2]

Well all we have to do is fine a few more people like me
That are willing to put out some money every month like I do
It is Like DC-ing DC-ers share there resources to help the World
Well giving your resources to Free-DC so they can help the world does the same
I Pledge $50.00 a month I have set it up with my bank Auto-pay to pay once a month

So come on Gay/Gals Look into your Harts and your pockets
And make a pledge to help the Free-DC world
If you have the Bucks to DC You have the Buck to HELP

[Bok]

Thanks Lauren, I do appreciate that.

I've been spending time trying to figure out where the problems are. Found and blocked one IP address which seems to have been trying to get the same file every second or so in bursts, was taking approximately 1/3 of available bandwidth usage. Next I may replace my standard cable modem and also the router I have in place, I just wish I had a good way to check where the problem is occuring. Running ntop on the webserver shows that it can handle the traffic with ease.

[Johnny Cool]

Where is that PayPal link here? I saw it when I was not logged in, now I cannot see it. I don't have that much, but I guess that it is 'BTN' (Better Then Nothing).

[Johnny Cool]

Where is that PayPal link here? I saw it when I was not logged in, now I cannot see it. I don't have that much, but I guess that it is 'BTN' (Better Then Nothing).

Sorry — your last action could not be completed

If you were making a purchase or sending money, we recommend that you check both your PayPal account and your email for a transaction confirmation after 30 minutes.

If you came to this page from another website, please return to that site (don't use your browser's Back button) and restart your activity.

If you came from PayPal's website, click the PayPal logo in the upper-left corner to return to our home page and restart your activity. You might have to log in again.

Hmmmm...

[ChertseyAl]

So come on Gay/Gals Look into your Harts and your pockets

I just threw a few dollars in. Not many, as I don't know how big a dollar is, being from the colonies

Still not entirely sure why I should look into a male deer (hart), but my *heart* suggested I pay a little

Al.

[Bok]

Not sure why the paypal link does not work. A few others have used it recently ok??

Thanks Al, got that.

Let me give you all an update

Anyways I'm almost stuck as to what the problems are right now. I do see a lot of traffic so it must be mostly working but there is obviously some bottleneck. I wish I knew someone with more networking experience than I have to help out so if anyone knows please get in touch.

I've just replaced my switch with a $100 netgear gigabit version and it's not made any difference at all that I can tell.

This is the setup right now.

I'm on a 15MBps down/768K up connection. Comes into to a splitter using RG6 which splits the signal into 4 components. One goes to my Motorola SB5100 cable modem, one for voip and the other 2 for cable tv.
From the cable modem it goes into the Netgear FVG318 firewall, from there to the gigabit switch and then to the servers.

From either side of the switch if I ping say 8.8.8.8 I can get anything from 23ms to 1500ms.

Three places I could attempt to alter.

1. Firewall - it should be capable of handling the bandwidth.

2. Cable Modem - I could buy a newer one capable of docsis 3 (mine is docsis 2 )

3. I could try to alter the initial splitter to split it into 2, once going to the cable modem and the second one then splitting out for cable tv and voip.

I mention 3 as I managed to log into the cable modem and the downstream power level seemed a little high

Frequency	609000000 Hz Locked
Signal to Noise Ratio	35 dB
Power Level	-10 dBmV The Downstream Power Level reading is a snapshot taken at the time this page was requested. Please Reload/Refresh this Page for a new reading

Upstream	Value
Channel ID	7
Frequency	33000000 Hz Ranged
Power Level	42 dBmV

I have a nagging doubt in my mind that DNS requests are somehow causing the problems though. I've been messing with nameservers locally for the last few days and even if I have one on the same machine I'm 50% of the time getting timeouts.

I'm really not sure where to go next and would really appreciate any help people can think of. I did try getting in touch with Dyyryath (old free-dc guru) but he's not responding..

Bok

[Johnny Cool]

Sorry — your last action could not be completed

If you were making a purchase or sending money, we recommend that you check both your PayPal account and your email for a transaction confirmation after 30 minutes.

If you came to this page from another website, please return to that site (don't use your browser's Back button) and restart your activity.

If you came from PayPal's website, click the PayPal logo in the upper-left corner to return to our home page and restart your activity. You might have to log in again.

Hmmmm...

It went through fine (Terry from Colorado Springs).

[LAURENU2]

It went through fine (Terry from Colorado Springs).

Your the Man Mr COOL

[Bok]

I've built a smoothwall firewall from an old P4 server I had, been trying to get it configured which is why the site was down but it looks like one of the nics is bad. I'm going to have to buy a new nic for it as I don't seem to have any others in my 'stash'.

I'll probably have to sever the connection again sporadically whilst I get it up and running. This will replace the Netgear Firewall I have which at the moment is at the top of my list for the case of the bandwidth saturation.

[ChertseyAl]

Mucho snippage, but ...

I'm on a 15MBps down/768K up connection. Comes into to a splitter using RG6 which splits the signal into 4 components. One goes to my Motorola SB5100 cable modem, one for voip and the other 2 for cable tv. From either side of the switch if I ping say 8.8.8.8 I can get anything from 23ms to 1500ms. 2. Cable Modem - I could buy a newer one capable of docsis 3 (mine is docsis 2 ) 3. I could try to alter the initial splitter to split it into 2, once going to the cable modem and the second one then splitting out for cable tv and voip. I mention 3 as I managed to log into the cable modem and the downstream power level seemed a little high. I have a nagging doubt in my mind that DNS requests are somehow causing the problems though. I've been messing with nameservers locally for the last few days and even if I have one on the same machine I'm 50% of the time getting timeouts.

Taken me all day to get to post this. Site is V slow. Actually F slow Mostly timeouts today So this is very terse as I've got a feeling I won't get to post it ...

A few random thoughts:

Only 768k upstream? Seems slow. My home use bargain basement cable connection is 1m for only 10m downstream.

From my days working with the UK cable supplier, a four way splitter was a no-no. Three ways max. And that need much tweaking in the street cab to get the levels right.

SB5100 - Isn't that kinda outdated now? I don't think we use them over here any more. Hasn't your ISP given you something better?

Pings to 8.8.8.8 taking 1.5s seems odd. Hardware problem?

Power level - Not familiar with your network levels (only ever done satellite for US clients). 4-way split is going to be 14dB down. Seems odd to have too high level.

DNS - I don't think that's the problem. Download my 'big' graphic is slower than dialup. It's a bandwidth problem.

My guess is failing hardware. But then if your upstream is overloaded I think of malware, but you're not likely to be daft enough to not notice that!

Al.

[Bok]

Appreciate the comments!

I think the UK are ahead of the US in cable connections. I can't get better than 768K up without getting what's termed a business connection and those are not cheap..

7 Mbps down, 786 Kbps up - $97.95/mo.


10 Mbps down, 1 Mbps up - $167.95/mo.


12 Mbps down, 1.5 Mbps up - $279.95/mo.


15 Mbps down, 2 Mbps up - $389.95/mo.

I've got the two way splitter so I can try that part tomorrow I think.

I've had that SB5100 for quite some time so they may be giving out better ones these days. It's the next thing to try anyway even if I buy my own.

If I switch off the http redirect on the firewall pings are a lot better but still I don't think the connection is saturated as such. Get to that in a moment.

I've got the smoothwall server running now in place of the netgear firewall so we'll see if that makes a difference. I'm not going to tweak anything again till tomorrow. Let me know if it appears better or not. It does have much better diagnostics and is currently showing a snapshot of

malware - it's possible but pretty unlikely - I do keep on top of that kind of thing, but I'll certainly check it out. I did see a few kb going to eDonkey when I installed ntop on the webserver but I think that was a red herring. It's a linux box and I never use p2p clients.

I did tweak apache a little as I was very concerned about reverse dns lookups. Found out that the standard logging format includes %h which can cause hostname lookups regardless of the settings and switched that to %a but I don't think it made a difference.

192.1 Kbit/s / 296.4 Kbit/s (Out / In) which is not too bad.

Just been watching the logs and someone has been trying to do some sql injection attacks from China.. Sneaky but it won't work. Banned the ip - I'm going to get more aggressive on doing this kind of thing.

So : next steps if needed.

1. redo rg6 splitter
2. new cable modem
3. give up unless I win the lottery tonight

There is a very real possibility that I'm just not going to be able to host the stats any longer but we aren't there yet. Forum and the other websites I run are not an issue.

Bok

[Bok]

Is it any better? Certainly seems a lot better from within my network. Smoothwall is running quite nicely. Still got some challenges but I'd like feedback first..

[gopher_yarrowzoo]

Seemed quick enough from here when it loaded a bit faster than that last time I looked, that's for sure. I'll work on the Paypal link on the front page / stat server see what's up with it if you like.

[LAURENU2]

Is it any better? Certainly seems a lot better from within my network. Smoothwall is running quite nicely. Still got some challenges but I'd like feedback first..

Yes Faster Stats are loading in 14 sec. much faster then the 208 sec it took before
forum is only 4 sec NOW down from the 18 sec it took
it looks a LOT better
Your The Man Bok

[ChertseyAl]

A lot quicker today. Have been able to get to this thread easily each time I tried throughout the day. My big project stats graphic is still pretty slow to download, but quicker than yesterday.

Overall, things are much better

Al.

[Bok]

I think it's going to be ok then. I am seeing a lot of bandwidth activity but it's still not maxing out. So, perhaps now it's time to stat some bandwidth saving issues instead. I think I've spent a little under $1000 in the recent weeks on hardware, that part can take a rest for now, donations have definitely helped

Taking a look at the w3compiler though, free trial, we'll see what it does..

[Bok]

So, is everything still looking good ? No issues with speed at all?

As of about 30mins ago I've finally figured out why my cacheControl wasn't working so all of the javascript files and static images should now be cached locally. This will hopefully speed up the pages quite a bit more I think.. Try it and see.

I'm also considering making the default on the pages to NOT include the graphs but instead have a button to show them if required. This will just be a parameter in the url, so easily accessible still. The graphs take up about 80% of the size of some of the pages so it will drop the footprint of most pages to only a few kb.

Any thoughts?

[LAURENU2]

Graphs off to the side is OK with me
Some of the graphs/charts were hard for me to read
But That maybe just Me
Speed is vary GOOD now
_____Thank You_____
>>>Bok<<<

[gopher_yarrowzoo]

Graphs off to the side is OK with me
Some of the graphs/charts were hard for me to read
But That maybe just Me
Speed is vary GOOD now
_____Thank You_____
>>>Bok<<<

Yes good plan I guess I should maybe go warm up some nice swishy code to do a nice reveal without having to reload the page

Additional also making all the tags STATIC automagically would that not help with bandwidth so it's not on having to do a call for each tag as they are cached too?

[Bok]

No, making them static would not change the bandwidth as the end result is still the same size. It would save on db calls and such, but that's not an issue. They are cached locally on the server anyway.

For the 'reveal' function you are thinking about, what we have to ensure is that the data is not built at all until the reveal is done.

Say the fullpage with graphs and all is 20k of html + 100k of graphs, we want initially to only send that 20k. If the user hits reveal then the extra data for the graphs is transmitted. I guess what I'm geting at is not to just hide it via javascript. We are aiming to get the pages to load much faster from the client side.

this is actually a real nice resource for testing webpages with lots of ideas...

I'm actually not worried about bandwidth as such as long as it's not affecting speed, really I'm just trying to make the client experience that much better. With the caching of the javascript and css files and saving of graphs when not needed the page load time should be a few seconds at most.

[gopher_yarrowzoo]

Nope was never thinking of having the graphs hidden but called upon when asked would need to grab the variables for them from someplace but that is doable

[LAURENU2]

Ok Who changed my 3 crowns in the stats from Gold to Pink and why
I have a hard enough time with the way my Mother spelled my name
With 2 Pink crowns I am sure people will think I am a Gal

[Bok]

Where is it showing as pink? They should not have changed at all..

[LAURENU2]

hahaha I'm sorry they are gold Except when you High lite that row
And I had it highlighted ha ha ha
http://stats.free-dc.org/stats.php?page=users&proj=mil

[gopher_yarrowzoo]

hahaha I'm sorry they are gold Except when you High lite that row
And I had it highlighted ha ha ha
http://stats.free-dc.org/stats.php?page=users&proj=mil

Lauren check your monitor, your seeing things, looks fine on my screen

[LAURENU2]

Lauren check your monitor, your seeing things, looks fine on my screen

My monitor is just OK. I can see you just fine
Go comb your hair it looks like you had your head tucked under your pillow all night

[LAURENU2]

OK since the Move on the main stat page http://stats.free-dc.org/stats.php?page=index
we have 2 Eon's listed
#1 http://stats6.free-dc.org/stats.php?page=proj&proj=eon
#2 http://stats6.free-dc.org/stats.php?page=proj&proj=eon2
And Both are wrong. The right link is
http://stats.free-dc.org/stats.php?page=proj&proj=eon2

We Need to take the ( 6 ) out of the link to make it right

And The old Eon is dead and should be laid to rest

[Paratima]

I agree with Lauren on this one. Graeme and his crew now have their own forum at U. Texas.
We no longer need to preserve the EON forum space. Guess we could put up a linky, but even that seems excessive. Anyone wants to find him should be able to. And the stats for EON[0] are truly only historical. It wouldn't break my heart to reclaim that space. Just my .02 worth.

[Bok]

OK since the Move on the main stat page http://stats.free-dc.org/stats.php?page=index
we have 2 Eon's listed
#1 http://stats6.free-dc.org/stats.php?page=proj&proj=eon
#2 http://stats6.free-dc.org/stats.php?page=proj&proj=eon2
And Both are wrong. The right link is
http://stats.free-dc.org/stats.php?page=proj&proj=eon2

We Need to take the ( 6 ) out of the link to make it right

And The old Eon is dead and should be laid to rest

It doesn't link to stats6 for me? All of the stats pages have dynamic links to whichever versions they are on, basically using a variable of $root_page.

Can you check again at http://<a href="http://stats.free-dc...page=index</a>

for the two Eon projects and see where they link to?

We can put the old Eon into Archive then - Done.

I'll switch off the scripts which do the updates tomorrow. Sad day really, I remember writing those scripts whilst in a hotel room in Sao Paulo, Brazil...

[LAURENU2]

OK I found where the problem lies
It is with the BOINC link to Free-DC main stat
that they list in all projects Statistics and leaderboards
http://milkyway.cs.rpi.edu/milkyway/stats.php

There link brings you here http://stats6.free-dc.org/stats.php?page=index

But the right link should be http://stats.free-dc.org/stats.php?page=index

Bok you need to contact Boinc and have them change there link
Or redirect all your file links
If you do not see what I am saying go to any Boinc project and click on there
Statistics link Click on the link to Free-DC stats
and you will see it leads you to a OLD stat page
http://stats6.free-dc.org/stats.php?page=index

[Bok]

ah, no that's me. It links to http://stats.free-dc.org/ which then redirects to stats6. It's an apache vhost config issue. I'll fix.

[gopher_yarrowzoo]

OK I found where the problem lies
It is with the BOINC link to Free-DC main stat
that they list in all projects Statistics and leaderboards
http://milkyway.cs.rpi.edu/milkyway/stats.php

There link brings you here http://stats6.free-dc.org/stats.php?page=index

But the right link should be http://stats.free-dc.org/stats.php?page=index

Bok you need to contact Boinc and have them change there link
Or redirect all your file links
If you do not see what I am saying go to any Boinc project and click on there
Statistics link Click on the link to Free-DC stats
and you will see it leads you to a OLD stat page
http://stats6.free-dc.org/stats.php?page=index

Hmm Lauren, I just looked at that page I it directly linked me to stats.free-dc.org which is correct and also what it shows on the mouse over on my status bar.