PDA

View Full Version : Too close for comfort!



CaptainMooseInc
11-15-2006, 09:46 PM
XtremeSystems has been ramping up their MJ-12 production recently and it's getting a littttttlllleee too close for comfort. Thankfully PY 22 is producing TONS of our teams daily %age.

Today we did 33 million URLs.
XtremeSystems did 31 million URLs.

That means XtremeSystems just needs another 4mbps connection to match our daily output. I'm going to up my limite by 400kbps and try to add a few hundred thousand more URLs/day.

If anyone has a spare 512kbps then please consider it but be very careful if you're a Comcast Cable user. If you're a DSL user then you shouldn't have any bandwidth limitations but this isn't guarenteed.

I figure if we can get 4 people at 512kbps then we can push ourselves up to the 35-37 million URLs/day mark and keep our large lead on XtremeSystems.

Not TOO big of a worry though seeing that they are over 2 billion URLs behind us in the stats so they'd have to be producing double what we are daily to be a real threat.

Team Norway is #2 in stats but they're only doing 9 million URLs/day.

birdman2584
11-15-2006, 10:21 PM
I actually put my home computer on this last night, and it has been going pretty well for me, I did aroudn 600k urls today. Hopefully this will slow their ramp a little.:thumbs:

CaptainMooseInc
11-15-2006, 10:25 PM
Very good birdman! If you need any help optimizing the client for your particular system (so long as it's windoze) then lemme know. There's really not much I haven't tried to do with the program and I can help you get as much use out of your connection as you're willing to offer.

Just lemme know if you need anything.

-Jeff

birdman2584
11-15-2006, 10:28 PM
Yeah, Im still playing around with the # of workers with # of open buckets thing. It also seems to take awhile for things to archive, but thats probably because i have another dc project going. It seems that a lot of time is being spent with only one domain left on a lot of the buckets too, which is really slowing down my url count. If you have any suggestions throw em my way.:hifi:

CaptainMooseInc
11-15-2006, 10:37 PM
What bandwidth are you working with? Like how much do you want to give to the project? What's your max downstream/upstream?

Also what is your other DC program you're running?

Once I know this I can tell u what you need to do.

birdman2584
11-15-2006, 10:44 PM
Well, I just got a letter from my cable company yesterday saying they were upgrading download speed from 4 Meg to 10 Meg. My upload speed is being upgraded from 384k to 1 Meg. I am not really sure what this means though. I am also running LLR for riesel base 5. I dont think I have any limit per month on how much bandwidth i can use.

Edit:
I forgot to add, my setting right now are set at 6,000k for download and 512 upload, and 15 buckets allowed to be open with 100 workers.

CaptainMooseInc
11-15-2006, 11:04 PM
Well who's your cable company? You want to be EXTREMELY careful. I can't have Comcast Cable back until August 2007 but thankfully I moved out of their jurisdiction and into Insight.

10mbps down and 1mbps up........

That means you need to make sure you have your TCPIP.CONFIG file patched (if you're using WinXP SP2). You'll need the tool from here to do it: http://www.lvllord.de/?lang=en&url=tools

Under Tools->Crawler:

After you've done that then I would set your max async workers to 125 (for using 6mbps, if you wish to go higher then do 200 for up to about 8.5mbps usage). Set max open buckets to 65. Set delay before crawling to 1. Set pre-cached buckets to 25. Use fixed upload chunk of 64kb. Use persistent connection. No delays between upload chunks. Wait between barrel uploads 2 seconds. Stop if uploads greater than 200.

Now, to get MJ-12 to work well with other DC projects but not really ever utilize more than 3-5% of CPU cycles I have the following for you...

Under Tools->Misc:

CPU Priorities: Set all to Below Normal except Upload thread which should be set to Normal priority. Normally for DC projects this kind of process affinity would be a very bad thing but this project focuses more on bandwidth instead of CPU usage. Giving it these priorities will allow it to "get a word in edgewise" when trying to act properly. You should see very minimal impact on your main DC project. Also check the box that says "Clear log file on start".

Under Tools->Archiving:

Check the "Run archiving in a seperate process space". Process priority "below normal". Archiving delay 3. LZMA compression enabled. Word size 24 bytes. Dictionary size 4mb. DO NOT enable external archiver.

Archiving for me using OGR-25 takes about 2mins per bucket. With no other program running it takes like 45 seconds but this is acceptable wait time to allow another DC project to take precedence, once again with very little impact.

Once you've done all this then click OK and then just File->Restart so that way all the changes you've made will take effect. :)

Bed time now. If you need further help I'll be back tomorrow afternoon.

-Jeff

birdman2584
11-15-2006, 11:14 PM
Yeah, I am under Insight also, but I dont really understand what that patch does that you put the link on there for. I think I will just lower my bandwidth a little until a get a bit more info on it. Either way, I am still cruising pretty well. Thanks for your help!:hifi:

CaptainMooseInc
11-16-2006, 05:18 AM
When WinXP SP2 came out it essentially "capped" your max number of TCIPIP connections at one time. So really it is stopping you from using your connection to its fullest potential. By removing that cap (with that patch) it allows you an unlimited number of TCPIP connections instead of the low cap that Windoze has set. That way programs like MJ-12 works better. :)

-Jeff

movieman
11-17-2006, 07:19 PM
Now whose too close for comfort?:lmao:
Just being neighborly and trying to get a little closer to all our good friends over here at Free Dc.
The old expression that "Imitation is the sincerest forn of flattery" fits well here.
You guys set the bar that the rest of us try to strive to so this ones for you!:cheers:

CaptainMooseInc
11-17-2006, 08:41 PM
Ahh you just wait movieman! I'm upping the ante about 1.2mbps every Sunday through Thursday!

And don't make me cancel my cable internet to get another 1mbps down and 512kbps up higher than what I have now through a no-limit naked DSL line.

Even worse for you would be if I just went ahead and kept the cable too. :)

Frisch
11-17-2006, 09:10 PM
XtremeSystems has been ramping up their MJ-12 production recently and it's getting a littttttlllleee too close for comfort. Thankfully PY 22 is producing TONS of our teams daily %age.

Today we did 33 million URLs.
XtremeSystems did 31 million URLs.

That means XtremeSystems just needs another 4mbps connection to match our daily output.
Well, i'm doing a little over 2 mill per day with a 2mbps connection, so we don't need 4 :D ....good tweaking that's all...and good choice of url's can lead one to wonders... :D

CaptainMooseInc
11-17-2006, 11:35 PM
what kind of tweaking are you doing? and what domains are you crawling?

I'm using 2.8mbps daily and 4mbps at from 9pm to 4am Sun-Thurs and only achieving 1.5 million URLs/day. Yeah I want more stats but I also wanna use my connection for the best crawling possible.

If there's any way you can help me get some more URLs/day out of my 2.8mbps connection plz lemme know.

CaptainMooseInc
11-18-2006, 12:23 PM
Ahhhh! It looks like a TON of ppl are coming together under 1 XtremeSystems team name (like our Free-DC Mercs) to try and topple PY in daily output! I love how it takes so many of them though. :)

Also, XS has REALLY ramped up how hard they are hitting MJ-12. They are sitting at 39 million URLs crawled today compared to our 22 million for today. Most of those URLs are PY's too.

I'm running all out at the moment and during the nights stretching my connection to its fullest. If anyone has any bandwidth they can spare plz bring it to MJ-12. Dunno if PCZ can bring back his connection for a few days or not but we are getting killed in the dailys.

On the other hand though, not too much to worry about when it comes to getting overtaken by them.

And in the Good News, we are less than a week away from having 5 billion URLs crawled as a team!

em99010pepe
11-18-2006, 12:44 PM
Last time I ran M12 my antivirus, NOD32, detect it as a virus. I will try it again as soon as I finish the download of EA FIFA 2007.

Carlos

movieman
11-18-2006, 01:11 PM
Ahh you just wait movieman! I'm upping the ante about 1.2mbps every Sunday through Thursday!

And don't make me cancel my cable internet to get another 1mbps down and 512kbps up higher than what I have now through a no-limit naked DSL line.

Even worse for you would be if I just went ahead and kept the cable too. :)
Glad to hear it. We did do a couple of subteams but that is more internal competitions than trying to topple PY222..
I just figured(with help) how to get all from my 15mbit/2mbit FIOS line and it's sucking like a $2.00 Ho in a Marine Barracks!:thumbs:

ToshPower
11-18-2006, 01:44 PM
Last time I ran M12 my antivirus, NOD32, detect it as a virus. I will try it again as soon as I finish the download of EA FIFA 2007.

Carlos

Nothing to worrie about, it's described on the MJ12 board as harmeless.

So come on, FDC can use the help at the moment :Pokes: :thumbs:

KAMCOBILL
11-18-2006, 07:42 PM
Well who's your cable company? You want to be EXTREMELY careful. I can't have Comcast Cable back until August 2007 but thankfully I moved out of their jurisdiction and into Insight.


I lost mine in February for a year and I was only doing 1TB a month. They said I was abusing it so I asked what the limit was and they said there wasn't any limit. :confused: I need to get a PY222 conection:D

XS_The_Machine
11-18-2006, 08:12 PM
Hey we gotta have fun while we can :clap:
I hear this internet thing is just a passing fad :harhar:
Mad:roadkill: MikeE for XS_THE_MACHINE

CaptainMooseInc
11-18-2006, 08:51 PM
Roadkill? Far from it. In the dailys you may be stomping us but you need a LOT more than what you're doing to catch us!

And with Insight going to up my connection to 10mbps/1mbps by the end of the month then I swear I'm gonna rock some socks a little harder when I get done!

movieman
11-18-2006, 09:00 PM
Roadkill? Far from it. In the dailys you may be stomping us but you need a LOT more than what you're doing to catch us!

And with Insight going to up my connection to 10mbps/1mbps by the end of the month then I swear I'm gonna rock some socks a little harder when I get done!
Glad to hear it! The best part of competing with Free Dc is you get the best!
Let's go and set some records on this app so that all you hear at the MJ12 forum is:
"Who are those guys?"
Apologies to Paul Newman( Butch Cassidy and the Sundance Kid)

I wonder what the one day record is on MJ12? Got to look that up.

and to Py222::thumbs: :cheers: :clap: :rock: :cheers: :notworthy
:allhail:

XS_The_Machine
11-18-2006, 10:43 PM
We been doing it back & forth for years, just not here, sorry if I prickled anyone :jester: (Like I said, these friendly competitions are for FUN :looney:

There's a reason you guys have gotten into 1st in a lot of stuff, and I / we all respect it.:rock:
(Hey you guys even have really nifty/cool smileys, just seems a shame to waste 'em :D )

We're all on the same side in the long run - DC fellows out to help projects :D

Thanks for giving us something to aspire to/chase after:clap:

Hey I couldn't even have my sig w/o this place & Phil/Bok :D And Thank you Mr. Moose for the sharing the settings & advice that help us all to get ahead in this project. (Heck I didn't think we'd get anywhere near what we did today :cheers: )

I figure you guys will have adjusted in a day or 2, so I was 'givin you the business while we could :D


Hey we gotta have fun while we can :clap:
I hear this internet thing is just a passing fad :harhar:
Mad:roadkill: MikeE for XS_THE_MACHINE

XS_Diablo_Legion
11-18-2006, 11:21 PM
The only problem of a Free DC vs XS Throwdown is that we may fritz the servers at MJ12. They may need some liquid Nitrogen to cool them down.

The best part is that even though we may kid each other we respect each other and we have fun while showing the rest of the DC world that when two quality teams collide , civility, respect and fun doesnt have to suffer.

But if PY doesnt open his :fridge:soon....

:haddock: :haddock:

CaptainMooseInc
11-18-2006, 11:31 PM
heh. I'm always up for a good race. I don't have a large pharm but I will push it as far beyond 100% capacity as I can possibly do. I'll do anything for that 1 extra barrel of URLs/day or couple of stat points just to chuck up another one. :)

And I didn't think my advice on the tweaking of the client would be as helpful as you claim it has been Machine. I have spent hours upon hours with each new build Alex puts out and though I may not find the most errors when it comes to the client I always figure out what alex may have unlocked when it comes to even better utilizing a network connection.

Anything for a good run! Glad to see someone is actually getting us on our toes. I was getting realllllllllyyyy comfortable sitting up there thanks to PY. :D

Now I have a reason to make sure that once Insight gives me my extra 6mbps that I try maxing that out as much as possible. And if they threaten to cut me off like Comcast did once they do up the speed then I'll just cut them out of the picture and pick up the 5mbps/1mbps naked DSL I can get w/o limits. :rock:

I'm sure you'll see a fight coming back at ya soon XS. So don't get TOO comfy. :hifi:

movieman
11-18-2006, 11:37 PM
heh. I'm always up for a good race. I don't have a large pharm but I will push it as far beyond 100% capacity as I can possibly do. I'll do anything for that 1 extra barrel of URLs/day or couple of stat points just to chuck up another one. :)

And I didn't think my advice on the tweaking of the client would be as helpful as you claim it has been Machine. I have spent hours upon hours with each new build Alex puts out and though I may not find the most errors when it comes to the client I always figure out what alex may have unlocked when it comes to even better utilizing a network connection.

Anything for a good run! Glad to see someone is actually getting us on our toes. I was getting realllllllllyyyy comfortable sitting up there thanks to PY. :D

Now I have a reason to make sure that once Insight gives me my extra 6mbps that I try maxing that out as much as possible. And if they threaten to cut me off like Comcast did once they do up the speed then I'll just cut them out of the picture and pick up the 5mbps/1mbps naked DSL I can get w/o limits. :rock:

I'm sure you'll see a fight coming back at ya soon XS. So don't get TOO comfy. :hifi:
Hello Cap'n: Lets have some fun the next few days and blast the the living crap out of MJ12 and get the rest of them wondering as as Jose said, make Alex have to get some LN2 for his servers!:cheers:
PS: Your right! 2 BILLION is a huge lead and all we're looking at right now is tossing Norway into a fjord and watching them slowly sink away!:rotfl:

CaptainMooseInc
11-19-2006, 11:20 AM
movieman or someone from XS please do me a BIG favor! That admins over at your team have yet to approve my account creation so I can't reply to what moddolicious has said but it needs said so plz post the following over there to him from me:

Mod, do NOT use the MJ-12 library. Yes it does crawl more URLs than the .NET one but alex has said to discontinue use of the MJ-12 library unless you're using the Mono version of MJ-12 (non-Windows). The library itself is very buggy and is producing poor success rates. We're looking for quality of crawled URLs and not quantity really so that 83% success w/ .NET versus the ~50% success with MJ-12 is a very large difference. It should be noted that no one with Windows should use the MJ-12 library until Alex gets it fixed up and released way later on (probably after January). The MJ-12 library is not searching faster and finding more bad, it's that the library itself is bad and isn't doing what it's supposed to do!

-Jeff

P.S. Thanks in advance Movieman or anyone from XS that may get this over there! And ask an admin over there to approve me plz! :-D

XS_The_Machine
11-19-2006, 12:00 PM
Reply posted with a supporting blurb.
Msg sent to XS admins, hopefully they will LMK when you are set up so I can pass it along, or they'll just send someone over to let you know...
(Can't understand why my page loading is sorta slow :umm: :jester: )

While I'm Thinking of it for those the celebrate it.

:hiya: HAPPY THANKGIVING! :rock: :cheers:

May you all have a reason to do so!:clap:

XS_Diablo_Legion
11-19-2006, 12:14 PM
[QUOTE]=CaptainMooseInc]Now I have a reason to make sure that once Insight gives me my extra 6mbps that I try maxing that out as much as possible. And if they threaten to cut me off like Comcast did once they do up the speed then I'll just cut them out of the picture and pick up the 5mbps/1mbps naked DSL I can get w/o limits. :rock:

[QUOTE]

BooooooooooooooooooooooooooK...mose said naked in this forum:blush: :blush: :blush: :partypop: