Boitho stats

[Bok]

Added today..

http://stats.free-dc.org/new/projpage.php?proj=boi

They are slightly different to the projects own stats as they have a leaderboard based on a combination of the pages and robots. I'm just ranking on the pages.

Bok

[PY 222]

Where did this come from?

A linky would be good.

[Bok]

I'll leave the explanations to PCZ as I know very little about this..

the link I'm using for the stats is

http://dcsetup.boitho.com/cgi-bin/dc/topTeams.cgi

Bok

[PCZ]

http://www.boitho.com/dc/
http://www.boitho.com/forum/

[PY 222]

Ahh...

This is a distributed crawler, where everybody can donate there superfluous computer resources and spare bandwidth, to help us create a bigger and better search engine.

Our goal is to make a general internet search engine with a thumbnail picture of all the pages. The problem is that the bandwidth and computer resources needed to make a thumbnail of a internet page is several times the resources needed just to download a HTML page. This means that Boitho has to spend more resources on crawling than other search engines.

To make the most of our available resources, and to allow volunteers to donate their superfluous bandwidth and idle CPU time, we have developed a distributed crawler for Boitho, like seti@home and Grub. That way people can install a program on their computers and help us with the crawling.

Just like another Majestic12.

[PCZ]

A small warning.

The client can crawl robots.txt and URL's.
Crawling robots.txt the load on the PC is small but when it starts crawling URL's it a CPU hog.