Results 1 to 13 of 13

Thread: Boitho stats

  1. #1
    Administrator Bok's Avatar
    Join Date
    Oct 2003
    Location
    Wake Forest, North Carolina, United States
    Posts
    24,444
    Blog Entries
    13

    Boitho stats

    Added today..

    http://stats.free-dc.org/new/projpage.php?proj=boi

    They are slightly different to the projects own stats as they have a leaderboard based on a combination of the pages and robots. I'm just ranking on the pages.

    Bok

  2. #2
    Keeper of the Fridge PY 222's Avatar
    Join Date
    Jul 2002
    Location
    San Jose, CA
    Posts
    2,706
    Where did this come from?

    A linky would be good.

  3. #3
    Administrator Bok's Avatar
    Join Date
    Oct 2003
    Location
    Wake Forest, North Carolina, United States
    Posts
    24,444
    Blog Entries
    13
    I'll leave the explanations to PCZ as I know very little about this..

    the link I'm using for the stats is

    http://dcsetup.boitho.com/cgi-bin/dc/topTeams.cgi

    Bok

  4. #4

  5. #5
    Keeper of the Fridge PY 222's Avatar
    Join Date
    Jul 2002
    Location
    San Jose, CA
    Posts
    2,706
    Ahh...

    This is a distributed crawler, where everybody can donate there superfluous computer resources and spare bandwidth, to help us create a bigger and better search engine.

    Our goal is to make a general internet search engine with a thumbnail picture of all the pages. The problem is that the bandwidth and computer resources needed to make a thumbnail of a internet page is several times the resources needed just to download a HTML page. This means that Boitho has to spend more resources on crawling than other search engines.

    To make the most of our available resources, and to allow volunteers to donate their superfluous bandwidth and idle CPU time, we have developed a distributed crawler for Boitho, like seti@home and Grub. That way people can install a program on their computers and help us with the crawling.
    Just like another Majestic12.

  6. #6
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    A small warning.

    The client can crawl robots.txt and URL's.
    Crawling robots.txt the load on the PC is small but when it starts crawling URL's it a CPU hog.

  7. #7
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    Bok hope this helps with the stats

    From AMD users forum
    runarb wrote:

    Stats formula is as follow:
    Rank = crawledpages + (robots.txt / 6)

    But one should probably use the "Place" column, not generate it's one. Sow we can change this later.

    Also project does have a forum
    http://www.boitho.com/forum/

  8. #8
    Hi

    I am one of the people behind Boitho.

    Where is the name Boitha coming from? The name is Boitho. Typo?

    More easy parsable stats are coming on Sunday. Probably tab separated. I will keep you informed.



    Any how, I watching this board sow just post here, or mail if I can bee of any assistants. Mail is: runarb [at] boitho dot com

  9. #9
    I started using this yesterday, I went out and when I came back in my computer had been restarted (sat waiting on login page), this has happened twice both times when I have not been here to see what happens.

    Any one have any ideas as to why this program would cause XP to shut down?

  10. #10
    OK happened again slightly earlier comp just reset itself, possibly due to me over working it and having 30 crawlers all going at once over the last few hours, have set it at a smaller number to see what happens next

  11. #11
    Administrator PCZ's Avatar
    Join Date
    Jun 2003
    Location
    Chertsey Surrey UK
    Posts
    2,428
    30 crawlers is far too many.
    Probably XP ran out of resources.

    It's OK when crawling robots.txt but when it starts grabbing pages your PC will be hammered.

  12. #12
    30 crawlers can be to much, depending on your hardware. But you shouldn’t get any other problems that that you don’t crawl as many pages as you could.


    XP crashing like that is almost always a hardware error.

    What probably happens is that 30 crawlers get the computer working hard, and it overheats.

    Have this happened before when you did run programs that uses much cpu?

    You can try to use a program that monitors cpu and motherboard temperature, and se if it increase to much.

  13. #13
    Does it even set on a small amount so something else is happening when this program is running also, using trial & error now at the moment to see what setting it can cope on, set it now to just 2 will see what happens in the morning when I wake up, hopefully the computer won't have restarted its doing nothing for my uptime project!! lol

    Edit: PS Thanks for advice hard trying to work out what was wrong due to limited documentation, thought it may have been too much for it to handle but wasn't sure

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •