Reply With Quote
...time to shutdown and move on to something else then, not really a challenge running 10 clients/instances.
Any idea when you may have a revised client out? Or an infrastructure that can handle unlimited clients per user?
DLV Client limits
Taken from the DistComp Forums on RR.org
Howdy all,
This is to notify DLV project participants that we have (unfortunately) been forced to re-implement a client limit to the project.
The maximum number of clients a user can now run is 10 clients.
I know this is a pain, especially to those of you who are running a lot more at the moment....but we have to do this to keep our server load under control. You may have noticed that in the last couple of days the server has been up and down (or very slow) - this is due to the number of partipating clients reaching 3000+ !!! In the past we have been used to handling about 200 or 300 clients...so this is a massive change.
We thank you all for supporting the project, and we will be aiming to resolve this issue shortly. We are also working on DLV 2.0 which will be a massive change to the project and will introduce a lot of exciting new features and functionality....so watch this space!
Thanks,
Wang
...time to shutdown and move on to something else then, not really a challenge running 10 clients/instances.
Any idea when you may have a revised client out? Or an infrastructure that can handle unlimited clients per user?
How many new user accounts can you create?
So - now the project team competition is down to how many users you have, not how much crunching power you can bring?
100 users x 10 clients will always win over 20 users x 10 clients.
Hardly seems fair at all.
Last edited by Angus; 08-14-2006 at 06:49 PM.
Killing off the clients now. Can't even run ONE!
[5:15 PM] Client started
[5:15 PM] Connecting to Red Library server...
[5:15 PM] Sorry - you are running too many clients on this project
[5:15 PM] The maximum number of clients you can run is 3
[5:15 PM] Client stopped
My SOB output will go up some now.
How many new user accounts can you create?
Loads if you have your own mail server
I ~think Free-DC had more than 3,000 clients by themselves.
A project not wanting more firepower... wow !
Hope you get resources to upgrade your servers...
I will come back when you can hang with the big dogs.
>>TeeJay (1,000 client cruncher)
Originally Posted by darkstorm777
darn that means I can only run 1 client for every 6 PC's running here Thats not Good
There was some speculation in the past (when they had the three client limit) that they also checked IP addresses, and would disable multiple user accounts from the same IP. I don't really know first hand, though, because I never tried it - but thought I'd pass it on just so it didn't surprise anyone if it happened.How many new user accounts can you create?
Loads if you have your own mail server
Yep they did but I think they gave up eventually as those that wanted to compete just kept making more clients and I don't think they completely banned IP's. It's a shame they have to do this really as although they say they don't need more than 3 clients per user this was getting more people to run the project, certainly over at XS when they saw the competition to be had people joined up. I just hope they lift the limit soon otherwise we may never close the gap you guys have pulled out recently.
Damn. I was hoping to start raiding this project when i got my DSL sorted out. Damn!
It seems that even running 10 clients, the credit granting is stuck or broken.
I've only got 1000 credits since the limit went into effect, and the clients are working away.
Hi guys,
I'm Wang from the RedLib DLV project. I wanted to come over to the forum and fill you in on the recent changes to the DLV project.
We apologize for the issues with the client limits - I know it's a pain, but really we had no other choice but to implement the limit (or watch the whole project become inaccessible due to server load).
The server had been responding extremely slowly (if at all) for 2+ days, and this is basically due to the large number of clients that were participating since the removal of the client limit. For the latest DLV release we optimized the project considerably, to the point where our servers could handle 2500+ clients as opposed to only 300. However, the number of active clients went up to 3500+ which was more than expected.
The DLV project now has a 10 client limit, until we work something out (this is surely better than the old 3 client limit though!). We really appreciate your support, and it is only through testing and finding issues that we will be able to make the project better in the longrun. We are not a professional team of developers, we are just people who like Distributed computing and wanted to put up a new project for people to enjoy.
The RedLib DLV client is technically v1.0 of our project plan....and it is in it's final stages. The next stage (v2.0) is already in development and it will be a completely new project, based off of the DLV code. The client will still verify the RedLib links, but that will be a secondary task (and the links in the db will only be fully checked once a day). The primary task will be to monitor the uptime of servers...here is how it will work:
1) You sign up to the dlv project, and let us know where in the world you are located.
2) You add up to 3 URL's to our database (these could be your own sites/servers, or just some sites that you like visiting).
2) You will join a pool of clients who are also located in the same country (i.e. if you are from the UK, you will join the UK pool).
3) A map on our site will show you all the locations in the world that we have active clients responding from (this is pretty cool to see, and to get a better idea of how global a distributed computing project really is).
4) Our server will then take one of your 3 URL's that you added during registration, and it will send that URL to two clients in every country pool. The clients will then verify whether or not that URL is responding (as well as some additional data, such as how fast it is responding etc).
5) If the URL is found to be inaccessible from any of the countries, we will (optionally) send you an alert to warn you that your URL/site is down and give you extra information (such as which countries report it down etc).
This kind of service is already provided by a few web sites out there (e.g. www.alertra.com) - however, our distributed network would be much more powerful and accurate since it would check your servers from all around the world....and it's free! (Alertra.com is not free). We understand there are security implications with a project of this nature, and we will of course we taking every action necessary to ensure the project/network is used correctly.
This new project idea will be launched shortly, and we hope that you will be a part of it! We would be very interested in hearing your feedback/comments/suggestions, so please reply either here or on the www.readyresponse.org forums.
Thank you for your support,
Regards
Wang
I can't even start 1 client let alone 3 or even 10.
I think from a stats and team competition perspective, that this isn't a viable DC project.
If they can't handle a couple of thousand clients, any couple of big teams will continue to sink them. Adding limits per user only gives an advantage to teams with many users, not those with the most crunching power to bring to a project.
Fozzie: If you haven't already, you should report this issue to staff@readyresponse.org or post on the www.readyresponse.org/forums/ so that we can check your account. If you haven't tried running the client since yesterday, I would re-try running it today as the daily stats will have reset.
Angus: Fact: we can't handle 3500+ clients with our 1 low-spec server, and more to the point, this project does not require 3000+ clients (RedLib only has 20,000 links). Ideally we only need around 350-400 clients participating.
Please read my message above. We are working on DLV 2.0 which is the overall goal of the DLV project. The purpose of the DLV 1.0 is to see how feasible it would be to create our own distributed project. If you want a large scale distributed project, then DLV is not for you (at this moment in time). We are simply distributed computing enthuisiasts who are slowly working towards developing our idea (DLV 2.0) by experimenting and testing the DLV 1.0 client.
If you don't feel that DLV is ready for your participation at this time, then that is fine and you are probably right. However, your support is very much welcomed and appreciated - and if you actually want to help us test, develop, and improve our project....then I encourage you to continue participating. The choice, as always, is up to you!
regards
Wang
just an opinion here...
I think that many of you are missing the whole point of a distributed project. It is my understanding that the RR dc project is used to validate RedLibrary links. Many of you treat distributed projects as a competition of numbers. The higher my stats the cooler I am, a very childish mentality imho.
Instead of being happy that you are helping keep up such a great IT library like RR, some of you just seem to bish and complain when RR server admins try hard to acomodate your requests. The fact of the matter is that Ready response is financed and created by a group of individuals who believe in freedom of information, not by some huge corporation with financial gains. The Project was built by volunteers who have given endless hours of their personal time to create a library of links that is free for all, and a distributed project to help keep that library up to date...not to generate meaningless statistics for some virtual competition.
As someone who uses RedLibrary...I can only wonder how many of you have actually used the Library.
Don't get me wrong, I am very greatful that so many of you are lending your computer time and clients to help those of us who use the library, but I find it absurd that a link is checked 100s of times a day just to satisfy some egocentric competition.
Before you bish and complain so much just take a moment and think about the goals of distributed applications, and whether you care about helping red Library or about overloading the server so much with your number crunching that you make the site that you are trying to help go down, and in doing so keep users who are actually interested in the content of the library from being able to access it.
Sorry if I am harsh and feel free to flame me all you want, those are my two cents of this issue.
There is a point you are also missing too. Stats are a very large part of distributed computing. Like them or not, without them I'd bet there would only be 1% of the current contribution to DC there currently is...
While I agree that it was kind of absurd repeating so much of the information, the site admins allowed it, so competition was had.
Just where the hell do people say 'bish'?
Bok
atlantis
It is you who are missing the point.
Distributed computing is all about competition.
Projects want as many people as possible to contribute to there cause.
To keep these people onboard they foster a spirit of competition between Individuals/Teams.
How do they do this ?
Simple Stats.
RL has Stats therby encouraging competition.
If they didn't want participation from DC teams why the hell didn't they set up the stats.
People such as yourself would willingly give up computer resources to help out RL wouldn't you ?
No of course you wouldn't thats why we are here, hardcore DC'ers.
PS
We don't take kindly to winging trolls so dont rush back.
I think the point that Atlantis is making is that the primary purpose of distributed computing is not to gain stats. We shouldn't lose track of the goals for each of the projects you participate in. Sure, stats are fun...but there is a higher purpose to each and every project, and the stats will always be second to that.
If the project only NEEDS a few hundred clients, they should limit registrations to only that many. If folks leave, open it up for a few more registrations every so often.
A wide open DC project with stats will get jumped on very quickly by large teams. They should know that by now if they've been around DC much at all.
Angus: That is exactly why DLV has a client limit.
We always had a limit in the past, and we temporarily opened it up to see what the load would be like without one (as I have mentioned before, DLV is used for testing). We made it clear that if we saw load issues again we would re-implement the client limit, and that is exactly what we have done. Therefore, I do not see the problem.
Actually - you haven't limited accounts. You could still get 10,000 accounts, each of whom will try to run the maximum number of clients. As long as one of your parameters is open, you haven't limited anything.
The client limit serves it's purpose. I am not saying that it's impossible to get around (sure, we could limit registrations if we felt the need) but it keeps load under control. I am merely explaining why we have the limit in place.
ps. I notice you are in Seattleif you ever come over to the east-side let me know, perhaps we can meet up to discuss distributed computing!
I could live with 3 clients per PC But there is no fun in a 10 limit per account
Since RR doesn't actually make any money (and we have to pay for a dedicated FreeBSD server, trusted SSL cert, domain registrations, DNS services/failover, etc) - any donations are always more than welcome!
For the DLV 2.0 we will need multiple servers running the project, as we consider 2.0 to be a full-scale distributed project that we intend to invest money into.
Thanks for all the information given!
Basically, this project is not
1) geared up for the pressure that large pharms can bring.
2) was never intended to be a real DC project in that you don't need or want thousands and thousands of folks/computers to help out all at the same time.
I understand completely, keep up the good work!
Oh, and let us know when you think you're ready for the 'big dogs'
I appreaciate you taking the time out of your busy schedule to keep us abreast of the changes.
Let us know if there is anything we can do to assist.
Thanks for one of the only "nice" replies re: this matter Ironbits. We didn’t expect it to kill the server, but we gave lots and lots of fair warning beacuse we knew it was a possibility. And as Wang stated, this is only a test. A working test for that matter. The next project that we are looking into developing will not have these issues plus we will be as our own team putting a lot more time, effort and money into it.
Again we would absolutely LOVE any of your feedback (as long as its constructive, some of the childish comments and attitudes I can really do with out) and/or assistance with the new project.
If anyone’s interested shoot one of us an email or join up to the forums and express your interest.
Again for those of you that were understanding, its you we work for, for those that wernt, you lay in the bed you make for yourselves in life.
Peace.
P.S.
PCZ- Your comment:
People such as yourself would willingly give up computer resources to help out RL wouldn't you ? No of course you wouldn't that’s why we are here, hardcore DC'ers.
Actually we give up much more than our computer resources to help out RL, in fact, we have dedicated YEARS to more organizations than I can count on two hands that help Free Speech and Net Neutrality further education AND Distributed Computations. So please if your going to make assumptions and act presumptuous, at least let them be accurate.
It may not hurt to work on your communications skills as well.
You being the #1 Spot on ANY team, you must understand you are; to others, a representation of your team.
A "hardcore" anything would know at least that much.
We can't fault your project when it wasn't designed for the heavy weights
I think there were only 15,000 links, so that would be only 15 TeeJay's worth of work, at 10 links per run, so in about 15 seconds he ripped thru all of em.
Too much redundancy at that rate anyways...
We are all a bunch of STATS hoes around here, so don't take anything anyone said to seriously.
Get yer project ready for some serious battles and get back to use when you're ready, as it doesn't interfere with other DC projects.
Get-R-Done!
lol you guys really are good at what you do!
And we will continue doing what we do best, and thats to do our best and bring interest where it hasnt been before with new projects.
A man with out failures has never learned a thing
So why you 'bashing' PCZ ?Thanks for one of the only "nice" replies re: this matter Ironbits. We didn’t expect it to kill the server, but we gave lots and lots of fair warning beacuse we knew it was a possibility. And as Wang stated, this is only a test. A working test for that matter. The next project that we are looking into developing will not have these issues plus we will be as our own team putting a lot more time, effort and money into it.
Again we would absolutely LOVE any of your feedback (as long as its constructive, some of the childish comments and attitudes I can really do with out) and/or assistance with the new project.
>>Hey! I resemble that remark...
If anyone’s interested shoot one of us an email or join up to the forums and express your interest.
Again for those of you that were understanding, its you we work for, for those that wernt, you lay in the bed you make for yourselves in life.
Peace.
P.S.
PCZ- Your comment:
People such as yourself would willingly give up computer resources to help out RL wouldn't you ? No of course you wouldn't that’s why we are here, hardcore DC'ers.
>>I believe that was for Atlantis... why you butting in ?
Actually we give up much more than our computer resources to help out RL, in fact, we have dedicated YEARS to more organizations than I can count on two hands that help Free Speech and Net Neutrality further education AND Distributed Computations. So please if your going to make assumptions and act presumptuous, at least let them be accurate.
It may not hurt to work on your communications skills as well.
You being the #1 Spot on ANY team, you must understand you are; to others, a representation of your team.
A "hardcore" anything would know at least that much.
YOU as a representative of RL should have better forum manners...
A 'professional' anything would at least know that much.
Now quit picking on us and get back to RL...
Just my childish .02 worth
There is nothing on the Red Library project page about it having client limits or only needing a few hundred crunchers.
Generally the info for DC projects gets passed around on team forums. We see someone make some noise about a project, and it's off to the races. I saw that the team here was looking for more power to beat the arch-rival (ars) and followed the links in this forum to register and started spawning clients. Frankly, I never even saw the forum link on your site until I started having issues with the clients and went looking for an explanation, so any verbage in there about client limits is too far removed from where it needs to be to be of any use.
Why don't you do everyone a favor and put a BIG sign on the project page (the red-and-black part) saying something to the effect of "We got plenty of help - we don't need any more users right now", and duplicate that with maybe more of an explanation on the sighn-up/registration page. Buried in the forum is the wrong place for that message.
WangRR
Your limit does not seem to be not working for me. Looking at all the nodes I have and all seem to be doing links About 100 clients. only one thing different, I had them all set at 5 Min But now there testing links every 10 Min even tho it is still set at 5 MIN
Statistics for user LAURENU2
Team Free-DC
Total Links Validated 1,962,527
Total Links Validated Today 1,000
Total Bad Links Found (and verified) 0
Overall Project Rank 38
Statistics for user ironbits
Team Free-DC
Total Links Validated 23,028,945
Total Links Validated Today 0
Total Bad Links Found (and verified) 0
Overall Project Rank 4
What [b]is[/b] RedLibrary?
OK- I've been poking around on the RedLibrary and ReadyResponse web sites, and I've yet to find the answer to my basic question -
What the heck IS Red Library, and why is so important to 404-check the same 6000 or 7000 links all the time? One PC could do that easily once a day.
Who uses this thing? It can't be much of a search engine - some of the links we're checking are just old news items, and other search engines probably add that many links every minute to their database.
So= not trying to sound smart-ass - what gives?
What a dissapointment this project has become. At one stage I was running nearly 800 clients, but now Im reluctant to run even 1.
Cant help but think drop like a hot stone at the moment.
There should be statements as said above on the site that say limit the clients and we only need x amount of users.
What a shame. Was good to compete in though.
Hello, My Name is Skyloer Gray and I work on the Red Library Project..
First I would liek to say thank you to everyone that has put time and effort into running this client. I know there has been a lot of issues with it, but in actuality the problems have been more help to us than anythign else, as liek what was said previously... this project was a test to see what we need to do for a large scale DLV project like you guys are use to.
Secondly the idea about placeing a message on the actual project page is a great idea, and I will look into that. I will poke and prod until we get somthing that works better than the notice we currently have in place.
Lastly to answer the questions I have seen repeated... WHAT is RL, WHO uses it, and WHY is it important. To answer these questions let me give just a real quick history of Red Library, and its parent orginisation, Ready response.
Ready Response started out as a Brigade in Cyber Army, which im sure some of you have heard of and some of you have not. At this time Cyber Army had a game known as Zebulun. Zebulun was a hacker challenge. During this phase, Redlibrary (much smaller at the time,,, some 500 links if I recall) was used to research things you might need to know to pass a level.. such as, about how password protecting a website works, etc etc.
After that, Ready Response started doing "missions" that is used entered mini projects, such as learning how to do somthing, or signing a petition. or just general ways to help your self or others on the internet. At this time Redlibrary was used to reseach some of the proposals, and how to complete them.
Slowly as Red Library grew it slowly started being used as an unofficial IT library, as at the time, ALL links were hand submited by users, and verified by a small team, which I worked on and then was in charge of at one time. While I can't tell you the names of specific companies or groups that use it off the top of my hand, I can tell you that haveing seen the logs of searches and hits, that the Library is used quite often by all sorts of people all over the world, with many returning guests.
As we began to see the popularity of Red Library growing we began to supliment the user entered links with links submitted automaticly from other IT and computer security sights around the net. all well known and accurate in hopes of supplying accurate and timely information in regards to specificly internet security.
At this time Red Library is used as a small (relitive to other general search engines) specialised search engine in the IT/Computer Security area for free.
Why is it important? becuase any free accurate information is important to the internet as a whole. Why do we need to check the links so much? The answer is simply .. we don't. We basicly killed two birds with one stone. We needed a way to veryify links that offered faster results and less server strain then the php script I was working on. We also needed somthing to test or skills at programing a DC project, so that we can learn everything that would be needed in a large scale DC project. To this end we have been successfull on both counts.
I hope that this has cleared some things up, if you ahve any more questions please feel free to ask. Valid points and reccomendations have always been implemented as best as possable under our current resources.
Again thank you , you guys are a great help in this project.
one easy solution was to remove the 5 minutes limitation and the 10 links limitation ...
each computer has one client with the full link list ... and then loop through all the 20000 link ...
after one hour running (or when quitting app), the client send the number of link visited and report the 404 ...
with this kind of configuration, even my p2-300 can be used as a server ...
if your server can handle 500 (underestimated) client at the same time, it means that if you have a mean connection length of 10 seconds (overestimated), you can handle 50 client /second or 180.000 client (computer) an hour ...
you will never have that much computer participating ... you can expect something around 500-1000
the problem with your server is only due to the client limitation (5 minutes - 10 links) removing these two limitation is the answer ... (DC-addict point of view)
my 2 cents !
tnerual does have a valid point
Your Servers were flooded with 3500 request Per Hr that is a lot of disk writes
With tnerual way you would only have 292 request that is a lot less disk writes for the same number of clients
Thank you.
That is a great idea worth looking into. I will be sure to link this to the devs, if they have not seen it already. Great idas everyone.
Posting Permissions
Way to go bro! 