Suggestion: Personal proxy for farmers

[pointwood]

The distributed.net project and others have this:

The possibility to host your own personal proxy. That means that all your client installations will (when configured to that of course) connect to that proxy instead of the main distributedfolding.org server.

Now, I'm not an expert in this - far from it, but I'm sure there are others here that can explain this in much greater detail.

I don't know if this is at all possible with this project, but if nobody suggests it, then you will probably not consider it ;)

[Paratima]

I LIKE this idea! Could save us a lot of... :bang:

Maybe if we get some of the network-smart guys in here...

Like Dyyryath or IronBits or Darkness Productions, whaddaya think?

("When in doubt, shout." :) )

[Starfish]

I like the idea, but you would like a proxy smart enough to detect that a certain phase is almost ending so that it can start uploading ;)

[FoBoT]

what is the advantage?

[Angus]

What FoBot said...

Is there a network/server bottleneck now? I've never seen any retries on sending since the server was upgraded.

The old saying still holds true -



If it ain't broke, don't fix it.

[pointwood]

Some people have large farms on a network where only few machines have an internet connection.

Another idea would be to make it easier to sneakernet. Maybe it could be solved by having a little util that could harvest all the WU's from alle the other clients, create the filelist.txt and upload them?

[Dyyryath]

Hey guys. I'm sure Howard will weigh in with his opinion here, but let me say a few words since he & I have already discussed this.

During the time when they were having lots of dropped connections & such, I made a handful of suggestions about what might be done to help combat the problem. This was one of them.

At the time, I didn't really know much about the way his upload process worked. He basically replied that he didn't think it would be worth the amount of effort it would take to build. Now that I know how his uploads operate, I'd tend to agree.

I'm not saying that proxies aren't possible. I could write one myself, but there are things that would have to be worked out with Howard & crew. I doubt that Howard is going to feel like the advantages to proxies are worth the hassle & extra layer of complexity that would be involved at this time.

What would we gain, anyway? The ability to store completed work on a proxy server when the main site is down? Well, the client will already store completed work locally when the main site is down, so there's not much of a benefit there.

There isn't any work being handed out, so a proxy isn't going to gain you a 'cache' of work to use when the main servers are down either.

They would theoretically help with server loads on the main site, but at the moment, Howard seems to have that problem licked.

Really, the only really good use I see for a personal proxy server would be additional stats tracking. Now, I'm never against more stats data :D, but that's probably not a good enough reason at this point for Howard to be spending time on something. I'm quite sure he's got a list of other things to do. ;)

Now, if he does see a good reason for this or he doesn't mind seeing it done just to allow for more stats, and he just lacks the time to do it himself, I'd be more than happy to write something and submit it to him...

[Paratima]

Stats was something I hadn't even considered. :o Also, I didn't realize it would take any effort on Howard's part, or I might not have chimed in. Howard is kinda the "scarce resource" at the moment.

What I was thinking: Yes, the uploads are buffered in the client, especially if you remember the -df. However, once buffered, they sit there until the next set of results is ready to go, or until a restart. Now the longer those results stay in the client, the more danger there is to them, like being inadvertently deleted or, more likely, being orphaned by a protein change! Yes, the server is MUCH better, but there are still periodic logjams, like the first couple of hours after a break for maintenance or a network calamity, or just plain busy times.

The best thing, of course, is to DON'T WATCH your machines. Unattended folding is much better on the nerves. But still, if there were a proxy to pick up the result packets and then send them in as soon as the server is ready for them, it would take a lot of the pucker factor out of participating in this project.

[pointwood]

You are probably correct Dyyryath.

What I'm actually looking for is for a method to harvest generated structures/WU's from other clients in an easy way. If you have a lot of clients that have no internet connection, but are on a network, then Distributed Folding is not an option currently - it is simply too much work.

Now, I actually believe it would be pretty easy to create such a little util like that.

First, the client is already capable of just uploading the results and then exit.

Writing a little util that harvest the WU's from clients on a network and creates the filelist.txt file, shouldn't be too hard. There could be a problem: The clients probably needs to be shutdown when you harvest the WU's, but if you run the client as a service, wouldn't that then be possible to do then?