REQ: Upload bandwidth reduction

[Halon50]

Howard, do you have an approximate ETA for a new version of the client (as alluded to in previous threads) that doesn't upload every structure calculated? I ask because I have a tightly capped upstream bandwidth (50Kb/s I believe), and a single machine uploading 5k structures pretty much takes out my network's Internet capabilities until the upload is complete.

[dtsang]

You can write a few batch/script files that would:
1) run the foldtrajlite program with buffering on, so that you generate lots of structures without uploading them
2) upload all structures, then quit the program

All of the instructions and switches for doing this are in the ReadMe1st file.
When you are using your Internet connection, you would run batch file 1). When you are not using your machine, run batch file 2).

[Mikus]

I also have a dial-up connection.

With the latest protein, I'm spending more than 20 minutes per day uploading filesets.

The documentation states that 10000 structures is the upper limit for a fileset. I don't know if the total upload time would be reduced, or not, if this limit were allowed to be higher.

I believe it was said somewhere that the servers will be going into a new mode, where the raw data for ALL structures would no longer be archived, just "interesting" excerpts.

Whenever this new structure-retention mode is introduced, let me suggest changing the CLIENT (rather than the servers) so that "non-interesting" structure data does NOT have to uploaded.

Having to spend more than 1 percent of my computer time on uploads seems wasteful.

mikus

[Halon50]

dtsang, I looked into running batchfiles and scripts for my internal network (about 17 machines running at the moment), but it would require more handholding than my current policy of laissez faire which I'm rather more inclined to follow...
:p

It also doesn't solve the bandwidth issue. I recently dumped a mere 1.1 million structures, and it took well over an hour of babysitting my machines and clogging my network's upstream bandwidth to complete. It would clearly not be an option to save up a full week's worth of results and try to deal with them over an entire weekend.

By the way, I only bring this up now due to the timely release of the UT2003 demo over the weekend... one 5k dump would cause around 15 agonizing seconds of dropped UDP packets, making the game pretty much unplayable.
:help:

EDIT: I neglected to mention, I am not the only one on this internal network...

[Brian the Fist]

The 'newer method' is already in place - we are storing much less data than before. This does not mean you have to upload less though. If we are to keep proper statistics we must have evidence that your computer has actually done the work. If you only uploaded one piece of data out of every 5000, it would be too easy to spoof results. I realize this may be a fair bit of uploading for modem users, but they have managed until now with few complaints.. I realize the project is suited more to people with high speed connections though, which I suspect the majority of project users have.

Anyhow, not to fear. The newer algorithm, which will work very differently from the present one (although look stunningly on the client side :D ) and hopefully much better, will require much less data transfer, in both directions. It is defnitely a ways away, but something to look forward to.

[Halon50]

OK. Thanks for the update!
:cheers:

[Scoofy12]

If you don't keep the data, and the only thing that makes people upload it is keeping them honest, surely there's another way. Maybe if the client only sent certain specific data about the protiens it generated, or you were to make the client somehow encrypt/hash things to prevent cheating? Not really sure how, maybe someone else has an idea.


Originally posted by Brian the Fist
Anyhow, not to fear. The newer algorithm, which will work very differently from the present one (although look stunningly on the client side :D ) and hopefully much better, will require much less data transfer, in both directions. It is defnitely a ways away, but something to look forward to.

/me perks up...
Newer algorithm? Neato! how does it work? is there any info available on it for enquiring minds, or is it one of those trade(s):jester: secrets?

[FoBoT]

Originally posted by Brian the Fist
I realize the project is suited more to people with high speed connections though, which I suspect the majority of project users have.
...... will require much less data transfer, in both directions. It is defnitely a ways away, but something to look forward to.

i realize i may not fit the normal profile , but i am having a hard time with the bandwidth requirements even with my cable modem.

i bring home 300-500 MB of structures and it still takes a couple hours to send them all up through my cable modem

the new stuff sounds good, maybe if the next protein is slower, it will take some pressure off, since a slower protein will take longer to get the 10,000,000,000 and make the uploading smaller/less

[Paratima]

Howard, it's clear from your response that YOU don't, on a regular basis, have to upload over a modem. Or send a BIG sneakernet dump over a cable! And your point:
they have managed until now with few complaints.. ...doesn't mean it's been painless. I recall reading several posts concerning protein changeovers where the difficulties of modem users in Europe were detailed. The fact that we are now suffering fewer changeovers (How sweet it is!) doesn't mean that uploading is any easier, just that the urgency has decreased, plus the fact that you have brilliantly solved most of the other problems... credit for late submissions, faster server code etc., etc.
It is defnitely a ways away, but something to look forward to. The joy of eager anticipation is wearing a little thin. Can we get a better EWAG at "a ways away"? ;)

[runestar]

Originally posted by FoBoT


i realize i may not fit the normal profile , but i am having a hard time with the bandwidth requirements even with my cable modem.

i bring home 300-500 MB of structures and it still takes a couple hours to send them all up through my cable modem

I'd think if I was a dial-up user I'd be rather unsympathetic and my response would be along the lines of: "Oh, shut up!" If I were a modem user.

300-500MB going to take a whiles short of being right on a T1 or higher. =)

RS½

[Alpha_7]

I'm realitively new to DF, and have been suffering bandwidth problems in silence since joining. I have 56k dialup and can't upgrade to anything faster, yet with only 1 client uploading my whole outside line is tied up. I basically have to go and do something else till I the upload is complete. (Email, www, etc grind to a halt). I have a whole network crunching here and I am throughly enjoying the change from SETI my "home" project, is there any way I can make the client use less bandwidth when upload ? (say half not all over it? )

[Brian the Fist]

Well it already does upload minimal info per structure - about 100 bytes per structure, and that then compressed with bzip. Can't make it any smaller really unless it is left out altogether. We'll try to make the next protein a little bigger and 'slower' and maybe that will reduce your bandwidth usage in the mean time.
Alternatively, I could rewrite it I suppose to only upload the minimal data thats actually stored, which would drastically reduce the upload time but make it easier for some people to cheat. Note when I say 'easy' you'd still have to go to some considerable effort as it is by no means trivial, but there's no 1024-bit RSA encryption here, that's all. If the majority of stats-mongers would be comfortable with such a change I could probably go ahead and do that at the next protein changeover.

[Digital Parasite]

Originally posted by runestar½
300-500MB going to take a whiles short of being right on a T1 or higher. =)

Yeah, have to get me one of those new residential 3 Mbps DSL lines.... :D

Jeff.

[Halon50]

I'm definitely for reduced upload sizes, and I'm pretty sure very few (if any) of us are willing to cheat just to inflate our stats.

It does sound like a valid concern though. How difficult would it be to add encryption schema to the up/download sequences?

[Scoofy12]

As far as not having complaints from people, while it may be true that not many people have mentioned it here on this DF forum, I have heard people mention it elsewhere, and even people that decided not to run the project because of the bandwidth requirements. It's not a major issue for me personally, but I think it would help to increase production overall by attracting both dialup users and people with large farms that would generate enough to saturate broadband connections. (of course a big problem with most DSL/cable connections in this case is that they are limited to 128/256k upload anyway, and it tends to kill the DL as well if the UL gets saturated.)
All this to say i'm in favor of less upload if it's feasible and not overly easy to cheat.

[pointwood]

I'm all for making the upload smaller, as long as it doesn't compromise the science.

I trust you, Howard, 100% to make the correct decision in that regard ;)

[Paratima]

Originally posted by pointwood
I'm all for making the upload smaller, as long as it doesn't compromise the science. What he said.

[IronBits]

Ditto! Don't compromise the project, but, reduce the bandwidth requirements as much as possible.

[MarcyDarcy]

I already read this somewhere, but what the hack :rotfl:

Can't you let the client check, after some work is done, at the server if there are already been better (read: smaller) structures. If so than he doesn't have to upload. Mayby you can make it if there are (for example) less than 100 smaller structures then the client will upload the results. Or that the client compares it with the users best/smallest result.

If there are smaller structures then the client writes something witch containes the names of the work that is done and then give credit for it. The writing of the names is a way so cheating isn't possible (not easy, like you already wrote).

If ( i use a lot of iff's :confused: ) this could work, than only the usefull results would be uploaded. And that will reduce the upload by hudge hudge amounts :)

I don't know if this is possible, but i like this project and as far as i know it is the only "bigger" project that isn't friendly for modem/dail-up users. The project could also use these kind of people