Return to smaller slices?

[Halon50]

Hi. I'm new here. Is there a chance you could return to smaller slices again? The last few values for n my machines have grabbed are quite large, and would take ... (checks the machine)

73 hours on my dual P3-450 machine to complete (getting close to 40 hours on a Duron-1.1GHz). I do have a small farm I'd like to switch over from SETI and other minor projects, but most of them are in the "low-class" category, K6-450's and the like.

Mind you, once I can afford some of those Tyan MP boards/CPU combos, I won't mind the large samples, but for the sake of mass production, would you consider possibly halving the current slices?

[prokaryote]

Along the same lines as Halon50's suggestion, what if the client could do a quick benchmark and assign slices that way. Also, since smaller n's take less time, what if we were to try to find primes for other k's, assigning the smaller n's based upon such a benchmark to the older/slower machines. That way, several k's could be tackled and the slower/older machines could return some completed units sooner.

Thanks,

prok

[smh]

Is there a chance you could return to smaller slices again?

Louis can probably answer this better but..

The current client is doing one complete proth test. The larger the number gets (current numbers are over half a million digits) the longer it takes to test them. To test a number n^k+1 takes about k itterations, and every itteration uses a value that was returned by the previous itteration. If the number is split into two or more chunks than the result of the last itteration of your chuck has to be sent to the server. The one who is taking the next part of that test has to get your result from the server. This was actually what the older client (now disabled) did. The results that have to be sent to and from the server are getting too large. Something of a couple of hundred KB for a current K (wild guess)

[jjjjL]

smh is right.

i liked the smaller slices too because it was psychologically more appealing from a user point of view. it's a tad daunting mentally to watch your computer get "slower" as the search progresses. that's why cEMs exist... they correct, more or less, for the fact that larger n values take longer for each individual iteration of the test and also have more iterations... a double killer. fortunately, it's not exponentionally increasing, only O(n log n).

and yes, just look at the size of your cache files. when the project was just 5 users and had lower n values, it wasn't as big a deal to burn the bandwidth. the numbers went something like:

5 users * ~3 computers/user * 10kb * avg block rate / block size

which worked out to around 4MB day. even that was enough that Dave eventually was strongarmed by his cable company into shutting it down. they didn't approve of his "unauthorized server".

so we moved it into Alien88's "server room" aka basement aka bedroom. ;) we were planing to move the operation out of the basement to a local ISP but we knew that no one would donate tons of bandwidth to us... so we did the numbers to figure out how much our server would use... and were terrified.

50 users * ~3computers/user * 200kB * 30 (block rate/ block size)

878MB a day!!! :cry:

and it gets worse... the 200kb number increases at a rate of about 2kb / day... but w/ more users joining, that causes n value transmission spread to ACCELERATE over time, not just increase. then take into account that there are more users over time and you have an exponentially growing bandwidth. :cry: :cry:

money i make working on SB: $0

money it costs for 50GB bandwidth: $too much

money it would cost for 1TB of bandwidth: $:cry:


this doesn't even take into account the fact that transmitting bases to the server also introduces a very real source of errors to creap in. i trust woltman's implementation of a prp test more than i trust window's implementation of winsock2.


then look at the current test method... transmit less than 32 bytes down, and 48 bytes back up at the end of the test. i didn't actually check my own code... it may be less. in any case, the sb server itself now uses less bandwidth than the sb webserver by a factor of at least 1000x now. this is good. 1TB transfer = :cry: .



that said, it would be possible for me to make blocks that couldn't be transfered between users. for instance, it would be no huge deal to make the client basically break the block down itself. so, for instance, it could easily have two seperate % bars, one overall and one per block.

this could create the situation where each block would still process about the same speed no matter what the k/n value, which i think would be supa cool :). so n = 2 million would have like 50 blocks while n = 3 million would have more like 200 blocks. i'm just making up numbers for this example, but i can make an algo that would correctly subdivide n values so as to make blocks = blocks.

i just thought of this whole idea just now while writing. how long would you like blocks to take on what kind of system? i've been involved as a user in several distributed computing projects, and i know that blocks that take a few hours on a system feel better than blocks that take days to finish.

let me know what you think. suggestions alone will guide me on this part of the design. ;)

-L

[Mr_Peabody]

I would prefer blocks that don't take as long. Because:

1. I primarily run my computers overnight and shut them off during the day. It would be nice if blocks or jobs roughly took, say 4-8 hrs on a 600mhz p3.

2. I run several other distributed projects and I like to complete a block, a test, a unit or what have you before I move on to the next dist-proj.

3. Sometimes however I may run 2 dist-proj at once on the same system, so blocks that only took 4-8hrs would be great. Currently if I run sob by itself it takes about 45hrs. If it shares cpu resources with another dist-application it can take 150hrs or so. Still well within the 2 week expiration time frame, but a 8-16hr run time would be great in those instances.

[prokaryote]

Hey jjjl,

Thanks for the explanation. Good luck with the bandwidth issues. As an aside, any reason why we have to only search or investigate 1 "k" at a time?

Thanks,

prok

[jjjjL]

we are investigating one k-value to show what a distributed computing project can do to push the bounds of a k value, to some extent.

also, we don't want to jump to a new k value until we have a good reason. i think i will consider other k values once i can assure that the majority of people doing those tests have upgradeed to 0.9.7 or above (to come soon).

-L

[Halon50]

I like the idea of blocks, but you said it couldn't be transferred between users? As in, if a single machine completes 99 of 100 blocks, then never sends the last block, the entire n workunit has to be redone by another machine? I'm not so sure I like this idea--the visualization of "progress" is already implemented in the counters within the program window.

Is there a way to deterministically divide each n value into smaller slices without the extra bandwidth overhead? Would it be possible to, for instance, allow one machine to compute factors up to a certain value a, then another machine to finish factorization for all values a to n/2 (or whatever the appropriate value is--I really don't understand the math... :confused: )?

[jjjjL]

i'll think about it, but there is no obvious way to break tests up.

the test involves repeated squaring, which quickly produces an enormous number. an enormous number which is very non-deterministic... another computer could not just hope to "guess" or account for the fact that it doesn't have that number and just restart a test half way through.


it just occured to me now that since the server is just a middleman, it would be possible to make the program deliver the intermediate state of the test to another computer directly. while this solves the initial problem for me of no bandwidth usage for my poor server, it only eliminates 1/2 the bandwidth usage overall for the project. ignoring the complexity of building a p2p interface for my project, the client would still use something like 1GB/day of total bandwidth. from a users point of view, it would probably only use a few MB/day, so it could be done reasonably, but i think it would still be terrifying from a design point of view. everything that is already hard (stats, network code, caching) would be made infinitely harder. and this doesn't solve the initial problem which is that a user would still have to count on someone else to finish his block to have it count. i prefer the idea of multiple blocks/test... it could still upload a report to the server when it finishes the individual blocks, it just wouldn't be able to transmit the result until it was finished.

i'll keep thinking.

-L

[Halon50]

It occurred to me...even if there isn't necessarily a way (an easy way) to reassign untested blocks to another user, it's still a very useful method to measure if work is being done on the n value at all. This way, you (and the server) can more easily find values that haven't been worked on, and thus you could reduce the "timeout" delay from 2 weeks to something a little more reasonable, like 2 or 3 days from the last block sent?

This, in addition to the possibility of more accurate measurements of stats values, make your "blocks" idea sound more and more ideal. I hope this is the "secret project" you're working on, due for release in a few weeks... :)