Results 1 to 33 of 33

Thread: SB v3 in Active Development

  1. #1

    SB v3 in Active Development

    The next generation of the Seventeen or Bust network, or Version 3, is under active development. I'll cut right to the chase. First, some design goals.

    Open development.
    The new system will be almost entirely open. Like distributed.net and most other DC projects, parts of the code will still be kept closed for security, but the vast majority of it will be released publicly. We want the development process itself to be open, too. At some point, hopefully sooner rather than later, we'll set up a development web site with news, anonymous CVS access and, I'm guessing, a Wiki. People that have been developing third-party tools for SB will have a chance to actually contribute to the code itself. It's better for us, and we think it's better for you too, but that's up to you guys to decide. We'll probably set up a sub-forum for development and testing here on FreeDC. In addition to being morally satisfying, I think opening development will be a lot of fun.

    Modularity.
    We want the client to be extensible. Right now, for example, the sieving and P-1 factoring sub-projects are sub-projects because the current client isn't able to accommodate anything other than Proth testing. We don't like this. The new client will use a plugin-based architecture to do its work. When you download the client, you'll be downloading a framework, which contains the user interface, network code, disk cache code, etc., that can run regardless of what kind of work the client is actually doing. Then the Proth testing code will become a plugin. The sieve code might become another plugin, although we have not yet approached the coordinators of those projects so I can't say that with any certainty. Same with the P-1 factoring sub-project.

    Portability.
    This next bit is going to shock some people. Please sit down before continuing. Are you sitting? Good. We're writing most of the client in Java. We know it's hard on resources. That's why I didn't say “all” of the client. The worker plugins make use of Java's “JNI” (Java Native Interface) support. It allows Java code to interact with low-level native code, like C and assembly. Using this, we can pull of a highly-portable and structured client framework without sacrificing performance. Each worker plugin (say, the Proth test plugin) will be available for several operating systems. For example, when you download the Windows client bundle, you get the Java framwork, plus the Windows native version of the Proth test code, which is of course written in assembly. When you download the Linux bundle, you get the Linux version of the plugin instead, and so forth. So, even though the worker plugins still have to be ported from platform to platform, the framework itself will only need to be written once.

    Exploration.
    We want to explore new approaches to distributed computing. As far as we know, no major DC project has ever attempted such a modular, portable design. I know it may sound crazy to some of you, but I really think it will work out very well.

    As far as a time table for development, we don't really have one at the moment. But we do have some broad stages in mind – an order in which we want to do things.

    The first stage is to complete the skeleton of the client framework. We'll need to re-implement the networking code, configuration, disk cache code, and so on. We'll also need to convert the current number-crunching code to a JNI library so that it can be used as a worker plugin. We will NOT be writing the GUI right away – the initial versions of the v3 client will be console-only.

    We'll also need to rewrite, or at least seriously modify, the server, to accommodate new features being introduced into the new client.

    When the console-only skeleton client and server are done, we'll make a “public” alpha release. The new server will run on a different port, and anyone interested in doing so can test out the new client.

    Throughout alpha testing, we'll be working on the user interface, JNI worker plugins for different platforms and, possibly, different tests (sieving? P-1 factoring?). The server will no doubt gain some meat in this second phase as well.

    Once the client's user interface and more robust features are finished, it'll go into beta release. During beta testing, we'll be working on the public web site and the statistics engine. We have some ideas of things we can do to make the stats a lot more accurate, more interesting and more informative. We'll also take this opportunity to make the web site more attractive to potential non-technical users.

    Finally, with the web site modifications done and having had a bit of time for beta testing to weed out the bugs, we'll do the official 3.00 stable release.

    That's about all I need to say about it right now. I'm sure a lot of you will have a lot of questions and I'll try to answer them as best I can.

    --
    David Norris
    danorris@seventeenorbust.com

  2. #2
    One parting note. Some of you may be wondering why it's called "version 3" when it really seems like version 2. To us, it really is version 3 -- it's the second major rewrite of the system. The first major rewrite did not, unfortunately, involve the client, which is why it remained at version 1.x. The server and web site are both in their second lives right now, so for consistency the current development cycle -- including the client -- is being called "version 3."

  3. #3
    What will you do now that the Java Virtual Machine is no longer available? I the Java Runtime Environment software I have used is inadequate in my opinion because it will conflict with things like Java chats on the web and will not load them properly. Perhaps I am just not understanding how to switch from the JVM to the JRE, but I don't think the JRE will work with some computers if I am using it right.

  4. #4
    Unholy Undead Death's Avatar
    Join Date
    Sep 2003
    Location
    Kyiv, Ukraine
    Posts
    907
    Blog Entries
    1

    wow

    how about sourceforge.net? they have all the infrastructure for development.

    and few Q about future client. Very useful feature for many of us is a possibility to run client without any trace to user who works on this PC.
    second feature is a possibility to works behind any sorts of firewall. usually firewall filters everything but http packets. so client must communicate through stangart 3128 or 80 port and use standart GET requests.
    wbr, Me. Dead J. Dona \


  5. #5
    Moderator ceselb's Avatar
    Join Date
    Jun 2002
    Location
    Linkoping, Sweden
    Posts
    224

    Re: SB v3 in Active Development

    Originally posted by kugano
    As far as we know, no major DC project has ever attempted such a modular, portable design.
    Have you looked into cosm? It's a framework for distributed projects, used by folding@home, genome@home and EON.

  6. #6
    Originally posted by Kugano
    As far as we know, no major DC project has ever attempted such a modular, portable design.
    I would also look into BOINC, since it can help making users aware of the project in addition to actually being such a framework.

    http://boinc.berkeley.edu/intro.php

  7. #7
    Sieve it, baby!
    Join Date
    Nov 2002
    Location
    Potsdam, Germany
    Posts
    959
    Originally posted by Jwb52z
    What will you do now that the Java Virtual Machine is no longer available? I the Java Runtime Environment software I have used is inadequate in my opinion because it will conflict with things like Java chats on the web and will not load them properly. Perhaps I am just not understanding how to switch from the JVM to the JRE, but I don't think the JRE will work with some computers if I am using it right.
    As far as I get it, you mean the Microsoft VM with JVM and the Sun version with JRE, right?
    Well, I wouldn't use Microsoft's VM anyway and stick with the JRE (JDK is not needed for most people). Maybe 3rd party VMs (e.g. IBM).
    Basically, the MS JVM is sometimes incompatible with the original Java standard. Some Java Applet writers seem to use it, though.

    When using Java applications (which I wouldn't directly compare to Applets), the JRE should provide the least problems to my experience.

  8. #8
    Could a method of incorporating double checking into the main effort be coupled with this new release? It would most likely be best if the controlling of the level that the double check effort processed at was done by the administrators of the project. Then a most beneficial level could be determined and the project could proceed at the highest rate. To obtain this we'll need some accurate statistic on error percentages. Do any of these exist?

    It would also be nice to be able to obtain stats credit for the work being done on the secret account.

  9. #9
    Could a method of incorporating double checking into the main effort be coupled with this new release? It would most likely be best if the controlling of the level that the double check effort processed at was done by the administrators of the project. Then a most beneficial level could be determined and the project could proceed at the highest rate. To obtain this we'll need some accurate statistic on error percentages. Do any of these exist?

    It would also be nice to be able to obtain stats credit for the work being done on the secret account.
    The "secret" account won't exist in the new system -- there won't be any need. Preferences like that can be handled serverside. Although, it might still be nice to have the ability to control what kind of work you want to do in the client itself. We're not sure how, if at all, we want to handle this. But if we did, I guess it would be something like a set of "constraints" or "preferences" you configure for the Proth test worker plugin. Maybe you could say, "I refuse to test n values less than / greater than X," or "I prefer doing new tests but I won't refuse a double-check if that's the only work available." The server can then use those preferences, together with its own algorithm for determining the best work to be done, to assign tests.

    We'll open more detailed discussions on things like this once the development Wiki / message board is up and running. But all are good ideas and I'm glad people are talking!

  10. #10
    Former QueueMaster Ken_g6[TA]'s Avatar
    Join Date
    Nov 2002
    Location
    Colorado
    Posts
    184
    Do you want to see the source code for my queue? Can/should I make it open source now?

    I've also been thinking about making a server for the SB client, so that people can run their own PRP tests more easily on multiple computers. Will the new client use the old protocol and/or will the old client still be available? Or are you making the server open source, too? (Good idea if you are.)
    Proud member of the friendliest team around, Team Anandtech!
    The Queue is dead! (Or not needed.) Long Live George Woltman!

  11. #11
    Senior Member
    Join Date
    Jan 2003
    Location
    UK
    Posts
    479
    I guess to add to the discussion and reiterate what others have said.

    Open source. Great, anything that allows more people to contribute to the project has to be good. But make sure you have the time to test, integrate and release new versions. There is a fine balance between releasing buggy code every five minutes, and taking so long to integrate and release some great feature on which someone has been working every waking hour, that he gets so pissed off that he is never seen again.

    Portability. Great, best way of getting every cycle that’s going. But keep a sense of reality, >90% of work on this project is with Windows PCs – that should always be the first priority.

    Java. Well this is the current favourite for schools and universities, so there shouldn’t be any lack of bodies that can understand and contribute to the code. All languages have their pros and cons, and this one is a reasonable choice. Only comment I’d make is the run time environment is about 15MB – yes you only need to download this once, then the client should be quite small, but this will deter dial-up users big time.

    Combining sieving and factoring in the client. Great idea, gives the whole thing a bit more exposure.

    Cosm and BOINC. Since you are talking about starting again from scratch(ish), worth a look at very least. I briefly looked at the BOINC stuff many months ago, and the bottom line is that you still have to do most of work yourself, the BOINC bit really just provides an introducer, a method to allow users to pick projects, and common stats. The only real advantage I see with BOINC is you’d pick up some cycles from the mass of SETI users, particularly if you’re the first prime number project on BOINC. But that’s all it would be, a little share of those cycles. And then you are always at the mercy of the big controller, my guess is SB would prefer to control it’s own destiny (and that’s fine).

    Firewalls, proxies and PCs not connected to the net. Don’t forget these. There is a wealth of resources that fit into this category just idling right now. Make life easy for people to use them on SB and they’ll be used. It’ll be great to have sieving included in the client, but if that’s then the only way to get work and submit results, that kills about 90% of my sieving. Sieving and P-1 factoring have been a great way of using unattended PCs which are on 24/7 but don’t have net access. In fact, why not extend this concept to allow PRP tests to be obtained and results submitted through a web interface – if work unit progress could be reported as simple HTML page query extensions (http://www.sob.com?progess=SomeRando...ingToTheServer), the client could output this is as a shortcut, then the user would only need to click on a few short cuts every week to keep a load of PCs happy. Of course dedicated proxies such as that from Ken_g6 are also very important, and it's these types of add-ons that give projects like d.net such a huge user base.

    Profile of work. As you suggest, let the user choose what they want to do, the user knows best! But you must allow this configuration to be on a PC by PC basis. To quote me as an example, I have a P4 that stays on 24/7 that I always want to do new PRP test, but I have a slow Celeron laptop where I manually kick off the odd DC test. If I ever got a new test on the laptop I’d just stop it as soon as I noticed, so all those cycles would be wasted.

    Abandoned PRP tests. As the tests get ever bigger, maybe a bit of lateral thinking is required here, an abandoned test at 50% complete with n=10M is a lot of wasted work. Yes, the save files are heading up towards 1MB, so bandwidth is an issue, but you could allowed abandoning users the option to upload the save file to SB (and maybe only take them if they are more than x% complete). I guess that could be another option for users “I am willing to take partially complete units, and to share the credit if I find a prime…”

    Torture test. GIMPS dictates that a new PC performs a torture test before doing any real work. This gives reasonable confidence that the PC is in good shape before doing any work. Maybe SB should do something similar, or maybe (stealing a little from GIMPS forums) a new PC should perform a few DC tests before being allowed to perform new tests. This would have a secondary advantage in that fickle users that give up after a week will at least have contributed something useful to the project. Maybe even enforce a rule that (say) once a month all PCs have to take a DC test (just in case things have physically changed). This would give a very high degree of confidence in all new test results.

    Cache of work units. This is essential. Ken_g6 appears to have this working with his proxy (yes?) Dial-up users exist, servers go down, but these don’t need to be reasons for the clients running out of work.

  12. #12
    Senior Member wirthi's Avatar
    Join Date
    Apr 2002
    Location
    Pasching.AT.EU
    Posts
    820
    Originally posted by Jwb52z
    What will you do now that the Java Virtual Machine is no longer available? I the Java Runtime Environment software I have used is inadequate in my opinion because it will conflict with things like Java chats on the web and will not load them properly. Perhaps I am just not understanding how to switch from the JVM to the JRE, but I don't think the JRE will work with some computers if I am using it right.
    I'm no expert on this so correct me if I'm wrong, but the JRE includes a full JVM. Do you mean the fact that Microsoft doesn't publish an own JVM any more? That has never been an official JVM.

    I've (almost) never had any problems with the JRE from http://java.sun.com/

    Greets,
    Wirthi

    (NOTE: This message was accidentally overwritten by me (kugano / David) because I clicked 'Edit' when I meant to click 'Quote.' I restored the message to its original contents; my sincere apologies to the author!)
    Last edited by kugano; 01-09-2004 at 10:37 PM.
    Engage!

  13. #13
    Do you want to see the source code for my queue? Can/should I make it open source now?

    I've also been thinking about making a server for the SB client, so that people can run their own PRP tests more easily on multiple computers. Will the new client use the old protocol and/or will the old client still be available? Or are you making the server open source, too? (Good idea if you are.)
    It's entirely up to you, of course, whether or not you want to open your code. We'd certainly benefit from seeing or even using your existing stuff. We'd of course like to see your queue, or at least the functionality it provides, included as a built-in part of the client so that users don't need a third-party tool. Rather than "run you out of the market," so to speak, it'd be cool if you wanted to develop stuff for the actual client.

    The new client will use an entirely new network protocol. It will, however, support clients using the old protocol. We'll hopefully remove support for the old protocol someday, but we have no choice but to keep it around until an acceptable percentage of our users have upgraded.

    Louie and I haven't decided on opening the server code. I have a feeling we will, but that certain parts (network protocol, and possibly some of the assignment logic?) will be held back for security. Don't quote me on this though =)

  14. #14
    A few comments on the "JVM" issue. Sun provides something called the Java 2 Runtime Environment, or JRE. The JRE is the collective name for both the Java virtual machine (JVM) *together with* all of the libraries that fulfill the API requirements of the Java environment. These libraries come in the form of a bunch of JAR files (Java ARchives, which are really just glorified ZIP files) that contain code essentially equivalent to the C standard library.

    I think the confusion arose because Microsoft distributed something called the "JVM." This was really a misnomer; it was really a JRE, because it contained not only the virtual machine, but also the standard libraries. They called it a JVM because, well, they're Microsoft.

    Anyway, this is a non-issue. The reason is that we'll be distributing the Windows client inside of a self-extracting package (there are several freeware Java on Windows packagers out there). I'm not sure if this package will actually include Sun's JRE, but I doubt it; however, the packager will no doubt take care of checking for a "compatible" JRE and, if none exists, either helping the user install it or telling them how to do it.

    On non-Windows platforms, it's generally the case that the user is experienced enough to install Sun's JRE themselves, but we can provide instructions on our FAQ. It's a simple process and merely requires the user to download a self-extracting binary, execute it, type "yes" to agree to the license, and set an environment variable.

  15. #15
    I guess to add to the discussion and reiterate what others have said.

    (snip, snip, snip, ...)
    Wow. Good ideas and suggestions all around. A few replies and/or additions:

    Portability -- I bet the percentage of cycles contributed by Windows users is a bit higher than 90%, actually. And your point is well taken. But a lot of people have been asking a long time for ports for various architectures and operating systems. Historically, there have been two things standing in the way of doing so -- the user interface, and the test code. Either one by themselves is less of a menace than both at the same time, especially when they have to be redone for *every* port, not just one. Java knocks one of them out of the way, and it encapsulates and modularizes the portion that does need to be ported, thus making it easier to do. The test plugin will have a formal specification and a reference implementation (namely the current x86 implementation).

    Java and the size issue -- True. I've been playing this redesign through in my head as a long-term solution, though. SB has made some great progress, far beyond Louie or I's expectations, and I hope it continues. But the fact is, I'm sorry to say, we're going to be here a while. We have eleven multipliers left to eliminate, and we won't be wrapping them up for a long time. So I'm trying to be as foreward-thinking as I practically can. JREs will become more commonplace and less painful to install. Bandwidth will increase. Computers will become powerful enough to minimalize the performance penalties of Java. I want to write something that will be useful for a long time, and I think Java is a promising possibility for achieving that, since it tends to lead to more maintanable and scalable applications.

    Cosm and BOINC -- I'll skip through and just add a *nod* after "... my guess is SB would prefer to control it's own destiny ..." I erroneously suggested in my original post that we would be the first project tackling an "abstract" distributed computing framework. Certainly not. But we will be putting into practice some new ideas and new solutions to old problems, and for me in particular and I'm sure Louie as well, That's Cool.

    Firewalls, proxies and PCs not connected to the net -- We won't forget. That's one of the areas I'm hoping will benefit from an open development process. You guys can tell us how you need it to work to get cycles out of all the machines you can, and we'll do what we can.

    Torture test, requiring double-checks first -- I really like that idea.

    Cache of work units -- This is one of the few parts of the new client that is actually already written, tested and in CVS =)

  16. #16
    Junior Member bagleyd's Avatar
    Join Date
    Mar 2003
    Location
    Northern New Jersey - US
    Posts
    9
    Well whatever you do please don't kill the strong points about sb, i.e. fast and easy to use. IMHO, Java does not live up to its expectations.
    Also I would only include a torture test if the time spent doing the test is actually worth it (ie don't solve a problem when there isn't one).
    Last edited by bagleyd; 01-10-2004 at 09:15 PM.
    Cheers,
    David Bagley

  17. #17
    Will the new client still be able to be used as service?

  18. #18
    Team Anandtech
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    50
    The only thing I request is that the client remains small and fast... not one of those bloated ones like in other projects, whose names I shall not mention. At present SoB starts in under a second, with no unnecessary caching, and is completely unnoticeable when running in the background. I'm not sure how a shift to java would affect that, but I'm hoping it won't make much of a difference.

    SoB is, in my experience, one of the easiest DC clients to install. Run the installer, enter your username and click start. Simple

    So there's my request. New features are all well and good, but please don't let them come at the expense of the already established speed and ease of use we've come to love about the client

  19. #19
    Junior Member
    Join Date
    Nov 2003
    Location
    The Netherlands
    Posts
    22
    If there will be a new client I would like it if it wouldn't store its settings in the registyr anymore, but in a config-file in the programfolder instead.

    Would make it a lot easier to for example take the client with me on a USB key and run it on whatever PC i'm working and connecting to the server when I'm on a PC that has internet.

  20. #20
    If there will be a new client I would like it if it wouldn't store its settings in the registyr anymore, but in a config-file in the programfolder instead.
    Hehe, and you're not the only one. Using the registry has been a very unpopular decision. The new client will use an XML-based configuration file that, by default, stays in the same directory as the client's JAR file, but can be placed wherever you want. All information about work units will go in a cache directory that can be changed in the XML configuration file.

  21. #21
    After this is done, are you saying that if someone wants to write a client for a non-x86 system, i.e. the piece doing the PRP test, they will have to do it themselves or are you saying that it is your goal to modify an existing FFT, such as YEAFFT or FFTW into a client to be run on other CPUs?

  22. #22
    That's too far ahead to make any certain plans. The point of the modularization is to give us options. It separates the portion of the code that needs to be ported. It may still be us that end up doing it, or it may be a contributor, but whoever it is, it'll be a lot easier than it would be with the current client design.

  23. #23
    Will there be a function that supports HT in the new client?

  24. #24
    Will there be a function that supports HT in the new client?
    Probably not, at least not right away. Hyperthreading doesn't do much for a single Proth test, because the algorithm can't be parallelized. But it could help if the client was able to do something else at the same time, like sieving. I think some others on the forums did some experiments with combining sieving and factoring or something to see if hyperthreading really helps; maybe one of them can comment?

    But at any rate, it'd be cool, down the road, for the client to be fully multithreaded so you can have it run more than one test at once. That way, people with SMP systems can take full advantage of it instead of having to run two instances of the client like they do now.

  25. #25
    Sieve it, baby!
    Join Date
    Nov 2002
    Location
    Potsdam, Germany
    Posts
    959
    Originally posted by kugano
    I think some others on the forums did some experiments with combining sieving and factoring or something to see if hyperthreading really helps; maybe one of them can comment?
    ATM, I'm doing sieving and PRPing on a HT-enabled P4. I guess doing one of each is the best way, as both applications use the resources to different extends, although I haven't tested this theory yet.
    I hope I can check it in the next days...

  26. #26
    Just a quick note: I switched the SeventeenOrBustWiki to a different Wiki engine. It seems to be a lot more robust, and I'm liking it a lot better. Anyway, the upshot of this is, of course, that the pages that had been put up on the old Wiki are now gone. I tried to recreate most of them at least skeletally, but there's still some stuff that went bye-bye. My apologies to everybody who used it.

    I promise I won't do it again; at least, if I need to switch engines again, I'll figure out a way to transfer pages from one to the other so no information will be lost =)

    The link to the Wiki again is <http://wiki.seventeenorbust.com/>. Enjoy.

  27. #27
    Originally posted by kugano
    Just a quick note: I switched the SeventeenOrBustWiki to a different Wiki engine. It seems to be a lot more robust, and I'm liking it a lot better. Anyway, the upshot of this is, of course, that the pages that had been put up on the old Wiki are now gone. I tried to recreate most of them at least skeletally, but there's still some stuff that went bye-bye. My apologies to everybody who used it.

    I promise I won't do it again; at least, if I need to switch engines again, I'll figure out a way to transfer pages from one to the other so no information will be lost =)

    The link to the Wiki again is <http://wiki.seventeenorbust.com/>. Enjoy.
    Looks way better,

    great job

  28. #28
    Unholy Undead Death's Avatar
    Join Date
    Sep 2003
    Location
    Kyiv, Ukraine
    Posts
    907
    Blog Entries
    1

    where is it?

    wbr, Me. Dead J. Dona \


  29. #29
    we got a better server and are moving to it.

    Cheers,
    Louie

  30. #30

    Suggestions for 1.25 before 3.xx release

    I am using the latest client but wanted to suggest some
    small changes - before a full 3.xx. release

    1) Could you provide more options of sending confirmation blocks.
    Currently it seems to only be set to every block or end of test.
    Specifically, I would like to send once every 5 blocks or so.
    This would reduce the pauses between each block (which are
    recently significant [30 seconds or so] and network traffic for
    both me and more importantly for the main server.

    2) Could you provide the ability to change the update frequency
    for the status screen (windows version).
    Currently it seems to update something once or more every second,
    I would prefer once every 10 seconds, or even setable to minutes.
    This would save some processor time, but also be less distracting
    for indidivduals that like to keep the status box up all the time.
    (For status updates, I personally prefer less granuality; for numbers
    less decimals places displayed, and for time seconds not listed)

    3) During the time the process connects to the main server and
    send results it does not continue calculating. Anyway this can
    be seperated out into a seperate thread. In my setup, it takes
    oftens takes 10's of seconds or more to complete the connection
    and communications....time spent not working on the project.

    Provided that the main calculations are well optimized, optimizing the intra-block or wraping code might be the next best place to optimize overall
    performance.

    Regards,

  31. #31
    Unholy Undead Death's Avatar
    Join Date
    Sep 2003
    Location
    Kyiv, Ukraine
    Posts
    907
    Blog Entries
    1
    wbr, Me. Dead J. Dona \


  32. #32
    Senior Member
    Join Date
    Jan 2003
    Location
    UK
    Posts
    479
    Just a reminder. The new server is up and running, but

    http://wiki.seventeenorbust.com/ points to http://www.seventeenorbust.com/ .

    There seem to be lots of new users who have lots of new ideas and suggestions. Recording them in the wiki again seems like a good idea.

  33. #33
    Junior Member
    Join Date
    Nov 2003
    Location
    The Netherlands
    Posts
    22
    wiki.seventeenorbust.com send me to the website, not the wiki

    Might be my DNS is not updated yet if you recently changed it but I thought I'd mention it in case it is an error.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •