RNA World

[Michael H.W. Weber]

Please just add another project in parallel. That will allow us to complete this set of WUs (which in fact is an extremely cool project, just read our scientific objectives section on this series) and your machines won't go idle as well. In the meantime, the important Linux x64 CMC set will be completed (it is nearly done now) and thereafter we will have plenty of long WUs plus I will add the first analyses of the human genome which are currently in preparation.

Michael.

[LAURENU2]

Please just add another project in parallel.
Michael.

RNA is one of 5 projects that I am running now at the same time
Look below at my sig

But I have noticed that RNA on some Nodes Runs like a weak puppy
and lets other projects eat up most of the power

I was Not going to Detach from RNA But rather pull back on the amount of the power ported to RNA
I got the feeling by the response time from your Server that it was being overwhelmed
Turning down my power would Fix it I can pump out a lot of bytes from my Garage

[Angus]

The length of the WUs is ridiculous, and to increase the time between requests is an insult.

I watched 8 or 10 WU start to download last night. The first 6 were done and uploaded before the others even finished downloading.

There's something basically wrong with their design if they can't produce a WU that lasts for more than 30 seconds.

Michael - I looked at your reference in the wiki about the current project -

Project: CRISPR

* The CRISPR elements are part of a prokaryotic defence system directed against external attacks by e.g. viruses and may be viewed as a simple immune system of microorganisms.

By employing RNA World to systematically screen organisms for the presence of the various types of this defence machinery, we hope to acquire important information on the global distribution and varieties of this system. There is an enormous repertoire of potential applications to the results of such analyses ranging from the improvement of industrially relevant microbial food production to novel ways of coping with multi-drug resistant pathogenic bacteria.

How is this supposed to explain why the WUs have to be so short???? Can you explain this in humanly understandable English?

[zombie67]

They already explained it, and a fix is in the works. Did you not read the post where they said that they are going to apply a filter to avoid sending out the very short tasks?

[Michael H.W. Weber]

How is this supposed to explain why the WUs have to be so short???? Can you explain this in humanly understandable English?

Yes, it simply is a result of the low complexity of this WU series. Moreover, with these analyses it is pretty hard to say which WU will take a lot of time and which not unless you run a test simulation on each of these and even then the true results will vary from expectation.

Michael.

[Angus]

They already explained it, and a fix is in the works. Did you not read the post where they said that they are going to apply a filter to avoid sending out the very short tasks?

Not trying to be pissy here :

I don't see that they explained WHY the WUs were so short, nor is there any resolution in place at this time.

I see Michael has sort of explained it now as "low complexity".

I find it interesting that they didn't know up front that this batch of WUs would be so short. I would think that they would run some sort of basic sanity check on the WUs they were generating to see if they even worked before releasing them to the wild, and that would have revealed the short execution time problem.

[gopher_yarrowzoo]

Angus "Not trying to get pissy here...." Oh really I think you are again.
Now this project is still in TEST phase and thus short / long WU's will occur they need to test server load and balancing..
What has WU Length got to do with ANYTHING anyway apart from wasting time waiting on the next one...
Like Micheal has suggested paralleling it with other stuff, so maybe you should do just that and relax a little!

[LAURENU2]

You got to remember This is still a Baby Project
It is just stating to WU-lk on it's own
You have to expect a stumble or two before it can run fast and smooth

[Angus]

Angus "Not trying to get pissy here...." Oh really I think you are again.
Now this project is still in TEST phase and thus short / long WU's will occur they need to test server load and balancing..
What has WU Length got to do with ANYTHING anyway apart from wasting time waiting on the next one...
Like Micheal has suggested paralleling it with other stuff, so maybe you should do just that and relax a little!

I was trying to make a response to zombie67's snarky comment about not reading another post that was unrelated to my question about WHY the WUs were so short.

As for being a "test" or "alpha" or "beta" project, that would be most BOINC projects.
Only 8 have identified themselves as "Production" projects. Project status


Also, both Michael and Yoyo have been around BOINC long enough to understand that releasing 300,000 very tiny WUs will swamp their server. Anyone who has tested with 'uppercase' exposed to the public has seen that behavior.

So, indeed, WU length has everything to do with how busy the server and database will be.

[zombie67]

I was trying to make a response to zombie67's snarky comment about not reading another post that was unrelated to my question about WHY the WUs were so short.

My post had nothing to do with your "why" question. It was purely a response to your bitching about the short WUs. You acted surprised and "insulted" that you were getting short WUs. If you had read the previous posts before posting, you would have known that they were aware already of the problem and working on implementing a solution.

[gopher_yarrowzoo]

My post had nothing to do with your "why" question. It was purely a response to your bitching about the short WUs. You acted surprised and "insulted" that you were getting short WUs. If you had read the previous posts before posting, you would have known that they were aware already of the problem and working on implementing a solution.

Zombie, IMHO sometimes is not worth trying to explain stuff to people if you don't give them the answer they expect see it all the time in my line of work...

Reminds me I really should pop over to your forum and say hi again

[Michael H.W. Weber]

So, indeed, WU length has everything to do with how busy the server and database will be.

Indeed. But it is a big difference between just knowing this and measuring the exact impact on your own machine in practice. And that's just what we did.

Michael.

[LAURENU2]

Well I hope you got the the exact impact on your own machine in practice I know I did
I never did see so many blinking lights before except for Jeff's X Mass display
By the way he is better at it then you Jeff's lights

[Bigred]

I'm glad somebody is getting some work because I sure ain't.

[LAURENU2]

I'm glad somebody is getting some work because I sure ain't.

Like I said

But I have noticed that RNA on some Nodes Runs like a weak puppy
and lets other projects eat up most of the power

When run with other projects you sometimes have to make the client get work by kicking the Update button
RNA seems to want to go to sleep if there is no work sent for a bit

[Michael H.W. Weber]

RNA seems to want to go to sleep if there is no work sent for a bit

The project should actually check every 20 minutes for work.

Maybe you have noticed that the average CMS run times now slightly increase. My 955 BE does not spend only a few seconds on the WUs but up to 15 minutes. This should persist for the remaining WUs that are still in the pipeline.
Concerning the two Linux x64 WU packets, around 35 and 63 very long WUs remain to be completed. After that we will have plenty of work to do at all ends.

Michael.

[LAURENU2]

Your project is still overwhelming your server and my network
I never did see so many blinking lights before you might think it was Las Vegas

[Michael H.W. Weber]

I never did see so many blinking lights before you might think it was Las Vegas

The current set of CMS WUs should be complete, soon. Still, some Linux x64 CMCs remain in the pipeline. And these are monsters... So, beware...

Michael.

[Bigred]

Maybe someone should turn the cms_validator back on. There are only about 42,000 workunits waiting on it.

[Bok]

Hi Michael,

I still have two monster wu's running on a 64bit linux box. They are both at 100hrs now and ~ 90% complete, but their deadlines were over this past weekend. I'll let them finish, just curious as to whether credt will still be applied?

Bok

[Angus]

Maybe someone should turn the cms_validator back on. There are only about 42,000 workunits waiting on it.

Since the validator isn't running, and credits are pending, does this impact the WU being marked as "completed" ? And, if the WU isn't completed, are the crunchers processing all 3 of the initially issued results, instead of meeting the quorum with two valid returned results, causing a 50% increase in server load to download and upload and process those extra results?

Edit: Never Mind. It appears that they have changed to initial replication setting to only 2. Good choice.

[Michael H.W. Weber]

@Angus: The current settings are efficient in avoiding the MySQL attack, so server issues are not seen anymore. We now just wait to complete the CMS WU set and, as said before, after that we will bring all setting back to normal on the server side and this type of problem should never occur again as we have found an efficient measure to avoid this accumulation of short WUs.
Concerning the the WU redundancy, yes, with this set we have more results to validate (and to credit of course) than usually, because as long as the validator is halted, no "WU already complete" signal will be sent to the clients to abort non-started computation efforts. This is clearly not so nice because it causes overhead in computational efforts, but the WUs are very tiny, so we think it should ot bother us too much. Just consider other projects that do not even make use of this possibility to reduce "WU redundancy".

@All: We seem to have identified a yet unknown problem with some of the long running CMC WUs. First, it seems they run about double the time which is indicated initially by the progress bar. This is unavoidable due to the type of computations we carry out and the unreliability of the progress bar in case of CMC WUs increases with their complexity (i.e. duration). So on the basis of the progress indicator, just do not make the decision to abort a WU. If possible, keep it running, please. The real baffling thing is that some of the long WUs are judged as "client error". Strangely, they seem to be intact (at present an assumption). We are currently checking that more in detail and we need more data to identify the source of the problem. So, please help us a bit in finding out what goes wrong by not giving up to run these WUs such that we can analyze the outcome one by one.

Michael.

P.S.: I am surprised that we encounter such an issue at this stage since I was quite convinced things work fine. But, of course, we focused on the smaller CMC WUS to test a lot in a short period of time. Now we released the "big birds" after all that testing and here we need to do some homework again. I have my 955 BE on these long WUs, so at least we are sitting in the same boat and that boat I like.

[LAURENU2]

The current settings are efficient in avoiding the MySQL attack, so server issues are not seen anymore.

Now that I have Pulled back to 40% I do see your sever a bit more responsive
What are your plans to handle a increas in power input to RNA

[Angus]

Michael

Can you explain what is being discussed at the bottom of this thread?

http://www.rechenkraft.net/phpBB/vie...p?f=76&t=10706

It's all in German, and the translation services don't speak well enough German to any sense of it make.

[Saenger]

Michael

Can you explain what is being discussed at the bottom of this thread?

http://www.rechenkraft.net/phpBB/vie...p?f=76&t=10706

It's all in German, and the translation services don't speak well enough German to any sense of it make.

It's about a WU of me that once was crashed by my computer after 125h as the puter crashed, and was finished, but declared erroneous after 144h of the second run.
So far 3 others have finished, all with the same error as well, and all seemingly with the same output file uploaded. If it wouldn't be for the computational error, they would probably be declared valid.

Ananas and Michael are looking into that issue, what had happened, how to avoid it in the future and probably what to do in this concrete case.

As I'm no programmer nor biology scientist, "just" a mechanical engineer, I can't really say more useful stuff

[Michael H.W. Weber]

We are on track of the issue. I figured that for yet unknown reason the client does not write the result data into the output file. Instead, it uploads the input file as result which is logical since the input file is overwritten with the same file name as soon as processing is complete. What puzzles us is the fact that this problem is occurring only with a subset of WUs. Since these run perfectly on my local machine, I suspect a problem with the wrapper. But we do not know for sure at present. The logs tell us that within the last step there is a break in processing. If you have ideas, just post them here. It might help us!

Michael.

[Bok]

One of my monster jobs finished and is marked as 'too late to validate'...

http://www.rnaworld.de/rnaworld/resu...ames=0&state=4

140 hours wasted? The other one is at 142hrs right now and 93% complete. I'm not too bothered about points, just hate to see any result go unused..

Bok

[LAURENU2]

One of my monster jobs finished and is marked as 'too late to validate'...

http://www.rnaworld.de/rnaworld/resu...ames=0&state=4

140 hours wasted? The other one is at 142hrs right now and 93% complete. I'm not too bothered about points, just hate to see any result go unused..

Bok

But The Claimed credit of 1,150.47 points sure would help in your Quest of World Domination

[Michael H.W. Weber]

One of my monster jobs finished and is marked as 'too late to validate'...

http://www.rnaworld.de/rnaworld/resu...ames=0&state=4

140 hours wasted? The other one is at 142hrs right now and 93% complete. I'm not too bothered about points, just hate to see any result go unused..

Bok

I will see whether this WU can be manually re-entered for validation by Yoyo to see whether it was computed OK or still shows an error. For now, please run on your Linux box the following command if your WU has reached more than 95%:

strace -p <pid of cmcalibrate> | tail -1000000000 > strace.out

This will create a trace file with a maximum size of 1GByte. This might be useful to find the error.

Michael.

[Michael H.W. Weber]

And please also try this one and post the output here:

ulimit -a

Thanks!

Michael.

P.S.: Except for 4 WUs the CMCs are now completed. We tweaked a bit again on our HR settings and by doing so improved the project's throughput.

[Bok]

[root@dual275 ~]# strace -p 3551 | tail -10000000 > strace.out
Process 3551 attached - interrupt to quit
times({tms_utime=52889500, tms_stime=2708, tms_cutime=0, tms_cstime=0}) = 542291 108
lseek(7, 0, SEEK_SET) = 0
write(7, "0.939167", 8) = 8
times({tms_utime=53009664, tms_stime=2711, tms_cutime=0, tms_cstime=0}) = 542411305
lseek(7, 0, SEEK_SET) = 0
write(7, "0.940000", 8) = 8
times({tms_utime=53130244, tms_stime=2715, tms_cutime=0, tms_cstime=0}) = 542531933
lseek(7, 0, SEEK_SET) = 0
write(7, "0.940833", 8) = 8
times({tms_utime=53250881, tms_stime=2719, tms_cutime=0, tms_cstime=0}) = 542652595
lseek(7, 0, SEEK_SET) = 0
write(7, "0.941667", 8) = 8
times({tms_utime=53371605, tms_stime=2724, tms_cutime=0, tms_cstime=0}) = 542773351
lseek(7, 0, SEEK_SET) = 0
write(7, "0.942500", 8) = 8
times({tms_utime=53491730, tms_stime=2726, tms_cutime=0, tms_cstime=0}) = 542893510
lseek(7, 0, SEEK_SET) = 0
write(7, "0.943333", 8) = 8
times({tms_utime=53612159, tms_stime=2729, tms_cutime=0, tms_cstime=0}) = 543013963
lseek(7, 0, SEEK_SET) = 0
write(7, "0.944167", 8) = 8
times({tms_utime=53732545, tms_stime=2732, tms_cutime=0, tms_cstime=0}) = 543134373
lseek(7, 0, SEEK_SET) = 0
write(7, "0.945000", 8) = 8
times({tms_utime=53852178, tms_stime=2735, tms_cutime=0, tms_cstime=0}) = 543254039
lseek(7, 0, SEEK_SET) = 0
write(7, "0.945833", 8) = 8
times({tms_utime=53972575, tms_stime=2738, tms_cutime=0, tms_cstime=0}) = 543374466
lseek(7, 0, SEEK_SET) = 0
write(7, "0.946667", 8) = 8
etc etc

[root@dual275 1]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16383
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16383
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


This one will finish today at some point, though again the deadline was the 27th. It's at 165hrs right now.

[yoyo]

Hi Bok,
the strace is important at the end of the wu if it fails.
yoyo

[Bok]

no problem, strace is still running

wu is at 99.352% after 168hours.

[LAURENU2]

Michael

Results ready to send 1 we need more work

Results in progress 1,585

Workunits waiting for validation 13,296 are these all mine

[Angus]

I see over 40,000 results (tasks?) still waiting for validation, but that's about half of what it was yesterday, so the validator is crunching its way through the pile of work you gave it.

[yoyo]

no problem, strace is still running

wu is at 99.352% after 168hours.

I think this one finished and validated.
yoyo

[Bok]

yup. Just noticed myself. Here is the end of the strace if you are interested.

times({tms_utime=61225425, tms_stime=3868, tms_cutime=0, tms_cstime=0}) = 550682189
lseek(7, 0, SEEK_SET)                   = 0
write(7, "0.999167", 8)                 = 8
time(NULL)                              = 1267728664
times({tms_utime=61339827, tms_stime=3902, tms_cutime=0, tms_cstime=0}) = 550797050
time(NULL)                              = 1267728664
times({tms_utime=61339827, tms_stime=3902, tms_cutime=0, tms_cstime=0}) = 550797050
time(NULL)                              = 1267728680
times({tms_utime=61341235, tms_stime=3912, tms_cutime=0, tms_cstime=0}) = 550798650
write(1, "    48:43:39\n  filter      -  lo"..., 67) = 67
time(NULL)                              = 1267728680
times({tms_utime=61341235, tms_stime=3912, tms_cutime=0, tms_cstime=0}) = 550798652

mremap(0x2aaaaf386000, 292360192, 511516672, MREMAP_MAYMOVE) = 0x2aaaaf386000
time(NULL)                              = 1267736446
times({tms_utime=62065739, tms_stime=4064, tms_cutime=0, tms_cstime=0}) = 551575236
write(1, "    02:09:26\n# --------  ---  --"..., 1399) = 1399
munmap(0x2aaaaf386000, 511516672)       = 0
write(1, "//\n", 3)                     = 3
read(8, "", 4096)                       = 0
time(NULL)                              = 1267736446
times({tms_utime=62065739, tms_stime=4071, tms_cutime=0, tms_cstime=0}) = 551575245
lseek(8, 0, SEEK_SET)                   = 0
open("cmfile.xxx", O_RDONLY)            = -1 ENOENT (No such file or directory)
open("cmfile.xxx", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 9
read(8, "INFERNAL-1 [1.0.2]\nNAME     Intr"..., 4096) = 4096
read(8, ".497  -5.274  -4.934  0.000  0.0"..., 4096) = 4096
read(8, "10  -1.005  -6.446  -3.975  0.66"..., 4096) = 4096
read(8, "   104   103 3   106     3 -10.1"..., 4096) = 4096
read(8, "MP   141   140 1   145     6 -10"..., 4096) = 4096
read(8, "     D   175   173 3   176     5"..., 4096) = 4096
read(8, "             \n    IR   212   212"..., 4096) = 4096
read(8, "18  -0.006  -9.594  -9.874 -10.2"..., 4096) = 4096
read(8, ".164                          0."..., 4096) = 4096
read(8, " 4  -9.716  -9.923  -0.008  -8.3"..., 4096) = 4096
read(8, " 352 6   355     3 -10.745  -0.0"..., 4096) = 4096
read(8, "MATP  103 ]\n    MP   389   388 6"..., 4096) = 4096
read(8, "  -3.908  0.660 -0.612 -0.293 -0"..., 4096) = 4096
read(8, "                  0.000  0.000  "..., 4096) = 4096
read(8, "  -5.695  -0.829  -3.908  0.660 "..., 4096) = 4096
read(8, ".906                          0."..., 4096) = 4096
read(8, "]\n    ML   568   567 2   570    "..., 4096) = 4096
read(8, "                    0.000  0.000"..., 4096) = 4096
read(8, " 6   644     6  -6.988  -5.717  "..., 4096) = 3245
fstat(9, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaaad000
write(9, "INFERNAL-1 [1.0.2]\nNAME     Intr"..., 4096) = 4096
write(9, " -9.399                         "..., 4096) = 4096
write(9, "  1.652 -2.459  0.633  0.114 -1."..., 4096) = 4096
write(9, "         0.000  0.000  0.000  0."..., 4096) = 4096
write(9, "    6  -7.451  -7.797  -2.512  -"..., 4096) = 4096
write(9, "  151     6  -6.988  -5.717  -1."..., 4096) = 4096
write(9, ".062  0.095 -0.746 -2.041 -2.246"..., 4096) = 4096
write(9, " 0.000  0.000  0.000  0.000 \n\t\t\t"..., 4096) = 4096
write(9, "5  -0.829  -3.908  0.660 -0.612 "..., 4096) = 4096
write(9, "920  -4.087  -5.193          0.0"..., 4096) = 4096
write(9, "3  -1.564  -1.458  -1.748       "..., 4096) = 4096
write(9, "\n     D   363   361 3   364     "..., 4096) = 4096
write(9, "ATL  105 ]\n    ML   398   397 3 "..., 4096) = 4096
write(9, "000  0.000  0.000 \n\t\t\t\t[ MATP  1"..., 4096) = 4096
write(9, " -0.001                         "..., 4096) = 4096
write(9, "       0.000  0.000  0.000  0.00"..., 4096) = 4096
write(9, "649  -6.162  -0.021             "..., 4096) = 4096
write(9, "2 -3.397 -1.054 \n     D   578   "..., 4096) = 4096
write(9, "  3 -10.691  -0.856  -1.162     "..., 4096) = 4096
close(8)                                = 0
munmap(0x2aaaaaaac000, 4096)            = 0
write(9, "408  -0.496  -5.920  -4.087  -5."..., 2190) = 2190
close(9)                                = 0
munmap(0x2aaaaaaad000, 4096)            = 0
rt_sigprocmask(SIG_BLOCK, [INT], NULL, 8) = 0
unlink("cmfile")                        = 0
rename("cmfile.xxx", "cmfile")          = 0
rt_sigprocmask(SIG_UNBLOCK, [INT], NULL, 8) = 0
write(1, "#\n# CPU time: 620657.39u 40.71s "..., 64) = 64
exit_group(0)                           = ?
Process 3551 detached
[root@dual275 ~]# 

[Michael H.W. Weber]

Ok, guys, fasten your seat belts: as noted in our NEWS section, the last two CMS work packets alone returned about 900,000 results with a total traffic of 1 TB.

The upcoming weekend will bring new work, I guess.

Michael.

[LAURENU2]

Ok, guys, fasten your seat belts: as noted in our NEWS section,
Michael.

With this new work can I go back to Full Power Without impacting your server like I did last time?
40% power is No Fun

Have you Beefed up you DB to do Better then it did ?

[Michael H.W. Weber]

Well, the upcoming CMC WUs, even those of the "monster type" should run smoothly now. Although they still do run long. At least we have extended the deadline for these by a whole week. Concerning the upcoming CMS work, we will first send out a small batch to see how it is doing.

Michael.