Wasted Bandwith Brian please Read

[DGROMS.com]

Brian,
A fellow folder over at Overclockers Australia posted this. A very valid point and something that should save everyone dollars to fold with

http://forums.overclockers.com.au/sh...52#post1025352

or

Originally posted by Doc-of-FC
can somone ask brian on the free DC forums CO myself about if he can at least UPX the widows foldtrajlite.exe in the updates.

after doing some fiddeling around myself i have found by UPX'ing the windows DF exe i've shrunk the exe to 1/3 its size
results here.

C:\df>upx foldtrajlite.exe
Ultimate Packer for eXecutables
Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002
UPX 1.22w Markus F.X.J. Oberhumer & Laszlo Molnar Jun 27th 2002

File size Ratio Format Name
-------------------- ------ ----------- -----------
1355776 -> 493056 36.36% win32/pe foldtrajlite.exe

Packed 1 file.

so if he shrunk the 1.3 MB file to 470 KB, wouldn't that make the doze update FASTER and smaller, hmmm.

1000 users all trying to D/L a 1.3 MB file = 1.26 GB
1000 users all trying to D/L a 470 KB file - 458.98 MB

both scenarios over a 1 mbit line, serving the same file to 1000 users and saving 2/3 the original bandwidth. go fig

edit
after doing some more testing i've managed to knock at least 600 KB off the update size.
recreating the self extracting executable in winrar.

[jlandgr]

Hi,
AFAIK foldtrajlite.exe has not been updated, at least during the last few automatic updates. There is the possibility to update it automatically, too, but for bandwidth reasons this is only done when really required.
Just check the date when foldtrajlite.exe was last changed on your hard drive.
Jerome

[Brian the Fist]

compressing the EXE will not make the update smaller, because as you may or may not know, the updates are all zipped with WinZip which does effectively the same thing. It is usually the protein which changes with each update, and these data files are compressed as well. So basically, we already use compression wherever possible.

Yes, we could spend hours arguing over which compression scheme works better and faster, etc. but the differences are typically on the order of 10-20%.

[DGROMS.com]

Originally posted by DGROMS.com


edit
after doing some more testing i've managed to knock at least 600 KB off the update size.
recreating the self extracting executable in winrar.

And I requote though from earlier 600Kb over 250 machines for me = 146Mb less every update and at 10c per Mb = $15 per week more that I can spend on other things

[Paratima]

1. CASP5 is nearly over. This means that the weekly new proteins will end. We will only have to download when we have reached the target saturation for each particular protein. Maybe monthly, 2-3 months, who knows?

2. There is available, on the download page at the DF website right next to each download file, a "daemon" file. The column is titled, suspiciously, "Auto-update Daemon (for users with lots of machines)". Each of these comes with a .txt file explaining how to set up the proxy server for (pardon) users with lots of machines.

Have a on Brian da Fist with the money you'll save!

[tpdooley]

If you've got 250+ machines behind an internet connection (calbe, dsl, t-1, etc) then why aren't you using the client update proxy server setup.. so you only download the client once, and it gets dispersed from the proxy server?

(I still have to play around with it a bit, since I still haven't managed to get the internal web server to actually work; you might ask for other's suggestions on web servers that actually do what the DF client needs..

[DGROMS.com]

All my machines are spread across over 30 locations. The logistics of using the autoupdate proxy are huge, not to mention internal WAN traffic which whilst not being actual Internet Traffic is still VPN traffic across the Net so it actually costs to send and recieve so this would not be of much benefit. So don't be a smart arse before you know the facts and take your beer back until the weekend

[bubbadog]

In the interest of harmony, just give the beer to me!

(Oh, the sacrifices I make for this project! )

[runestar]

rant rant rant

[DGROMS.com]

yadda yadda yadda

[RipItUp]

Maybe a name change to runestar½wit is in order ?

Only kidding, couldn't resist



Regards


Andy

[runestar]

blah blah blah

[Brian the Fist]

Ok, that's enough, if you want to have a flamewar there's better places for it than here. To get back to the original point,

The updates are zipped, so please tell your friend/whoever that this is the case and UPX is not needed. Yes we have a (legal) license for Winzip Self-Extractor which we purchased.

And yes, if you have many computers, even in different locations, definitely take a look at the proxy daemon. Setting up one or several will definitely reduce download bandwidth required though you'll still have the internal traffic of course. I am always open to suggestions on how to improve the daemon too as I am unfamiliar with all the crazy network topologies people are running out there.

P.S. if this silly arguing continues I will delete the whole thread so don't waste your breath.

[Darkness Productions]

Howard:

Is there a way that you can allow us to change where the proxy daemon points for updates? This may allow people like the above a simple and very cost effective way to stagger their data. Aka, he could have 5 VLANs pointed at 1 main site, and possibly save himself some bandwidth.

Also, I've noticed on my linux box, running the proxy daemon, that a *lot* of connections are left open by it. Is there anyway to just store the md5sums of the files in the filelist, and check against it, then go fetch the data? This would save some system resources, and possibly save some bandwidth as well

Glen

[runestar]

Well, I want to apologize to not only Howard, but everyone if I inadverntly set off a flame war although I stand by my original comments... which however, I could have perhaps set a better tone...

(Me thinks I shouldn't post when I'm half asleep. Maybe it WAS ½wit RipIt... )

So as Brian said its already compressed and it ain't likely to change....take it, leave it, or send them a large check... Well, okay he didn't say send them a large check, although I'm sure they wouldn't mind it...

Humbly,

RuneStar½

[Jodie]

Working in the compression industry - I understand that one scheme or another may buy another 5-10%... But as far as an EXE packer goes - since they updates are zipped, I thought I'd just toss in here that the download would get LARGER not smaller. There's the overhead of the unpacker's code in the executable header.

[DGROMS.com]

People please,
It was a suggstion made and asked of Howard by someone else in OCAU which doesn't read this thread, it wasn't made by me, I simply passed it on for him. The biggest mistake I made was not to make it a PM to Howard directly, but as he doesn't seem to reply to PM's I didn't have a choice. The case is closed and is all over it was a suggestion and nothing more it was for Howard to think about and others to not really worry about, Howard has made his decision and it is all done

And to finally set things straight after it has been deleted. I fold because I sat and watched someone die from cancer and I think it helps there cause. I don't give a shit about the size of the download myself as we have more bandwidth than Howard and his team do and we can afford a midnight download which it is in Australia

[runestar]

Originally posted by Jodie
Working in the compression industry - I understand that one scheme or another may buy another 5-10%... But as far as an EXE packer goes - since they updates are zipped, I thought I'd just toss in here that the download would get LARGER not smaller. There's the overhead of the unpacker's code in the executable header.

WinZip used to have a lite version of the self-extractor... I believe it wasn't as full featured, but the difference in the self-executable size was quite noticeable and fairly close to the original archive size. Its not a big of a deal for those with broadband, but every little bit helps for those on modem.

I suppose Howard could possibly find a freeware or public domain ZIP program and encode that in so that there would a straight archive file would be somewhat smaller, but as noted it ultimately would be a question of effort vs the results.

TTFN,

RS½

P.S. Some years when WinZip was getting popular, I compared WinZip vs the Windows version of PKWARE... The PKWARE version granted about a 1 to 2% increase in compression, but it was a god ugly interface. I tested it a few more times later on, and although it improved SOME, WinZip still just had a much nicer interface.

[runestar]

Originally posted by DGROMS.com
People please,
It was a suggstion made and asked of Howard by someone else in OCAU which doesn't read this thread, it wasn't made by me, I simply passed it on for him. The biggest mistake I made was not to make it a PM to Howard directly, but as he doesn't seem to reply to PM's I didn't have a choice. The case is closed and is all over it was a suggestion and nothing more it was for Howard to think about and others to not really worry about, Howard has made his decision and it is all done

And to finally set things straight after it has been deleted. I fold because I sat and watched someone die from cancer and I think it helps there cause. I don't give a shit about the size of the download myself as we have more bandwidth than Howard and his team do and we can afford a midnight download which it is in Australia

DGROMS...

I think we both got a little carried away. =) So tell you what, if you make that a ROOT Beer I'll spring for a mug or a chilled bottle for both of us... we'll shake hands... say we each had valid points... chalk it up to being a little overzealous in our belief... and we'll all be friends. There's bigger fish to fry in the world than ourselves.

RuneStar½

[DGROMS.com]

Originally posted by runestar½



DGROMS...

I think we both got a little carried away. =) So tell you what, if you make that a ROOT Beer I'll spring for a mug or a chilled bottle for both of us... we'll shake hands... say we each had valid points... chalk it up to being a little overzealous in our belief... and we'll all be friends. There's bigger fish to fry in the world than ourselves.

RuneStar½

I agree it got way out of hand, but I still want a real Aussie Pint of James Boags or Cascade, you can have all the root beer you want I hate that stuff I'll shake hands but do one thing never ever question someones reasons for being part of a project like this unless you have a real reason too, questioning my reasons and making bold comments about the maybes is what always starts me off <--This ones the root beer

[RipItUp]

Originally posted by Jodie
Working in the compression industry - I understand that one scheme or another may buy another 5-10%... But as far as an EXE packer goes - since they updates are zipped, I thought I'd just toss in here that the download would get LARGER not smaller. There's the overhead of the unpacker's code in the executable header.

Hi Jodie,

This is the case when comparing Winzip to Winzip EXE maker for instance, but I believe the compression utility mentioned in the original post is a lot more efficient then Winzip, so can afford to convert it to an executable and still end up being smaller. You can test it to see if you wish.

Touche to runestar on my new name

Regards

0.25witAndy

[runestar]

Well assuming you don't go trying to compress things like compressed graphics (GIFs, JPEGS, etc.) or archives of some sort (like CABs) the self-executable should still be smaller than the overall file(s). The real question I think you mean to put out there is how much difference it is between the straight archive (ZIP, ARJ, RAR, etc.) and the executable (i.e. EXE).

I'd like to get my hands on the CAB format though... A long time ago I decompressed a CAB file and then recompressed it in ZIP... the CAB kicked butt if I remember correctly...

Generally speaking unless there is just an overwhelming reason to use another compression format, ZIP is going to get you by in the online community almost all the time. Its pretty much a brand name become common usage... "I'll ZIP these documents for you and send it to you."

RS½

[Brian the Fist]

Originally posted by Darkness Productions
Howard:

Is there a way that you can allow us to change where the proxy daemon points for updates? This may allow people like the above a simple and very cost effective way to stagger their data. Aka, he could have 5 VLANs pointed at 1 main site, and possibly save himself some bandwidth.

Also, I've noticed on my linux box, running the proxy daemon, that a *lot* of connections are left open by it. Is there anyway to just store the md5sums of the files in the filelist, and check against it, then go fetch the data? This would save some system resources, and possibly save some bandwidth as well

Glen

What do you mean 'where it points for updates'? It gets updates from any of the 'official' mirrors. So you want a proxy proxy, so to speak?
And what do you mean by connections left open - how do you determine this? I'm also not sure I understand what you're getting at with md5sums.. how will this make things any better?