Hi Mike & everyone on udpcast
I fiddled with my new bootdisks again today. I can now get about 93.6Mb/s on a switched 100Mb/s network.
As you said, the main limiting factor lies in how quickly the receiver can write the data to disk. I tried using compression, but that made it slower, so I decided to use hdparm to set the disk: "hdparm -c1 -d1 /dev/hda". This made a _huge_ difference!
My next step is to recompile everything on my boot disk, using ulibc. Then there should be enough space to include a compression program on the boot disks, without using higher densities. I expect this should make quite a difference as well, seeing as the cpu is now free to do this kind of thing. I'm just not sure whether it's really possible to compress data fast enough. Gzip is not fast enough on the hardware I have. It would have to be faster than 95Mb/s to be of benefit in this situation.
I'll make my disk images and sources available on http://tlug.up.ac.za/downloads.html asap.
Cheers
Andrew
On Tue, 11 Mar 2003, Mike McCall wrote:
Andrew,
I would love a copy of both the sender and the receiver disks! As you might have noticed from some of my posts to the list, the off-the-shelf disks don't work for a couple of our PCs due to a funky bios. Just let me know how to get the files! Thanks!
As for udpcast printing stuff to the screen, when I have used both statically compiled binaries and the boot disks, there is a LARGE amount of printing on the screen for the sender, and an activity status (transfer rate) on the receiver. You might want to make sure the binaries compiled successfully. If you want to try the binaries I have, I can send them to you.
By the way, does your boot disk have hdparm on it? I have noticed that the receivers can't keep up on a high speed connection without it.
Thanks again!
Mike McCall Computer Technician Rio Rancho Public Schools
----------------------- Andrew Cooks TechTeam TechTeam -- "We make it work." Computer Science dept. University of Pretoria -----------------------
"I don't make jokes in base 13. Anyone who does should get help." -Douglas Adams
On Wed, 12 Mar 2003, Andrew Cooks wrote:
My next step is to recompile everything on my boot disk, using ulibc. Then there should be enough space to include a compression program on the boot disks, without using higher densities. I expect this should make quite a difference as well, seeing as the cpu is now free to do this kind of thing. I'm just not sure whether it's really possible to compress data fast enough. Gzip is not fast enough on the hardware I have. It would have to be faster than 95Mb/s to be of benefit in this situation.
I doubt that compression will give you any improvement. With compression, the CPU and the memory subsystem become the bottlenecks quite quickly. In my own experience, with compression there are simply to many memory copy operations [1].
- Felix
[1] http://www.cs.inf.ethz.ch/~rauch/index.html#CCPE2002
rauch@inf.ethz.ch writes:
On Wed, 12 Mar 2003, Andrew Cooks wrote:
My next step is to recompile everything on my boot disk, using ulibc. Then there should be enough space to include a compression program on the boot disks, without using higher densities. I expect this should make quite a difference as well, seeing as the cpu is now free to do this kind of thing. I'm just not sure whether it's really possible to compress data fast enough. Gzip is not fast enough on the hardware I have. It would have to be faster than 95Mb/s to be of benefit in this situation.
I doubt that compression will give you any improvement. With compression, the CPU and the memory subsystem become the bottlenecks quite quickly. In my own experience, with compression there are simply to many memory copy operations [1].
- Felix
[1] http://www.cs.inf.ethz.ch/~rauch/index.html#CCPE2002
Felix Rauch | Email: rauch@inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307
Udpcast mailing list Udpcast@udpcast.linux.lu http://udpcast.linux.lu/mailman/listinfo/udpcast
when your are compressing the image, are you makeing a smaller imafe file. i use udp cast to store my images and a server. if i used no compression, what would the file size be? i have yet to try it. i always just compressed.
(John Allison )
I'm not too sure what you're saying, but if you're asking about saving an image on a server in compressed form instead of uncompressed, I think it works as follows:
If you don't use compression, the size of the file will be the same as the size of the partition or disk it is an image of. This could be what you want if you have lots of space on a slow machine, or a super fast network.
Compression might be cool if you have a slow network, or super fast machines on a 10/100Mb network, or just a lack of storage space.
IMHO, storing images on a server is not such a good thing. People should patch the software installations before casting a new image and to do that the image needs to be running on a machine. When this machine (image) is ready for deployment, then you could just as well cast that drive sector-by-sector. I do admit that it is easier to start from the previous image than to reinstall a machine, but using an older image promotes laziness of installing patches and bug-fixes.
Andrew
when your are compressing the image, are you makeing a smaller imafe file. i use udp cast to store my images and a server. if i used no compression, what would the file size be? i have yet to try it. i always just compressed.
(John Allison )
----------------------- Andrew Cooks TechTeam TechTeam -- "We make it work." Computer Science dept. University of Pretoria -----------------------
"I don't make jokes in base 13. Anyone who does should get help." -Douglas Adams
I'm trying to get compression working on the custom disks that I'm making. The problem that I'm having is that the Receiver complains that "Pipeline full" and then starts dropping packets like crazy and disconnects.
Any ideas why?
I suspect it has to do with the buffer in the receiver being too small or not being emptied to disk quick enough.
I'm using a program that was written by a friend of mine, based on the lzrw compression algorithm. I've found it to be faster than lzo. It works exactly the same way as gzip, just much faster, as far as udpcast is concerned.
Does anyone on this list actively code on udpcast?
Andrew
On Wed, 12 Mar 2003, Felix Rauch wrote:
On Wed, 12 Mar 2003, Andrew Cooks wrote:
My next step is to recompile everything on my boot disk, using ulibc. Then there should be enough space to include a compression program on the boot disks, without using higher densities. I expect this should make quite a difference as well, seeing as the cpu is now free to do this kind of thing. I'm just not sure whether it's really possible to compress data fast enough. Gzip is not fast enough on the hardware I have. It would have to be faster than 95Mb/s to be of benefit in this situation.
I doubt that compression will give you any improvement. With compression, the CPU and the memory subsystem become the bottlenecks quite quickly. In my own experience, with compression there are simply to many memory copy operations [1].
- Felix
[1] http://www.cs.inf.ethz.ch/~rauch/index.html#CCPE2002
Felix Rauch | Email: rauch@inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307
Udpcast mailing list Udpcast@udpcast.linux.lu http://udpcast.linux.lu/mailman/listinfo/udpcast
----------------------- Andrew Cooks TechTeam TechTeam -- "We make it work." Computer Science dept. University of Pretoria -----------------------
"I don't make jokes in base 13. Anyone who does should get help." -Douglas Adams
-
Andrew Cooks -
Felix Rauch -
John Allison