Udpcast April 2003

udpcast@udpcast.linux.lu

7 participants
6 discussions

Pipe Line Full message...
by Saikrishna Dasari 30 Apr '03

30 Apr '03

Can anybody explain me what the "Pipe Line Full message" on receiver side and when does it occur?? Thanks, Sai __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com

2 1

great results using compression!
by daniel petersen 25 Apr '03

25 Apr '03

Found a way to effectively implement udpcast with compression, and how to get around the "pipeline full" problem! Here's what we did, and the results! On most of our hard drives, they're around 1/3 to 2/3 full, so we thought it was a waste of time to send all those empty blocks across the wire when multicasting, so I seriously looked into compression, opting for a fast compressor rather than an efficient one, basically so that when the sender hits the empty blocks full of zeroes, they'll get compressed and send very little data across the wire. I tried gzip, but it wasn't fast enough, and ended up using lzop for compression. In order for compression to be worthwhile, the sender's cpu power and the compression speed in combination have to be fast enough so that it can compress nearly as fast as the hard drive can read it. The receiver could probably have a somewhat slower cpu, as decompression takes less cpu crunching. For testing sake, we plugged two P4 1.8GHZ machines w/ fast 80GB drives (7.2k rpm, ultra ata-100 seagates) into a 100Mbps switch. We piped the transmission through lzop, using the fastest compression setting possible. the problem we ran into was that when the sender would start hitting the unused blocks, compressing them and sending them on the wire, the receiver couldn't write them fast enough to the hard drive! This resulted in the "pipeline full" message, and then the connection ended up timing out. We ended up changing the sender timeout in the source code from the default (.2 seconds) to 60 seconds. Probably overkill, but thanks to a well written program (thanks Alain!) the sender just keeps trying until the receiver asks for more after it's caught up and the receiver has more room in its buffer. We wanted to make sure no matter what that none of the clients were dropped, thus the excessive timeout of 60 seconds. The "pipeline full" msg still pops up many times, but the clients aren't dropped, and the task finishes successfully! So, here's the commandline we used on sender and receivers piping through compression: udp-sender -p "lzop -1" -f /dev/hda udp-receiver -p "lzop -d" -f /dev/hda ---- and here are the results of copying an 80GB HDD to an 80GB HDD(approximately 28GB of actual data on the drive) between two P4 1.8GHZ machines across a 100Mbps switch(compressed versus uncompressed): 93 minutes w/ compression (approx. 50GB sent across the wire) - avg. 860 MBytes/min 114 minutes w/ out compr. (80GB sent across wire, of course;^) - avg. 700 MBytes/min ---- same hardware setup, but just copying a 4GB partition w/ approx 1.2GB of data on it: 3 min, 30 seconds w/ compression - avg. 1142 MBytes/min! 6 min, 7 seconds w/ out compression - avg. 660 MBytes/min When piping through compression the Mbps measurement that shows onscreen on the receivers end is misleading because it's only showing compressed data that it's received, not what it's writing to the hard drive. This is especially noticable when the sender hits unused blocks as the Mbps drops drastically, but large sections of the hard drive are being copied per second! So it isn't an accurate measurement of performance when using compression, that's why I went with overall time measurements and then pulled some per minute averages from the numbers. Some observations we made from the above tests. Obviously the more zeroes on the drive, the faster it goes. We suspect that with 50GB sent across the wire in the case of the compressed test for the 80GB drive, there was a fair amount of data in unused blocks from previously deleted files on the hard drive, otherwise it would have been much smaller(and much faster!). With the 1.8GHZ P4s the compression was slightly slower than the hard drive was reading used data blocks(probably around 80%), so speaking only of speed, compression wouldn't be worthwhile if the hard drive was completely full with the P4s used. But in most situations where hard drives have some empty space, it is a great improvement in overall speed! A nice side effect is less traffic on the LAN. Also, I would assume that with a P4 or equivalent in excess speeds of 2.2GHZ the compression should be able to completely keep up with the read data transfer rate of the hard drive. During these tests, it was noticed that during the transfer of unused blocks, on the receiving end the hard drive frequently couldn't write fast enough and so the buffer would fill up, making the sender wait before sending more data. In summary: using computers with sufficient processing power in combination with fast compression on a 100Mbps network performs very well when multicasting a typical hard drive! Our image master can only copy drives at approx 500MB/min, and that's with the added hassle of shuffling the drives in and out of the computers with trays, etc. The udpcast method is faster, and the hard drives can stay in the computers! A very useful program, our thanks to the programmer! Regards, Daniel Petersen PS. udpcast is working beautifully when transferring directly from hard drive to hard drive, but we're having some trouble when trying to dump to a file that will be larger than 2GB. We get the message "File size limit exceeded" when it hits the 2GB mark on the receivers, despite the fact that we're putting the file on a redhat box w/ an ext3 file system, which handles large files just fine. We think it might be udpcast bailing out as a fail safe feature. Alain, or those who might be "in the know", have you ever seen this and/or know how to solve the problem? ____________________________________________________________ Get 25MB of email storage with Lycos Mail Plus! Sign up today -- http://www.mail.lycos.com/brandPage.shtml?pageId=plus

1 0

Udpcast sender problem ???
by Saikrishna Dasari 25 Apr '03

25 Apr '03

Hi friends, I have been using UDPCast successfully for the fast few months and it was successful. in most cases. But recently when I was trying to clone a 5.5 GB Partion from one machine to other machine ( of same architecture..both are dell machines, both have 256 MB RAM, Identical HDD geometry, Identical NICs). on the sender side after getting a few retransmission messages it kept on saying that nrBlocks=36 lastGoodBlock=0nrBlocks=130 Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=3969 bytes=1 232 425 333 re-xmits=001037 (0.0%) slice=0036 1 232 425 333 and reciever doesn't give any messages. what does this message mean. Does it assure us a safe transfer ..if not how to rectify this problem.Any help is highly appreciated. Thanks, Sai __________________________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo http://search.yahoo.com

3 2

problem with sis900 module
by Chris Locke 11 Apr '03

11 Apr '03

hello all, I have just found udpcast today and it fit's in perfect with a project I am working on for my boss. I have 1 problem though I need to load the sis900 module. not a big deal, I went ahead and loaded the one for the 2.2.18 kernel per the documentation, it didn't work but I did notice that the software is using the 2.4.10 kernel. So i went ahead and tried both the 2.4.18 and 2.4.20 modules from our kernels. none work. anyone else have problems or do i just need to actually download the 2.4.10 kernel and use it? -- Chris Locke <clocke(a)stratitec.com> Stratitec

1 0

get "pipeline full" when piping through gzip...
by Daniel Petersen 11 Apr '03

11 Apr '03

Hello Alain, Thanks for sharing this useful program with the Internet community! I've come across an issue that I hope you might have some insight to;^) I've been trying to pipe udpcast through gzip but get the message "pipeline full" a ways into the operation before it finishes. The specific setup is two computers with the exact same setup: P4 1.8Ghz w/ 256ram, and 80GB seagate hdds. I'm sending a 4GB partition from one to the other across a 100 hub. The exact command and arguments I'm using for the respective computer are: udp-sender -p "gzip -1" -f /dev/hda1 udp-receiver -p "gzip -d" -f /dev/hda1 The partition has about 1.4GB of data and the rest is empty. (I made sure the drive was zeroed out before installing the os) The interesting thing is that it seems to do fine until it hits the empty part of the drive at which point the compression ratio of course gets extremely high, and then after it's compressed up to the 3GB mark, the receiving computer crashes with the message "pipeline full". Also, you mention on the udpcast commandline page that the pipe argument can also be used to strip out unused blocks on the device. How would I go about doing this? This would be a viable alternative if the gzip issue can't be resolved as my bottomline purpose is to not have to send all those empty blocks across the wire if I don't have to! Thanks in advance for any help you or anyone else can offer on the issue, Daniel P. ************************************* This email may contain privileged or confidential material intended for the named recipient only. If you are not the named recipient, delete this message and all attachments. Any review, copying, printing, disclosure or other use is prohibited. We reserve the right to monitor email sent through our network. *************************************

1 0

Re: Udpcast Alternative Boot Disks
by Andrew Cooks 04 Apr '03

04 Apr '03

Hi Mike & everyone on udpcast I fiddled with my new bootdisks again today. I can now get about 93.6Mb/s on a switched 100Mb/s network. As you said, the main limiting factor lies in how quickly the receiver can write the data to disk. I tried using compression, but that made it slower, so I decided to use hdparm to set the disk: "hdparm -c1 -d1 /dev/hda". This made a _huge_ difference! My next step is to recompile everything on my boot disk, using ulibc. Then there should be enough space to include a compression program on the boot disks, without using higher densities. I expect this should make quite a difference as well, seeing as the cpu is now free to do this kind of thing. I'm just not sure whether it's really possible to compress data fast enough. Gzip is not fast enough on the hardware I have. It would have to be faster than 95Mb/s to be of benefit in this situation. I'll make my disk images and sources available on http://tlug.up.ac.za/downloads.html asap. Cheers Andrew On Tue, 11 Mar 2003, Mike McCall wrote: > Andrew, > > I would love a copy of both the sender and the receiver disks! As you might > have noticed from some of my posts to the list, the off-the-shelf disks > don't work for a couple of our PCs due to a funky bios. Just let me know > how to get the files! Thanks! > > As for udpcast printing stuff to the screen, when I have used both > statically compiled binaries and the boot disks, there is a LARGE amount of > printing on the screen for the sender, and an activity status (transfer > rate) on the receiver. You might want to make sure the binaries compiled > successfully. If you want to try the binaries I have, I can send them to > you. > > By the way, does your boot disk have hdparm on it? I have noticed that the > receivers can't keep up on a high speed connection without it. > > Thanks again! > > Mike McCall > Computer Technician > Rio Rancho Public Schools > > ----------------------- Andrew Cooks TechTeam TechTeam -- "We make it work." Computer Science dept. University of Pretoria ----------------------- "I don't make jokes in base 13. Anyone who does should get help." -Douglas Adams

3 4

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Udpcast April 2003