Can anybody explain me what the "Pipe Line Full
message" on receiver side and when does it occur??
Thanks,
Sai
__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
Found a way to effectively implement udpcast with compression, and how to get around the "pipeline full" problem! Here's what we did, and the results!
On most of our hard drives, they're around 1/3 to 2/3 full, so we thought it was a waste of time to send all those empty blocks across the wire when multicasting, so I seriously looked into compression, opting for a fast compressor rather than an efficient one, basically so that when the sender hits the empty blocks full of zeroes, they'll get compressed and send very little data across the wire. I tried gzip, but it wasn't fast enough, and ended up using lzop for compression. In order for compression to be worthwhile, the sender's cpu power and the compression speed in combination have to be fast enough so that it can compress nearly as fast as the hard drive can read it. The receiver could probably have a somewhat slower cpu, as decompression takes less cpu crunching.
For testing sake, we plugged two P4 1.8GHZ machines w/ fast 80GB drives (7.2k rpm, ultra ata-100 seagates) into a 100Mbps switch. We piped the transmission through lzop, using the fastest compression setting possible. the problem we ran into was that when the sender would start hitting the unused blocks, compressing them and sending them on the wire, the receiver couldn't write them fast enough to the hard drive! This resulted in the "pipeline full" message, and then the connection ended up timing out. We ended up changing the sender timeout in the source code from the default (.2 seconds) to 60 seconds. Probably overkill, but thanks to a well written program (thanks Alain!) the sender just keeps trying until the receiver asks for more after it's caught up and the receiver has more room in its buffer. We wanted to make sure no matter what that none of the clients were dropped, thus the excessive timeout of 60 seconds. The "pipeline full" msg still pops up many times, but the clients aren't dropped, and the task finishes successfully!
So, here's the commandline we used on sender and receivers piping through compression:
udp-sender -p "lzop -1" -f /dev/hda
udp-receiver -p "lzop -d" -f /dev/hda
----
and here are the results of copying an 80GB HDD to an 80GB HDD(approximately 28GB of actual data on the drive) between two P4 1.8GHZ machines across a 100Mbps switch(compressed versus uncompressed):
93 minutes w/ compression (approx. 50GB sent across the wire) - avg. 860 MBytes/min
114 minutes w/ out compr. (80GB sent across wire, of course;^) - avg. 700 MBytes/min
----
same hardware setup, but just copying a 4GB partition w/ approx 1.2GB of data on it:
3 min, 30 seconds w/ compression - avg. 1142 MBytes/min!
6 min, 7 seconds w/ out compression - avg. 660 MBytes/min
When piping through compression the Mbps measurement that shows onscreen on the receivers end is misleading because it's only showing compressed data that it's received, not what it's writing to the hard drive. This is especially noticable when the sender hits unused blocks as the Mbps drops drastically, but large sections of the hard drive are being copied per second! So it isn't an accurate measurement of performance when using compression, that's why I went with overall time measurements and then pulled some per minute averages from the numbers.
Some observations we made from the above tests. Obviously the more zeroes on the drive, the faster it goes. We suspect that with 50GB sent across the wire in the case of the compressed test for the 80GB drive, there was a fair amount of data in unused blocks from previously deleted files on the hard drive, otherwise it would have been much smaller(and much faster!). With the 1.8GHZ P4s the compression was slightly slower than the hard drive was reading used data blocks(probably around 80%), so speaking only of speed, compression wouldn't be worthwhile if the hard drive was completely full with the P4s used. But in most situations where hard drives have some empty space, it is a great improvement in overall speed! A nice side effect is less traffic on the LAN. Also, I would assume that with a P4 or equivalent in excess speeds of 2.2GHZ the compression should be able to completely keep up with the read data transfer rate of the hard drive. During these tests, it was noticed that during the transfer of unused blocks, on the receiving end the hard drive frequently couldn't write fast enough and so the buffer would fill up, making the sender wait before sending more data.
In summary: using computers with sufficient processing power in combination with fast compression on a 100Mbps network performs very well when multicasting a typical hard drive! Our image master can only copy drives at approx 500MB/min, and that's with the added hassle of shuffling the drives in and out of the computers with trays, etc. The udpcast method is faster, and the hard drives can stay in the computers! A very useful program, our thanks to the programmer!
Regards,
Daniel Petersen
PS. udpcast is working beautifully when transferring directly from hard drive to hard drive, but we're having some trouble when trying to dump to a file that will be larger than 2GB. We get the message "File size limit exceeded" when it hits the 2GB mark on the receivers, despite the fact that we're putting the file on a redhat box w/ an ext3 file system, which handles large files just fine. We think it might be udpcast bailing out as a fail safe feature. Alain, or those who might be "in the know", have you ever seen this and/or know how to solve the problem?
____________________________________________________________
Get 25MB of email storage with Lycos Mail Plus!
Sign up today -- http://www.mail.lycos.com/brandPage.shtml?pageId=plus
Hi friends,
I have been using UDPCast successfully for the fast
few months and it was successful. in most cases. But
recently when I was trying to clone a 5.5 GB Partion
from one machine to other machine ( of same
architecture..both are dell machines, both have 256 MB
RAM, Identical HDD geometry, Identical NICs).
on the sender side after getting a few retransmission
messages it kept on saying that
nrBlocks=36 lastGoodBlock=0nrBlocks=130
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0
nrPart=1 avg=3969
bytes=1 232 425 333 re-xmits=001037 (0.0%) slice=0036
1 232 425 333
and reciever doesn't give any messages.
what does this message mean. Does it assure us a safe
transfer ..if not how to rectify this problem.Any help
is highly appreciated.
Thanks,
Sai
__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com
hello all,
I have just found udpcast today and it fit's in perfect with a project I
am working on for my boss. I have 1 problem though I need to load the
sis900 module. not a big deal, I went ahead and loaded the one for the
2.2.18 kernel per the documentation, it didn't work but I did notice
that the software is using the 2.4.10 kernel. So i went ahead and tried
both the 2.4.18 and 2.4.20 modules from our kernels. none work. anyone
else have problems or do i just need to actually download the 2.4.10
kernel and use it?
--
Chris Locke <clocke(a)stratitec.com>
Stratitec
Hello Alain,
Thanks for sharing this useful program with the Internet community! I've come across an issue that I hope you might have some insight to;^)
I've been trying to pipe udpcast through gzip but get the message "pipeline full" a ways into the operation before it finishes. The specific setup is two computers with the exact same setup: P4 1.8Ghz w/ 256ram, and 80GB seagate hdds. I'm sending a 4GB partition from one to the other across a 100 hub. The exact command and arguments I'm using for the respective computer are:
udp-sender -p "gzip -1" -f /dev/hda1
udp-receiver -p "gzip -d" -f /dev/hda1
The partition has about 1.4GB of data and the rest is empty. (I made sure the drive was zeroed out before installing the os) The interesting thing is that it seems to do fine until it hits the empty part of the drive at which point the compression ratio of course gets extremely high, and then after it's compressed up to the 3GB mark, the receiving computer crashes with the message "pipeline full".
Also, you mention on the udpcast commandline page that the pipe argument can also be used to strip out unused blocks on the device. How would I go about doing this? This would be a viable alternative if the gzip issue can't be resolved as my bottomline purpose is to not have to send all those empty blocks across the wire if I don't have to!
Thanks in advance for any help you or anyone else can offer on the issue,
Daniel P.
*************************************
This email may contain privileged or confidential material intended for the named recipient only.
If you are not the named recipient, delete this message and all attachments.
Any review, copying, printing, disclosure or other use is prohibited.
We reserve the right to monitor email sent through our network.
*************************************
Hi Mike & everyone on udpcast
I fiddled with my new bootdisks again today. I can now get about 93.6Mb/s
on a switched 100Mb/s network.
As you said, the main limiting factor lies in how quickly the receiver can
write the data to disk. I tried using compression, but that made it
slower, so I decided to use hdparm to set the disk:
"hdparm -c1 -d1 /dev/hda". This made a _huge_ difference!
My next step is to recompile everything on my boot disk, using ulibc. Then
there should be enough space to include a compression program on the boot
disks, without using higher densities. I expect this should make quite a
difference as well, seeing as the cpu is now free to do this kind of
thing. I'm just not sure whether it's really possible to compress data
fast enough. Gzip is not fast enough on the hardware I have. It would have
to be faster than 95Mb/s to be of benefit in this situation.
I'll make my disk images and sources available on
http://tlug.up.ac.za/downloads.html asap.
Cheers
Andrew
On Tue, 11 Mar 2003, Mike McCall wrote:
> Andrew,
>
> I would love a copy of both the sender and the receiver disks! As you might
> have noticed from some of my posts to the list, the off-the-shelf disks
> don't work for a couple of our PCs due to a funky bios. Just let me know
> how to get the files! Thanks!
>
> As for udpcast printing stuff to the screen, when I have used both
> statically compiled binaries and the boot disks, there is a LARGE amount of
> printing on the screen for the sender, and an activity status (transfer
> rate) on the receiver. You might want to make sure the binaries compiled
> successfully. If you want to try the binaries I have, I can send them to
> you.
>
> By the way, does your boot disk have hdparm on it? I have noticed that the
> receivers can't keep up on a high speed connection without it.
>
> Thanks again!
>
> Mike McCall
> Computer Technician
> Rio Rancho Public Schools
>
>
-----------------------
Andrew Cooks
TechTeam TechTeam -- "We make it work."
Computer Science dept.
University of Pretoria
-----------------------
"I don't make jokes in base 13. Anyone who does should get help."
-Douglas Adams
