George,

I'm afraid not really, well.. a little. Will explain why and what.

It seemed that the problem with us was the fact that udpcast uses synchronous writes to disk by default, and the fact that we pipe data through tar.

For us, after some test scenarios (casting /dev/zero from sender to /dev/null on receiver, casting tarred data from sender to /dev/null on receiver), we found that the slow down/bottleneck was being caused by the synchronous writes. When setting udp-receiver to --nosync we had a (significant) speed increase.

We are running a Linux cluster of 275 machines which we install from a central server using SystemImager and udpcast.

Because SystemImager's image'ing tools dont compress or image the files, we need to cast an entire filesystem (lots of files) over the network. And because udpcast only supports sending/receiving a single file and writing to a single file descriptor, it can only write asynchronously to one file. Because of this we use tar to pipe all files through on both receiver and sender.

This is when the problem arose. When we didn't use tar, we could get high speeds and the (network/harddisk) hardware seemed to become the limiting factor. But only when using the --nosync writes. Because we use tar (which obviously has no --nosync option), now tar became the bottleneck.

Tar writes synchronous to disk by default and there seems to be no way to disable this. As an alternative Alain Knaff suggested using cpio, but we never came to this. This would require some massive changes to both server and client casting setup for SystemImager, and by then we got the speed up to 100 Mbps (still only a 10th of capability but better than before).

This was enough for us, since we are casting a image of approximately 2 Gb. I can image it being a bigger problem when you're casting 20 Gb on a daily basis.

So, to cut a long story short:

Try adding the --nosync argument to your udp-receiver's, and see what happens.

When you don't pipe the data through an external program you should be able to get (very) fast speeds (at least we did). If you _do_ use tar or something to pipe the data through, we found that the maximum lies around 100 Mbps, because of previously explained reasons.

Another approach could be to first send the data as one big tar to your clients, and extract the tar after transmission. Or to try pipe'ing through cpio (haven't tried that one myself) in stead of tar.

Hope this helps you a bit. I know it can be a frustrating problem ;). Let me know how things work out for you.

Kind regards, Ramon.

George Coulouris wrote:

I've managed to get reliable transfers to all 85 machines at this point (bad nic in the sender, I think).

I'm still bottlenecking at 45Mbit or so, though, despite the fact that all machines are on switched copper gigabit.

If fec is turned on (either 8x8/128 or 16x4/128), receivers disconnect throughout the transfer until only ~10 remain at the end.

If fec is turned off, I get high retransmission rates (15-25%), but the transfer at least succeeds to all hosts. Again, I'm sending about 20G of data, which should take about an hour at 45Mbit. In practice, it takes about an hour and a half, and sometimes close to two hours.

For simple point-to-point tests with udpcast (between 1 and 5 receivers), I can achieve higher bandwidths reliably (sending 128 megabytes at 80mbit), both with and without fec.

It sounds like my issue is similar to Ramon Bastiaan's: http://lll.lgl.lu/pipermail/udpcast/2004-September/000298.html

Ramon, have you had any luck?