24 Apr
2008
24 Apr
'08
1:56 p.m.
I'm looking at using udpcast to broadcast large disk images (10+ GB)
to a very large network of machines (1,000-10,000 receivers) over a
mostly switched, partially segmented gig-ethernet network.
Needless to say, the network of machines is all production-critical
and I cannot get access to perform real testing. However testing it
on my home network I can see some potential problems:
- If _any_ receiver is misbehaving or unreachable then this stops
all transmissions. Is there a way to get udpcast to drop
troublesome receivers in this situation (other than unicast)?
- Has anyone used the --ttl option to multicast over routers?
Does it work (the manpage is unclear)? Does it need special
routers?
- Any other scaling tips? Should I try to go for the full set of
machines at once or break up the broadcast into groups of machines?
If anyone has used udpcast on such large networks, can you share any
experiences.
Rich.
--
Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
24 Apr
24 Apr
3:06 p.m.
Hi Rich,
I tried to solve a problem much smaller than yours but still had
incredible difficulty. I was moving 10GB datasets out to 64 receivers
over a flat switched network using multicast. Unfortunately, for reasons
I never tracked down, files of this size would always get corrupted
along the way even though all the receivers had received all packets
(i.e. the md5sum would be different across all the different machines).
Eventually I ended up using small bittorrent clients instead of udpcast
since it checks the hash of each block. This also makes the process take
about twice as long, but better to get correct data slow than corrupt
data fast!
Hope you have better luck,
-Michael
Richard W.M. Jones wrote:
I'm looking at using udpcast to broadcast large
disk images (10+ GB)
to a very large network of machines (1,000-10,000 receivers) over a
mostly switched, partially segmented gig-ethernet network.
Needless to say, the network of machines is all production-critical
and I cannot get access to perform real testing. However testing it
on my home network I can see some potential problems:
- If _any_ receiver is misbehaving or unreachable then this stops
all transmissions. Is there a way to get udpcast to drop
troublesome receivers in this situation (other than unicast)?
- Has anyone used the --ttl option to multicast over routers?
Does it work (the manpage is unclear)? Does it need special
routers?
- Any other scaling tips? Should I try to go for the full set of
machines at once or break up the broadcast into groups of machines?
If anyone has used udpcast on such large networks, can you share any
experiences.
Rich.
6:51 p.m.
On Thu, Apr 24, 2008 at 11:06:03AM -0400, Michael Holroyd wrote:
I tried to solve a problem much smaller than yours
but still had
incredible difficulty. I was moving 10GB datasets out to 64 receivers over
a flat switched network using multicast. Unfortunately, for reasons I never
tracked down, files of this size would always get corrupted along the way
even though all the receivers had received all packets (i.e. the md5sum
would be different across all the different machines).
I haven't seen this problem (my tests are too small-scale probably)
but I notice that the protocol doesn't do any sort of error detection
for the dataBlocks. So we're relying on UDP's 16 bit checksum and
maybe ethernet's CRC32. Both types of checksum are known to be very
weak, and ethernet checksumming is even sometimes turned off.
Shouldn't be too hard to add a more robust checksum to the packets.
Is anyone interested in a patch? I might have a go at one later.
Rich.
PS. My cheap-ish consumer switch slows down from gigabit-ethernet to
10 Mbps as soon as I ask it to do multicast or broadcast. Is this
normal?
--
Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
9:15 p.m.
Hi Rich,
I'd be very interested in a patch. I might have worked on one myself
(I'd already edited the source to turn off the absurd stdout spam), but
unfortunately my sender is a windows box and I wasn't in the mood to
battle compiling udpcast for windows. The behavior I saw was that 50-55
of my recievers would have the same md5sum, but it was still the *wrong*
md5sum. The outliers could very plausibly be due to checksums being
unreliable, and perhaps there was some bit-flipping that occurred before
the first router sent everything out on multicast.
Let me know how it goes if you decide to try it out,
-Michael
Richard W.M. Jones wrote:
On Thu, Apr 24, 2008 at 11:06:03AM -0400, Michael
Holroyd wrote:
I tried to solve a problem much smaller than
yours but still had
incredible difficulty. I was moving 10GB datasets out to 64 receivers over
a flat switched network using multicast. Unfortunately, for reasons I never
tracked down, files of this size would always get corrupted along the way
even though all the receivers had received all packets (i.e. the md5sum
would be different across all the different machines).
I haven't seen this problem (my tests are too small-scale probably)
but I notice that the protocol doesn't do any sort of error detection
for the dataBlocks. So we're relying on UDP's 16 bit checksum and
maybe ethernet's CRC32. Both types of checksum are known to be very
weak, and ethernet checksumming is even sometimes turned off.
Shouldn't be too hard to add a more robust checksum to the packets.
Is anyone interested in a patch? I might have a go at one later.
Rich.
PS. My cheap-ish consumer switch slows down from gigabit-ethernet to
10 Mbps as soon as I ask it to do multicast or broadcast. Is this
normal?
9:24 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Just a quick addition to my earlier message. I just did a retest of doing the
imaging after opening both port 9000 and 9001 on both senders and
receivers. Sent the image to 19 receivers, and this time, it went thru with no
errors, and the files were the correct size? So, at least in this case, it would
appear that port 9001 is playing a part in this.
On 24 Apr 2008 at 17:14, Michael Holroyd wrote:
Date sent: Thu, 24 Apr 2008 17:14:37 -0400
From: Michael Holroyd <meekohi(a)cs.virginia.edu>
To: "Richard W.M. Jones" <rjones(a)redhat.com>
Copies to: udpcast(a)udpcast.linux.lu
Subject: Re: [Udpcast] Scaling udpcast
Hi Rich,
I'd be very interested in a patch. I might have worked on one myself
(I'd already edited the source to turn off the absurd stdout spam), but
unfortunately my sender is a windows box and I wasn't in the mood to
battle compiling udpcast for windows. The behavior I saw was that 50-55
of my recievers would have the same md5sum, but it was still the *wrong*
md5sum. The outliers could very plausibly be due to checksums being
unreliable, and perhaps there was some bit-flipping that occurred before
the first router sent everything out on multicast.
Let me know how it goes if you decide to try it out,
-Michael
Richard W.M. Jones wrote:
+----------------------------------------------------------+
Michael D. Setzer II - Computer Science Instructor
Guam Community College Computer Center
mailto:mikes@kuentos.guam.net
mailto:msetzerii@gmail.com
http://www.guam.net/home/mikes
Guam - Where America's Day Begins
+----------------------------------------------------------+
http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned: 19,471
Processing time: 32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)
BOINC@HOME CREDITS
SETI 5,269,727.070797 | EINSTEIN 1,573,038.609732 | ROSETTA
480,077.992597
On Thu, Apr 24, 2008 at 11:06:03AM -0400, Michael
Holroyd wrote:
_______________________________________________
Udpcast mailing list
Udpcast(a)udpcast.linux.lu
https://lll.lgl.lu/mailman/listinfo/udpcast I tried to solve a problem much smaller than
yours but still had
incredible difficulty. I was moving 10GB datasets out to 64 receivers over
a flat switched network using multicast. Unfortunately, for reasons I never
tracked down, files of this size would always get corrupted along the way
even though all the receivers had received all packets (i.e. the md5sum
would be different across all the different machines).
I haven't seen this problem (my tests are too small-scale probably)
but I notice that the protocol doesn't do any sort of error detection
for the dataBlocks. So we're relying on UDP's 16 bit checksum and
maybe ethernet's CRC32. Both types of checksum are known to be very
weak, and ethernet checksumming is even sometimes turned off.
Shouldn't be too hard to add a more robust checksum to the packets.
Is anyone interested in a patch? I might have a go at one later.
Rich.
PS. My cheap-ish consumer switch slows down from gigabit-ethernet to
10 Mbps as soon as I ask it to do multicast or broadcast. Is this
normal?
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.8 -- QDPGP 2.61c
Comment: http://community.wow.net/grt/qdpgp.html
iQA/AwUBSBBt8SzGQcr/2AKZEQIRUQCfYMFU8tBUgEu8EUr+/6YLQVUtlLkAnjVv
J6wbq6W2BfJk8zQOobciCaEX
=u158
-----END PGP SIGNATURE-----
8:41 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 24 Apr 2008 at 11:06, Michael Holroyd wrote:
Date sent: Thu, 24 Apr 2008 11:06:03 -0400
From: Michael Holroyd <meekohi(a)cs.virginia.edu>
To: "Richard W.M. Jones" <rjones(a)redhat.com>
Copies to: udpcast(a)udpcast.linux.lu
Subject: Re: [Udpcast] Scaling udpcast
Hi Rich,
I tried to solve a problem much smaller than yours but still had
incredible difficulty. I was moving 10GB datasets out to 64 receivers
over a flat switched network using multicast. Unfortunately, for reasons
I never tracked down, files of this size would always get corrupted
along the way even though all the receivers had received all packets
(i.e. the md5sum would be different across all the different machines).
Eventually I ended up using small bittorrent clients instead of udpcast
since it checks the hash of each block. This also makes the process take
about twice as long, but better to get correct data slow than corrupt
data fast!
I had a similar problem recently on a much smaller scale. I was testing
udpcast with a classroom to sent a just under 7GB ntfsclone image file to
varios machines. I had one sender and 4 receivers and it worked fine twice.
Then I tried with 8 receivers, and it failed. No error messages, and did it
twice, with same error. Not sure what is the cause?
I am planning on doing some more testing, and it might be a problem with
SELINUX and ports. To get it to work, I had to open port 9000 and 9001 on
the sender with udp, and port 9000 on the receiver with udp on the receiver.
Perhaps receiver also needs 9001?
The files on all the receivers seems to be that same size. Before this, I was
using a script to down the file via ftp using ncftp. On the successful runs, it
would udpcast the files to the linux partition, and then run the scipt to restore
the new XP partition. The script woulld then show the file as being the same,
and skip the download, and go straight to the retore. On the error batch, it
started downloading the file via ftp, since they were not the same.
I've used udpcast to image 19 machines from one sender with no errors
usign udpcast images, and have noticed any errors, and have systems run
the disk test on boot with no errors.
So, don't know if it is a size problem, or ports, or kernel option, or option or
something else? I'll try some more things, and try to see what is different
between a good file and a bad one, and see if it was always the same.
Hope you have better luck,
-Michael
Richard W.M. Jones wrote:
+----------------------------------------------------------+
Michael D. Setzer II - Computer Science Instructor
Guam Community College Computer Center
mailto:mikes@kuentos.guam.net
mailto:msetzerii@gmail.com
http://www.guam.net/home/mikes
Guam - Where America's Day Begins
+----------------------------------------------------------+
http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned: 19,471
Processing time: 32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)
BOINC@HOME CREDITS
SETI 5,269,727.070797 | EINSTEIN 1,573,038.609732 | ROSETTA
480,077.992597
I'm looking at using udpcast to broadcast
large disk images (10+ GB)
to a very large network of machines (1,000-10,000 receivers) over a
mostly switched, partially segmented gig-ethernet network.
Needless to say, the network of machines is all production-critical
and I cannot get access to perform real testing. However testing it
on my home network I can see some potential problems:
- If _any_ receiver is misbehaving or unreachable then this stops
all transmissions. Is there a way to get udpcast to drop
troublesome receivers in this situation (other than unicast)?
- Has anyone used the --ttl option to multicast over routers?
Does it work (the manpage is unclear)? Does it need special
routers?
- Any other scaling tips? Should I try to go for the full set of
machines at once or break up the broadcast into groups of machines?
If anyone has used udpcast on such large networks, can you share any
experiences.
Rich.
_______________________________________________
Udpcast mailing list
Udpcast(a)udpcast.linux.lu
https://lll.lgl.lu/mailman/listinfo/udpcast -----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.8 -- QDPGP 2.61c
Comment: http://community.wow.net/grt/qdpgp.html
iQA/AwUBSBBkASzGQcr/2AKZEQJoIACePSes1P2lrNvSDnuEaOFL2azBZ+YAoIR4
G3sccClKCiQw8ru/GZ4/lz15
=t3KH
-----END PGP SIGNATURE-----
5176
days inactive
5176
days old
5 comments
3 participants
participants (3)
-
Michael D. Setzer II -
Michael Holroyd -
Richard W.M. Jones