October 2003 - Udpcast

Re: [Udpcast] Reliable multifile transfer over slow links, downed hosts
by Christopher Curtis 18 Oct '03

18 Oct '03

[Private email being brought back on-list.] Alain Knaff wrote: > On Saturday 18 October 2003 00:39, you wrote: >>Alain Knaff wrote: >>>On Friday 17 October 2003 02:37, Christopher Curtis wrote: >>> >>> ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log >> >>I didn't understand this response at all, > > This is a Unix command that allows you to "remotely" connect from the I understand the command just fine. I didn't understand the *reponse*. >>I am correct in stating that udpcast >>does not guarantee delivery by requesting resends for missed packets >>through the return communication channel, yes? > > No. This is completely unclear, fwiw. >> Your comment made about >>not binding to the multicast address would indicate further problems >>with guaranteed delivery. > > Huh? Do you mean the --mcast-addr and --mcast-all-addr flags, which > are necessary for operation in a WAN? Yes, these settings are indeed > somewhat fiddly, but they do not force you to use the --async mode. Actually, I was referring to this: /** * Set socket to listen on given multicast address Not 100% clean, it * would be preferable to make a new socket, and not only subscribe it * to the multicast address but also _bind_ to it. Indeed, subscribing * alone is not enough, as we may get traffic destined to multicast * address subscribed to by other apps on the machine. However, for * the moment, we skip this concern, as udpcast's main usage is * software installation, and in that case it runs on an otherwise * quiet system. */ int mcastListen(int sock, char *ifname, struct sockaddr *addr) { My systems will not be "quiet", nor are they used for "software installation", and picking up random pcakets and inserting them into the files I am distributing would be a very bad thing. This furthered my belief that udpcast didn't do any checking for things like out of order or retransmits. > You can make a small script that starts the various udp-receivers (via > ssh) and then the udp-sender, and then put that script into a cron > job. Well, knowing this now, I see that two days' worth of my work was for nothing ... unless I can find an alternate use for it like server crash recovery. But even then, unicast is probably best in that case. Hmm. > The --nokbd and --autostart flags are useful for such operation. I chose '--full-duplex --nokbd --min-clients $ready --max-bitrate 1m'. The max bitrate was so I can roughly simulate the frame. How does udpcast maintain integrity when --full-duplex is chosen, as the documentation states that it doesn't wait for an acknowledgement before continuing transmission? > The only difference is a difference in network distance: most current > users use udpcast on a LAN (traffic needs to cross no routers), > whereas you seem to plan a usage on a WAN. That is fine, the only > difference is that in such case some tweaking with the mcast-addr and > --mcast-all-addr settings might be needed to help the multicast > traffic traverse the routers. I noted that in the documentation. It will be a while before these machines are shipped out. I'm still in the evaluation phase. >>I'm having (what I hope are) some network problems; the code runs [...which appear to be dead ports on a switch...] >>beautifully on my local machine. However, I'm still looking for >>something a bit more off-the-shelf, if anyone has any pointers... udpcast may be the answer. I'm going to set my client/server code aside for the time being. If I find I need it, I'll enhance it a tad to make it more bittorrent-like, since it's so close. Chris

1 0

Reliable multifile transfer over slow links, downed hosts
by Christopher Curtis 17 Oct '03

17 Oct '03

Hello, I've been looking for an application for some time now that doesn't seem to exist. The more I search, the more links lead to udpcast. My scenario is basically this: I need to reliably transfer large files (large as in >2GB) to multiple hosts over a slow WAN. I've read some of the messages in this list regarding FEC and retries, but I don't think this is sufficient. I've also looked into rsync batch mode without much success (I expect the directory layouts to be different). And in the background I keep thinking about BitTorrent. What I would ideally want is something like the following: 'N' clients, each running a daemon (or inetd...) Server Process connects via TCP to each client Server tells client "Listen on Multicast port <foo>" "I am going to send you XXXX bytes" "Save this in a file called <bar>" Assuming the client 'ok's these requests, the Server then begins multicasting data packets. I'm thinking the packets are of variable size, and are of some form: [seq#|size|<data...>] Or the size is negotiated up front and not included in the packet. However, what I also want is the client to record successful and unsuccessful packet receives, such as "I got packets 1-1003"; "I got packets 1005-9999". When the multicast transmission is complete, the server asks each client "What packets are you missing?" and the client responds in kind. The server then retransmits the missed packets, and repeats doing this until all the clients are happy. Each client will have to store this state on disk, of course. And it would be nice if (sometime way out in the future) resumption from a server (process) outage would be possible, but that's not needed initially. My question, then, is there anything like this available? If not, is there is any interest in this here? Since I'm not familiar with the code it would take me quite some time to find my way around. It may not even be possible with the current code (specifically the packetization portion). Or is this just outside the scope of udpcast? Thanks, Chris

3 3

excellent transfer rates (kind of)
by Eric Becker 14 Oct '03

14 Oct '03

I have an HP server that I'm going to use as an imaging server. It's a 2.8GHz Xeon with a gig of ram and 5 scsi hard drives in a raid 5 configuration. I have a number of new Dell desktop PC's that I'm going to be imaging. They're P4 2.4 GHz w/ 1 80gig 7200 rpm ide hd and 512 mb of ram. The network we're using is isolated. We're using a cisco 6509 as the main switch with a gig uplink to the server. From there, we ran 4 network drops to where we image the desktops. Each drop has an 8 port linksys 10/100 switch and the desktop(s) to be imaged are hooked up to a linksys switch. Here's what I have done to create an image on one of the Dell's 1) Zero out the hard drive dd bs=1024k if=/dev/zero of=/dev/hda 2) install windows xp on the system. 3) reboot the dell with the udpcast floppy image. I'm using an e1000 driver. I tell it to ghost /dev/hda. I choose lzop compression with no additional parameters. I then start the udpcast program as the send and it waits for the reciever (my hp server) to start. 4) start udpcast on the server. time udp-receiver -f dell_lzop.img The server starts and tells me that it has found a client so I hit enter again and the transfer starts. The transfer averages about 15 mbits/sec at first. After about 5 min it starts dropping to around 1.6 mbit/sec. When it's the finished, the size of the image is just over a gig. The transfer takes about 30 min. Slice % on the sender ranges from .1% to .2% with a few intermittant timeouts. Then to send the image back out to another dell PC. 1) boot up dell PC with udpcast floppy. I tell it to image /dev/hda. I'm still using the e1000 driver. I tell it to use lzop compression and set it up to be the receiver with no additional parameters. 2) start udpcast sender on the HP server. time udp-sender -f dell_lzop.img --half-duplex --max-bitrate 500m and hit enter. The transfer runs at over 80 mbits/sec for about a minute and a half and slows down after 680 mbytes of the 1 gig image have been transfered. That's 2/3 of the image is transfered in a minute and a half!. But then the transfer slows to a crawl of about 600 kbits/sec. Once it slows down, the sender get's constant timeouts with a couple transmits in between. The slice % is around .3. The total image time is 71 minutes. Anyone have any ideas on what I can do to keep this slow down from happening? I'm pretty sure I've ruled out the network as the problem. I tried doing a udp-sender -f /dev/zero and udp-receiver -f /dev/null and I can sustain 85+ mbits/sec for an indefinate amount of time on our 100mbit network. Are there any switches I should be throwing on the dell pc to keep it from bogging down when writing the image to the hd?

1 0

Re: [Udpcast] making a custom boot disk
by Eric Becker 03 Oct '03

03 Oct '03

Alain Knaff wrote: > There are two files in the library directory which list all modules > which are present: modules.dep and modules.pcimap > These are normally created automatically when doing a make > modules_install from the source kernel tree. > Now that we've found out that the initial problem was only a size > problem, maybe you can get back to your kernel that you downloaded >from ftp.kernel.org, and do a make modules_install there, check that > the files are present and both contain references to the bcm5700 > module. > Once this is ok, call makeImage with the --module bcm5700 flag, and it > should be ok. I have put the bcm5700.o in /usr/src/linux-2.4.21/drivers/net/ and then I did "make modules_install" from /usr/src/linux-2.4.21/. The bcm5700.o does not transfer get transfered over to /lib/modules/2.4.21-udpcast/kernel/drivers/net. It also does not show up in /lib/modules/2.4.21-udpcast/modules.dep, nor modules.pcimap. If I manually put the bcm5700.o in the /lib/modules/2.4.21-udpcast/kernel/drivers/net it disappears when I do a "make module_install" I searched around on google groups, I'm trying not to bother you too much. It looks like most people do a make config and that's where the I would tell the kernel to include the bcm5700 module?? Only, I'm using the .config file from the udpcast website so I don't think I can do a make config, I'm not sure which options need to be enabled. I tried adding a line in /usr/src/linux-2.4.21/.config that says "CONFIG_BCM5700=m" and then did a "make modules_install" but that didn't seem to do the trick either. I even read the install notes for the driver itself, but it uses an install script that only installs to the kernel I'm using....not an alternate one. The network driver is in the right dir (/usr/src/linux-2.4.21/drivers/net/bcm5700.c). I just can't get the "make modules_install" to recognize that it needs to actually use this module by making the correct entries in modules.dep and modules.pcimap and to also copy it to the /lib/modules/ dir. Any help would be appreciated, you've helped me out a great deal so far.

1 0

Re: [Udpcast] making a custom boot disk
by Alain Knaff 02 Oct '03

02 Oct '03

On Thursday 02 October 2003 23:44, you wrote: > Alain Knaff wrote: > > Yes, it looks indeed as if it this is not enough for initrd. Try > > only > > > including those modules that you actually need into the initrd in > > order to get a smaller initrd (with the --modules flag). Without > > --modules, a default selection is picked which may not fit, because > > your kernel is slightly larger than the default. > > I was able to run the makeImage pointing both to > /usr/src/linux-2.4.21/arch/i386/boot/bzImage and to > /lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage. I can either > just specify the --module "module name" or I can do a --bzip2 and it > writes to the disk fine. The boot disk works also. Now I need to put > the bcm5700 modules on to the disk. I'm not quite sure how to do this. > I tried putting the compiled bcm5700.o driver into > /lib/modules/2.4.21-udpcast/build/drivers/net and tried to recompile > (make oldconfig and then do the other makes). I recompiled from > /lib/modules/2.4.21-udpcast/build/. Then I did the makeImage and > pointed it to use the bzImage from > /lib/modules/2.4.21-udpcast/build/arch/i386/boot/. I also specified > --modules "bcm5700". However; now I get "Non-existant module "bcm5700". > Any ideas? There are two files in the library directory which list all modules which are present: modules.dep and modules.pcimap These are normally created automatically when doing a make modules_install from the source kernel tree. Now that we've found out that the initial problem was only a size problem, maybe you can get back to your kernel that you downloaded from ftp.kernel.org, and do a make modules_install there, check that the files are present and both contain references to the bcm5700 module. Once this is ok, call makeImage with the --module bcm5700 flag, and it should be ok. Note: with a self-compiled kernel, the --bzip2 flag won't work, unless you also got the patch from http://shepard.kicks-ass.net/~cc/ as described in http://udpcast.linux.lu/current/makeImage.html#kernelCompile But if you only include one single module (bcm5700), the bzip2 option shouldn't be needed anyways Alain

1 0

Re: [Udpcast] making a custom boot disk
by Eric Becker 02 Oct '03

02 Oct '03

Alain Knaff wrote: > Yes, it looks indeed as if it this is not enough for initrd. Try only > including those modules that you actually need into the initrd in > order to get a smaller initrd (with the --modules flag). Without > --modules, a default selection is picked which may not fit, because > your kernel is slightly larger than the default. I was able to run the makeImage pointing both to /usr/src/linux-2.4.21/arch/i386/boot/bzImage and to /lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage. I can either just specify the --module "module name" or I can do a --bzip2 and it writes to the disk fine. The boot disk works also. Now I need to put the bcm5700 modules on to the disk. I'm not quite sure how to do this. I tried putting the compiled bcm5700.o driver into /lib/modules/2.4.21-udpcast/build/drivers/net and tried to recompile (make oldconfig and then do the other makes). I recompiled from /lib/modules/2.4.21-udpcast/build/. Then I did the makeImage and pointed it to use the bzImage from /lib/modules/2.4.21-udpcast/build/arch/i386/boot/. I also specified --modules "bcm5700". However; now I get "Non-existant module "bcm5700". Any ideas?

1 0

Re: [Udpcast] making a custom boot disk
by Alain Knaff 02 Oct '03

02 Oct '03

On Thursday 02 October 2003 21:07, you wrote: > Alain Knaff wrote: > > Could you check with mdir a: which files are on the floppy? > > -rwxr--r-- root 12624 ldlinux.sys* > -rwxr--r-- root 644137 linux* > -rwxr--r-- root 80 syslinux* > > This leaves me with 781K available. I didn't notice it before, but when > I run the ./makeImage -t net-mod.tar.gz -k > /usr/src/linux-2.4.21/arch/i386/boot/bzImage -f /dev/fd0 I get "Disk > full". Although, I still have space on the floppy I guess it's not > enough for initrd? Yes, it looks indeed as if it this is not enough for initrd. Try only including those modules that you actually need into the initrd in order to get a smaller initrd (with the --modules flag). Without --modules, a default selection is picked which may not fit, because your kernel is slightly larger than the default. > Alain Knaff wrote: > > In that case, could you just once try with the pre- > > compiled kernel. This step will allow me to find out whether > >it is a problem > > with the kernel or with the makeImage script. > > # ./makeImage -t net-mod.tar.gz -k > /lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage -f /dev/fd0 > I get the same exact output as when I run the makeImage. Disk full. > The files are the exact same size also. The default kernel supports bzip2-compressed initrds. Activate this with the --bzip2 flag; this will make it small enough to fit. > When I run the makeImage for the precompiled kernel, am I pointing to > the wrong bzImage? I get the feeling that I'm not using the makeImage > command with the precompiled image correctly. The canonical name of the precompiled image, is /boot/vmlinuz-2.4.21-udpcast , but I just noticed that indeed there is an exact copy in /lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage as well. So the reason why it didn't fit is the abscence of the --bzip2 flag Alain

1 0

Re: [Udpcast] making a custom boot disk
by Eric Becker 01 Oct '03

01 Oct '03

Well, the PC that had the 2.4.20-20.9 kernel got hosed and I had to reinstall. It was a RH 9 install that had been updated. I installed this time with a Mandrake 9.1 distro. I installed the udpcast rpm and the kernel-udpcast rpm. I then downloaded the 2.4.21 kernel from ftp.kernel.org and extracted it into /usr/src/linux-2.4.21/. Then I copied the udpc-config.txt file into that dir and called it .config. Then I edited the the Makefile and put in the -udpcast. I then did a "make oldconfig" which ran fine. I also did the make dep && make bzImage && make modules && make modules_install and they all ran fine. Then I ran ./makeImage -t net-mod.tar.gz -k /usr/src/linux-2.4.21/arch/i386/boot/bzImage -f /dev/fd0 which seemed to run ok. I tried to boot off of the floppy and I get: Could not find ramdisk image: initrd boot: I've obviously missed something...what am I forgetting to do? I have compiled the network card module, but haven't tried to build it into the kernel yet. I'd like to get a working boot disk before I try and make it hit the network. >>> alain.knaff(a)lll.lu 09/30/03 18:09 PM >>> On Tuesday 30 September 2003 23:58, Eric Becker wrote: > I have some hp pc's that I'm trying to udpcast. They have broadcom > netxtreme network cards which don't seem to be supported by the default > boot disk, nor by the net-mod.tar.gz secondary floppy. > > I followed the directions here > http://udpcast.linux.lu/current/makeImage.html#kernelCompile on making a > custom boot disk by compiling my own kernel. > > Before doing anything I edited the Makefile and changed the > "extra-version" parameter. I just deleted what was there and put in > "-udpcast". I do a cd /usr/src/linux/2.4.20-20.9. Then I copy the > udpc-config.txt file to this dir and name it .config. Next I do a "make > oldconfig" which runs fine and so does the "make dep". However; when I > do a "make bzImage" it runs for a while and then it stops with errors > regarding devfs? What am I doing wrong? Could you post the exact error messages? Also, are you sure that this is an original kernel (as distributed by ftp.kernel.org) Indeed, the version number (2.4.20-20.9) strikes me as somewhat odd... > I even tried to download the RPM with a pre-compiled kernel and its > modules, but I'm unsure what do with it. Just install it with rpm -i , and then run makeImage Or, alternatively, use the online image generator at http://www.udpcast.linux.lu/cast-o-matic > Can someone please point me in > the right direction? All I need is a boot disk with the Broadcom > drivers that gets me udpcasting. Not sure whether the necessary module is in the pre-compiled kernel. If not, try getting kernel sources from ftp.kernel.org, and compile from these Alain

2 1