[Private email being brought back on-list.]
Alain Knaff wrote:
> On Saturday 18 October 2003 00:39, you wrote:
>>Alain Knaff wrote:
>>>On Friday 17 October 2003 02:37, Christopher Curtis wrote:
>>>
>>> ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log
>>
>>I didn't understand this response at all,
>
> This is a Unix command that allows you to "remotely" connect from the
I understand the command just fine. I didn't understand the *reponse*.
>>I am correct in stating that udpcast
>>does not guarantee delivery by requesting resends for missed packets
>>through the return communication channel, yes?
>
> No.
This is completely unclear, fwiw.
>> Your comment made about
>>not binding to the multicast address would indicate further problems
>>with guaranteed delivery.
>
> Huh? Do you mean the --mcast-addr and --mcast-all-addr flags, which
> are necessary for operation in a WAN? Yes, these settings are indeed
> somewhat fiddly, but they do not force you to use the --async mode.
Actually, I was referring to this:
/**
* Set socket to listen on given multicast address Not 100% clean, it
* would be preferable to make a new socket, and not only subscribe it
* to the multicast address but also _bind_ to it. Indeed, subscribing
* alone is not enough, as we may get traffic destined to multicast
* address subscribed to by other apps on the machine. However, for
* the moment, we skip this concern, as udpcast's main usage is
* software installation, and in that case it runs on an otherwise
* quiet system.
*/
int mcastListen(int sock, char *ifname, struct sockaddr *addr) {
My systems will not be "quiet", nor are they used for "software
installation", and picking up random pcakets and inserting them into the
files I am distributing would be a very bad thing. This furthered my
belief that udpcast didn't do any checking for things like out of order
or retransmits.
> You can make a small script that starts the various udp-receivers (via
> ssh) and then the udp-sender, and then put that script into a cron
> job.
Well, knowing this now, I see that two days' worth of my work was for
nothing ... unless I can find an alternate use for it like server crash
recovery. But even then, unicast is probably best in that case. Hmm.
> The --nokbd and --autostart flags are useful for such operation.
I chose '--full-duplex --nokbd --min-clients $ready --max-bitrate 1m'.
The max bitrate was so I can roughly simulate the frame. How does
udpcast maintain integrity when --full-duplex is chosen, as the
documentation states that it doesn't wait for an acknowledgement before
continuing transmission?
> The only difference is a difference in network distance: most current
> users use udpcast on a LAN (traffic needs to cross no routers),
> whereas you seem to plan a usage on a WAN. That is fine, the only
> difference is that in such case some tweaking with the mcast-addr and
> --mcast-all-addr settings might be needed to help the multicast
> traffic traverse the routers.
I noted that in the documentation. It will be a while before these
machines are shipped out. I'm still in the evaluation phase.
>>I'm having (what I hope are) some network problems; the code runs
[...which appear to be dead ports on a switch...]
>>beautifully on my local machine. However, I'm still looking for
>>something a bit more off-the-shelf, if anyone has any pointers...
udpcast may be the answer. I'm going to set my client/server code aside
for the time being. If I find I need it, I'll enhance it a tad to make
it more bittorrent-like, since it's so close.
Chris
Hello,
I've been looking for an application for some time now that doesn't seem
to exist. The more I search, the more links lead to udpcast. My
scenario is basically this: I need to reliably transfer large files
(large as in >2GB) to multiple hosts over a slow WAN.
I've read some of the messages in this list regarding FEC and retries,
but I don't think this is sufficient. I've also looked into rsync batch
mode without much success (I expect the directory layouts to be
different). And in the background I keep thinking about BitTorrent.
What I would ideally want is something like the following:
'N' clients, each running a daemon (or inetd...)
Server Process connects via TCP to each client
Server tells client "Listen on Multicast port <foo>"
"I am going to send you XXXX bytes"
"Save this in a file called <bar>"
Assuming the client 'ok's these requests, the Server then begins
multicasting data packets. I'm thinking the packets are of variable
size, and are of some form:
[seq#|size|<data...>]
Or the size is negotiated up front and not included in the packet.
However, what I also want is the client to record successful and
unsuccessful packet receives, such as "I got packets 1-1003"; "I got
packets 1005-9999".
When the multicast transmission is complete, the server asks each client
"What packets are you missing?" and the client responds in kind. The
server then retransmits the missed packets, and repeats doing this until
all the clients are happy.
Each client will have to store this state on disk, of course. And it
would be nice if (sometime way out in the future) resumption from a
server (process) outage would be possible, but that's not needed initially.
My question, then, is there anything like this available? If not, is
there is any interest in this here? Since I'm not familiar with the
code it would take me quite some time to find my way around. It may not
even be possible with the current code (specifically the packetization
portion). Or is this just outside the scope of udpcast?
Thanks,
Chris
I have an HP server that I'm going to use as an imaging server. It's a
2.8GHz Xeon with a gig of ram and 5 scsi hard drives in a raid 5
configuration. I have a number of new Dell desktop PC's that I'm going
to be imaging. They're P4 2.4 GHz w/ 1 80gig 7200 rpm ide hd and 512 mb
of ram. The network we're using is isolated. We're using a cisco 6509
as the main switch with a gig uplink to the server. From there, we ran
4 network drops to where we image the desktops. Each drop has an 8 port
linksys 10/100 switch and the desktop(s) to be imaged are hooked up to a
linksys switch.
Here's what I have done to create an image on one of the Dell's
1) Zero out the hard drive dd bs=1024k if=/dev/zero of=/dev/hda
2) install windows xp on the system.
3) reboot the dell with the udpcast floppy image. I'm using an e1000
driver. I tell it to ghost /dev/hda. I choose lzop compression with no
additional parameters. I then start the udpcast program as the send and
it waits for the reciever (my hp server) to start.
4) start udpcast on the server. time udp-receiver -f dell_lzop.img
The server starts and tells me that it has found a client so I hit enter
again and the transfer starts.
The transfer averages about 15 mbits/sec at first. After about 5 min it
starts dropping to around 1.6 mbit/sec. When it's the finished, the
size of the image is just over a gig. The transfer takes about 30 min.
Slice % on the sender ranges from .1% to .2% with a few intermittant
timeouts.
Then to send the image back out to another dell PC.
1) boot up dell PC with udpcast floppy. I tell it to image /dev/hda.
I'm still using the e1000 driver. I tell it to use lzop compression and
set it up to be the receiver with no additional parameters.
2) start udpcast sender on the HP server. time udp-sender -f
dell_lzop.img --half-duplex --max-bitrate 500m and hit enter.
The transfer runs at over 80 mbits/sec for about a minute and a half and
slows down after 680 mbytes of the 1 gig image have been transfered.
That's 2/3 of the image is transfered in a minute and a half!. But then
the transfer slows to a crawl of about 600 kbits/sec. Once it slows
down, the sender get's constant timeouts with a couple transmits in
between. The slice % is around .3. The total image time is 71 minutes.
Anyone have any ideas on what I can do to keep this slow down from
happening? I'm pretty sure I've ruled out the network as the problem.
I tried doing a udp-sender -f /dev/zero and udp-receiver -f /dev/null
and I can sustain 85+ mbits/sec for an indefinate amount of time on our
100mbit network. Are there any switches I should be throwing on the
dell pc to keep it from bogging down when writing the image to the hd?
Alain Knaff wrote:
> There are two files in the library directory which list all modules
> which are present: modules.dep and modules.pcimap
> These are normally created automatically when doing a make
> modules_install from the source kernel tree.
> Now that we've found out that the initial problem was only a size
> problem, maybe you can get back to your kernel that you downloaded
>from ftp.kernel.org, and do a make modules_install there, check that
> the files are present and both contain references to the bcm5700
> module.
> Once this is ok, call makeImage with the --module bcm5700 flag, and
it
> should be ok.
I have put the bcm5700.o in /usr/src/linux-2.4.21/drivers/net/ and then
I did "make modules_install" from /usr/src/linux-2.4.21/. The bcm5700.o
does not transfer get transfered over to
/lib/modules/2.4.21-udpcast/kernel/drivers/net. It also does not show
up in /lib/modules/2.4.21-udpcast/modules.dep, nor modules.pcimap. If I
manually put the bcm5700.o in the
/lib/modules/2.4.21-udpcast/kernel/drivers/net it disappears when I do a
"make module_install"
I searched around on google groups, I'm trying not to bother you too
much. It looks like most people do a make config and that's where the I
would tell the kernel to include the bcm5700 module?? Only, I'm using
the .config file from the udpcast website so I don't think I can do a
make config, I'm not sure which options need to be enabled. I tried
adding a line in /usr/src/linux-2.4.21/.config that says
"CONFIG_BCM5700=m" and then did a "make modules_install" but that didn't
seem to do the trick either. I even read the install notes for the
driver itself, but it uses an install script that only installs to the
kernel I'm using....not an alternate one.
The network driver is in the right dir
(/usr/src/linux-2.4.21/drivers/net/bcm5700.c). I just can't get the
"make modules_install" to recognize that it needs to actually use this
module by making the correct entries in modules.dep and modules.pcimap
and to also copy it to the /lib/modules/ dir. Any help would be
appreciated, you've helped me out a great deal so far.
On Thursday 02 October 2003 23:44, you wrote:
> Alain Knaff wrote:
> > Yes, it looks indeed as if it this is not enough for initrd. Try
>
> only
>
> > including those modules that you actually need into the initrd in
> > order to get a smaller initrd (with the --modules flag). Without
> > --modules, a default selection is picked which may not fit, because
> > your kernel is slightly larger than the default.
>
> I was able to run the makeImage pointing both to
> /usr/src/linux-2.4.21/arch/i386/boot/bzImage and to
> /lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage. I can either
> just specify the --module "module name" or I can do a --bzip2 and it
> writes to the disk fine. The boot disk works also. Now I need to put
> the bcm5700 modules on to the disk. I'm not quite sure how to do this.
> I tried putting the compiled bcm5700.o driver into
> /lib/modules/2.4.21-udpcast/build/drivers/net and tried to recompile
> (make oldconfig and then do the other makes). I recompiled from
> /lib/modules/2.4.21-udpcast/build/. Then I did the makeImage and
> pointed it to use the bzImage from
> /lib/modules/2.4.21-udpcast/build/arch/i386/boot/. I also specified
> --modules "bcm5700". However; now I get "Non-existant module "bcm5700".
> Any ideas?
There are two files in the library directory which list all modules
which are present: modules.dep and modules.pcimap
These are normally created automatically when doing a make
modules_install from the source kernel tree.
Now that we've found out that the initial problem was only a size
problem, maybe you can get back to your kernel that you downloaded
from ftp.kernel.org, and do a make modules_install there, check that
the files are present and both contain references to the bcm5700
module.
Once this is ok, call makeImage with the --module bcm5700 flag, and it
should be ok.
Note: with a self-compiled kernel, the --bzip2 flag won't work, unless
you also got the patch from http://shepard.kicks-ass.net/~cc/
as described in
http://udpcast.linux.lu/current/makeImage.html#kernelCompile
But if you only include one single module (bcm5700), the bzip2 option
shouldn't be needed anyways
Alain
Alain Knaff wrote:
> Yes, it looks indeed as if it this is not enough for initrd. Try
only
> including those modules that you actually need into the initrd in
> order to get a smaller initrd (with the --modules flag). Without
> --modules, a default selection is picked which may not fit, because
> your kernel is slightly larger than the default.
I was able to run the makeImage pointing both to
/usr/src/linux-2.4.21/arch/i386/boot/bzImage and to
/lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage. I can either
just specify the --module "module name" or I can do a --bzip2 and it
writes to the disk fine. The boot disk works also. Now I need to put
the bcm5700 modules on to the disk. I'm not quite sure how to do this.
I tried putting the compiled bcm5700.o driver into
/lib/modules/2.4.21-udpcast/build/drivers/net and tried to recompile
(make oldconfig and then do the other makes). I recompiled from
/lib/modules/2.4.21-udpcast/build/. Then I did the makeImage and
pointed it to use the bzImage from
/lib/modules/2.4.21-udpcast/build/arch/i386/boot/. I also specified
--modules "bcm5700". However; now I get "Non-existant module "bcm5700".
Any ideas?
On Thursday 02 October 2003 21:07, you wrote:
> Alain Knaff wrote:
> > Could you check with mdir a: which files are on the floppy?
>
> -rwxr--r-- root 12624 ldlinux.sys*
> -rwxr--r-- root 644137 linux*
> -rwxr--r-- root 80 syslinux*
>
> This leaves me with 781K available. I didn't notice it before, but when
> I run the ./makeImage -t net-mod.tar.gz -k
> /usr/src/linux-2.4.21/arch/i386/boot/bzImage -f /dev/fd0 I get "Disk
> full". Although, I still have space on the floppy I guess it's not
> enough for initrd?
Yes, it looks indeed as if it this is not enough for initrd. Try only
including those modules that you actually need into the initrd in
order to get a smaller initrd (with the --modules flag). Without
--modules, a default selection is picked which may not fit, because
your kernel is slightly larger than the default.
> Alain Knaff wrote:
> > In that case, could you just once try with the pre-
> > compiled kernel. This step will allow me to find out whether
> >it is a problem
> > with the kernel or with the makeImage script.
>
> # ./makeImage -t net-mod.tar.gz -k
> /lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage -f /dev/fd0
> I get the same exact output as when I run the makeImage. Disk full.
> The files are the exact same size also.
The default kernel supports bzip2-compressed initrds. Activate this
with the --bzip2 flag; this will make it small enough to fit.
> When I run the makeImage for the precompiled kernel, am I pointing to
> the wrong bzImage? I get the feeling that I'm not using the makeImage
> command with the precompiled image correctly.
The canonical name of the precompiled image, is
/boot/vmlinuz-2.4.21-udpcast , but I just noticed that indeed there is
an exact copy in
/lib/modules/2.4.21-udpcast/build/arch/i386/boot/bzImage as well.
So the reason why it didn't fit is the abscence of the --bzip2 flag
Alain
Well, the PC that had the 2.4.20-20.9 kernel got hosed and I had to
reinstall. It was a RH 9 install that had been updated.
I installed this time with a Mandrake 9.1 distro. I installed the
udpcast rpm and the kernel-udpcast rpm. I then downloaded the 2.4.21
kernel from ftp.kernel.org and extracted it into /usr/src/linux-2.4.21/.
Then I copied the udpc-config.txt file into that dir and called it
.config. Then I edited the the Makefile and put in the -udpcast. I
then did a "make oldconfig" which ran fine. I also did the make dep &&
make bzImage && make modules && make modules_install and they all ran
fine. Then I ran ./makeImage -t net-mod.tar.gz -k
/usr/src/linux-2.4.21/arch/i386/boot/bzImage -f /dev/fd0 which seemed
to run ok. I tried to boot off of the floppy and I get:
Could not find ramdisk image: initrd
boot:
I've obviously missed something...what am I forgetting to do?
I have compiled the network card module, but haven't tried to build it
into the kernel yet. I'd like to get a working boot disk before I try
and make it hit the network.
>>> alain.knaff(a)lll.lu 09/30/03 18:09 PM >>>
On Tuesday 30 September 2003 23:58, Eric Becker wrote:
> I have some hp pc's that I'm trying to udpcast. They have broadcom
> netxtreme network cards which don't seem to be supported by the
default
> boot disk, nor by the net-mod.tar.gz secondary floppy.
>
> I followed the directions here
> http://udpcast.linux.lu/current/makeImage.html#kernelCompile on making
a
> custom boot disk by compiling my own kernel.
>
> Before doing anything I edited the Makefile and changed the
> "extra-version" parameter. I just deleted what was there and put in
> "-udpcast". I do a cd /usr/src/linux/2.4.20-20.9. Then I copy the
> udpc-config.txt file to this dir and name it .config. Next I do a
"make
> oldconfig" which runs fine and so does the "make dep". However; when
I
> do a "make bzImage" it runs for a while and then it stops with errors
> regarding devfs? What am I doing wrong?
Could you post the exact error messages?
Also, are you sure that this is an original kernel (as distributed by
ftp.kernel.org). Indeed, the version number (2.4.20-20.9) strikes me
as somewhat odd...
> I even tried to download the RPM with a pre-compiled kernel and its
> modules, but I'm unsure what do with it.
Just install it with rpm -i , and then run makeImage
Or, alternatively, use the online image generator at
http://www.udpcast.linux.lu/cast-o-matic
> Can someone please point me in
> the right direction? All I need is a boot disk with the Broadcom
> drivers that gets me udpcasting.
Not sure whether the necessary module is in the pre-compiled kernel.
If not, try getting kernel sources from ftp.kernel.org, and compile
from these
Alain