17 Oct
2003
17 Oct
'03
12:37 a.m.
Hello,
I've been looking for an application for some time now that doesn't seem
to exist. The more I search, the more links lead to udpcast. My
scenario is basically this: I need to reliably transfer large files
(large as in >2GB) to multiple hosts over a slow WAN.
I've read some of the messages in this list regarding FEC and retries,
but I don't think this is sufficient. I've also looked into rsync batch
mode without much success (I expect the directory layouts to be
different). And in the background I keep thinking about BitTorrent.
What I would ideally want is something like the following:
'N' clients, each running a daemon (or inetd...)
Server Process connects via TCP to each client
Server tells client "Listen on Multicast port <foo>"
"I am going to send you XXXX bytes"
"Save this in a file called <bar>"
Assuming the client 'ok's these requests, the Server then begins
multicasting data packets. I'm thinking the packets are of variable
size, and are of some form:
[seq#|size|<data...>]
Or the size is negotiated up front and not included in the packet.
However, what I also want is the client to record successful and
unsuccessful packet receives, such as "I got packets 1-1003"; "I got
packets 1005-9999".
When the multicast transmission is complete, the server asks each client
"What packets are you missing?" and the client responds in kind. The
server then retransmits the missed packets, and repeats doing this until
all the clients are happy.
Each client will have to store this state on disk, of course. And it
would be nice if (sometime way out in the future) resumption from a
server (process) outage would be possible, but that's not needed initially.
My question, then, is there anything like this available? If not, is
there is any interest in this here? Since I'm not familiar with the
code it would take me quite some time to find my way around. It may not
even be possible with the current code (specifically the packetization
portion). Or is this just outside the scope of udpcast?
Thanks,
Chris
17 Oct
17 Oct
noon
New subject: Reliable multifile transfer over slow links, downed hosts
On Friday 17 October 2003 02:37, Christopher Curtis wrote:
Hello,
I've been looking for an application for some time now that doesn't seem
to exist. The more I search, the more links lead to udpcast. My
scenario is basically this: I need to reliably transfer large files
(large as in >2GB) to multiple hosts over a slow WAN.
I've read some of the messages in this list regarding FEC and retries,
but I don't think this is sufficient. I've also looked into rsync batch
mode without much success (I expect the directory layouts to be
different). And in the background I keep thinking about BitTorrent.
What I would ideally want is something like the following:
'N' clients, each running a daemon (or inetd...)
Server Process connects via TCP to each client
Server tells client "Listen on Multicast port <foo>"
"I am going to send you XXXX bytes"
"Save this in a file called <bar>"
You could set up something like this with ssh starting udp-receivers
at the various "clients":
ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log
The stdout.log and stderr.log (and similar output from the server)
will help you to assess how many packets are retransmitted (some
postprocessing would be needed, because this output is intended for
direct display on terminal, so you may need to stip out the escape
sequences to get the data into a processable form)
Alain
10:39 p.m.
Alain Knaff wrote:
On Friday 17 October 2003 02:37, Christopher Curtis
wrote:
I didn't understand this response at all, so I reread the documentation
and dug around the sources some. I am correct in stating that udpcast
does not guarantee delivery by requesting resends for missed packets
through the return communication channel, yes? Your comment made about
not binding to the multicast address would indicate further problems
with guaranteed delivery.
I decided that it was harder to find an answer than to make one myself,
so I wrote a couple basic perl scripts to implement the functionality I
described above (except for the listen on address/port line). I'll be
performing regular updates of the system, so it has to be something
automated.
And to clear up my previous message, the scenario is that there is one
central server regularly distributing data to single clients at multiple
remote locations or to multiple clients at multiple remote locations.
I'm having (what I hope are) some network problems; the code runs
beautifully on my local machine. However, I'm still looking for
something a bit more off-the-shelf, if anyone has any pointers...
Chris
scenario is basically this: I need to reliably
transfer large files
(large as in >2GB) to multiple hosts over a slow WAN.
What I would ideally want is something like the following:
'N' clients, each running a daemon (or inetd...)
Server Process connects via TCP to each client
Server tells client "Listen on Multicast port <foo>"
"I am going to send you XXXX bytes"
"Save this in a file called <bar>"
You could set up something like this with ssh starting udp-receivers
at the various "clients":
ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log 
11:36 p.m.
New subject: Reliable multifile transfer over slow links, downed hosts
On Fri, 2003-10-17 at 16:39, Christopher Curtis wrote:
Alain Knaff wrote:
Alain was describing an easy method for you to try out UDPcast by
starting the receiver on your multiple clients and logging the results
for analysis.
Yes, UDPcast is suited to your application. The default bidirectional
mode will retransmit missing data, and is reliable and
bandwidth-efficient. The unidirectional mode with FEC may be more
time-efficient in your WAN if you tune it. UDPcast is not hard to use
but will require patience with the documentation. I would definitely try
it for your application, but it sounds like you may be happier with your
own implementation.
On Friday 17 October 2003 02:37, Christopher
Curtis wrote:
I didn't understand this response at all, so I reread the documentation
and dug around the sources some. I am correct in stating that udpcast
does not guarantee delivery by requesting resends for missed packets
through the return communication channel, yes? Your comment made about
not binding to the multicast address would indicate further problems
with guaranteed delivery. scenario is basically this: I need to reliably
transfer large files
(large as in >2GB) to multiple hosts over a slow WAN.
What I would ideally want is something like the following:
'N' clients, each running a daemon (or inetd...)
Server Process connects via TCP to each client
Server tells client "Listen on Multicast port <foo>"
"I am going to send you XXXX bytes"
"Save this in a file called <bar>"
You could set up something like this with ssh starting udp-receivers
at the various "clients":
ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log
7102
days inactive
7102
days old
3 comments
3 participants
participants (3)
-
Alain Knaff -
Christopher Curtis -
Dave King