Reliable multifile transfer over slow links, downed hosts
Hello, I've been looking for an application for some time now that doesn't seem to exist. The more I search, the more links lead to udpcast. My scenario is basically this: I need to reliably transfer large files (large as in >2GB) to multiple hosts over a slow WAN. I've read some of the messages in this list regarding FEC and retries, but I don't think this is sufficient. I've also looked into rsync batch mode without much success (I expect the directory layouts to be different). And in the background I keep thinking about BitTorrent. What I would ideally want is something like the following: 'N' clients, each running a daemon (or inetd...) Server Process connects via TCP to each client Server tells client "Listen on Multicast port <foo>" "I am going to send you XXXX bytes" "Save this in a file called <bar>" Assuming the client 'ok's these requests, the Server then begins multicasting data packets. I'm thinking the packets are of variable size, and are of some form: [seq#|size|<data...>] Or the size is negotiated up front and not included in the packet. However, what I also want is the client to record successful and unsuccessful packet receives, such as "I got packets 1-1003"; "I got packets 1005-9999". When the multicast transmission is complete, the server asks each client "What packets are you missing?" and the client responds in kind. The server then retransmits the missed packets, and repeats doing this until all the clients are happy. Each client will have to store this state on disk, of course. And it would be nice if (sometime way out in the future) resumption from a server (process) outage would be possible, but that's not needed initially. My question, then, is there anything like this available? If not, is there is any interest in this here? Since I'm not familiar with the code it would take me quite some time to find my way around. It may not even be possible with the current code (specifically the packetization portion). Or is this just outside the scope of udpcast? Thanks, Chris
On Friday 17 October 2003 02:37, Christopher Curtis wrote:
Hello,
I've been looking for an application for some time now that doesn't seem to exist. The more I search, the more links lead to udpcast. My scenario is basically this: I need to reliably transfer large files (large as in >2GB) to multiple hosts over a slow WAN.
I've read some of the messages in this list regarding FEC and retries, but I don't think this is sufficient. I've also looked into rsync batch mode without much success (I expect the directory layouts to be different). And in the background I keep thinking about BitTorrent.
What I would ideally want is something like the following: 'N' clients, each running a daemon (or inetd...) Server Process connects via TCP to each client Server tells client "Listen on Multicast port <foo>" "I am going to send you XXXX bytes" "Save this in a file called <bar>"
You could set up something like this with ssh starting udp-receivers at the various "clients": ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log The stdout.log and stderr.log (and similar output from the server) will help you to assess how many packets are retransmitted (some postprocessing would be needed, because this output is intended for direct display on terminal, so you may need to stip out the escape sequences to get the data into a processable form) Alain
Alain Knaff wrote:
On Friday 17 October 2003 02:37, Christopher Curtis wrote:
scenario is basically this: I need to reliably transfer large files (large as in >2GB) to multiple hosts over a slow WAN.
What I would ideally want is something like the following: 'N' clients, each running a daemon (or inetd...) Server Process connects via TCP to each client Server tells client "Listen on Multicast port <foo>" "I am going to send you XXXX bytes" "Save this in a file called <bar>"
You could set up something like this with ssh starting udp-receivers at the various "clients":
ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log
I didn't understand this response at all, so I reread the documentation and dug around the sources some. I am correct in stating that udpcast does not guarantee delivery by requesting resends for missed packets through the return communication channel, yes? Your comment made about not binding to the multicast address would indicate further problems with guaranteed delivery. I decided that it was harder to find an answer than to make one myself, so I wrote a couple basic perl scripts to implement the functionality I described above (except for the listen on address/port line). I'll be performing regular updates of the system, so it has to be something automated. And to clear up my previous message, the scenario is that there is one central server regularly distributing data to single clients at multiple remote locations or to multiple clients at multiple remote locations. I'm having (what I hope are) some network problems; the code runs beautifully on my local machine. However, I'm still looking for something a bit more off-the-shelf, if anyone has any pointers... Chris
On Fri, 2003-10-17 at 16:39, Christopher Curtis wrote:
Alain Knaff wrote:
On Friday 17 October 2003 02:37, Christopher Curtis wrote:
scenario is basically this: I need to reliably transfer large files (large as in >2GB) to multiple hosts over a slow WAN.
What I would ideally want is something like the following: 'N' clients, each running a daemon (or inetd...) Server Process connects via TCP to each client Server tells client "Listen on Multicast port <foo>" "I am going to send you XXXX bytes" "Save this in a file called <bar>"
You could set up something like this with ssh starting udp-receivers at the various "clients":
ssh udp-receiver -p 9002 -f bar ... >stdout.log 2>stderr.log
I didn't understand this response at all, so I reread the documentation and dug around the sources some. I am correct in stating that udpcast does not guarantee delivery by requesting resends for missed packets through the return communication channel, yes? Your comment made about not binding to the multicast address would indicate further problems with guaranteed delivery.
Alain was describing an easy method for you to try out UDPcast by starting the receiver on your multiple clients and logging the results for analysis. Yes, UDPcast is suited to your application. The default bidirectional mode will retransmit missing data, and is reliable and bandwidth-efficient. The unidirectional mode with FEC may be more time-efficient in your WAN if you tune it. UDPcast is not hard to use but will require patience with the documentation. I would definitely try it for your application, but it sounds like you may be happier with your own implementation.
participants (3)
-
Alain Knaff -
Christopher Curtis -
Dave King