[Udpcast] More on the corrupt file problem

[Kyle Cordes]

Alain Knaff wrote:

> 
> The strange thing is, we do use udpcast for duplicating entire disks,
> most of which are larger than 50GB by now, and we never did notice any
> ill effect. A large piece of data missing in the middle would have been
> pretty obvious, but we've never have seen any of this so far.


Alain,

By any chance do you typically use it like so on your large files?

udp-receiver | some-process    ?

Contrary to my earlier findings, in ongoing testing I have found that if 
I used it like this:


udp-receiver --file foo

I sometimes get bad results; and things like this:

udp-receiver --pipe "lzop -d" --file foo

also sometimes get bad results.

but I noticed that my real scripts do this:

udp-receiver | lzop -d | pg_restore

and I tested like this:

udp-receiver | lzop -d >foo

... and I get correct results. To 5 or 6 receivers. Every night.

Also, I found that having lzop (or other common compression tool) in the 
  loop acts as a guard against data integrity problems - if udp-receiver 
skip or damages data, it would fail lzop's checksums and make the whole 
process fail.

Thus, it looks like there is some issue that comes in to play with 
--file, but not when simply letting the data fall out on stdout.

I'm sitting the issue down for the moment, but later I may beat on it a 
little more to try to track down the specifics of the failure.

I also think that perhaps the "--pipe" and "--file" features are 
unnecessary; that udp-receiver would be better by being simpler, and 
simply assume that the user will redirect the output where they need it.

-- 
Kyle Cordes
http://kylecordes.com