Kyle Cordes wrote:
Contrary to my earlier findings, in ongoing testing I have found that if I used it like this:
udp-receiver --file foo
I sometimes get bad results; and things like this:
udp-receiver --pipe "lzop -d" --file foo
also sometimes get bad results.
Is this also the case if you use the --pipe option with _both_ udp-receiver and udp-sender? It could well be that synchronization between write and lseek works correctly if both are performed by 2 different processes (which is the case when using --pipe, but you'll have to use it on both ends, because both udp-receiver and udp-sender use the same kind of code for "uncompressed" printout).
If so, I'll disable the "decompressed" file position printout for the case without pipe, as in that case it doesn't make sense anyways...
but I noticed that my real scripts do this:
udp-receiver | lzop -d | pg_restore
and I tested like this:
udp-receiver | lzop -d >foo
... and I get correct results. To 5 or 6 receivers. Every night.
Yes, in that case, udp-receiver will write to a pipe, which is not seekable, and which does is not affected by the bug.
Also, I found that having lzop (or other common compression tool) in the loop acts as a guard against data integrity problems - if udp-receiver skip or damages data, it would fail lzop's checksums and make the whole process fail.
exactly. However, if the bug still exists even with the --pipe option, it would unfortunately affect the data _before_ it is compressed by lzop (or _after_ it has been decompressed by lzop -d), so that would not help here...
Regards,
Alain