Trouble with tftpd and dhcp and broadcom ethernet client
The udpcast part is working well enough. I made a nice cast-o-matic image for PXE boot with all of the network devices we may need to support. I think cast-o-matic is the neatest think about udpcast and its tools. I'm using an initial menu in PXE boot to select how to launch udpcast (a solution suggested by Alain). For that process, DHCP client fires up and gets an address OK. The client is using a Broadcom BCM5702 based ethernet device. I select the menu item (say sender for whole disk) and then the linux kernel boots fine. DHCP gives out address 192.168.1.207 Now we get into the blue screen section and it shows: Sending bootp request... Alert info, udhcpc V0.9.9_pre) started debug, sending discover... (3 times) info, No lease, failing (OK) At the time the bootp request goes out the dhcp server shows this in the log file: in.tftpd[1902]: tftp: client does not accept options On the client, I use the OK button and see a prompt for Automatic Configuration. I select Yes for that, and I get another status screen showing that: info, udhcpc 0.9.9_pre started debug, Sending discover debug, sending select for 192.168.1.210 info, lease of 192.168.1.210 obtained lease time 2800 Again the daemon log for dhcpd and in.tftpd shows that "client does not accept options" However everything is fine on this second bootp pass. I read on Novell about the -r switch for in.tftpd to disable blksize for another Broadcom ethernet device in the same family. http://www.novell.com/coolsolutions/nnlsmag/features/a_install_via_pxe_boot_... So I've been using this to launch tftpd: in.tftpd -l -s /tftproot -vv -r blksize I'm avoiding inetd since it can stall when too many simultaneous requests come in (security feature, but no use to this situation). The -r blksize didn't change the problem. Once I manually click OK after the first bootp request and say Yes to automatically configuring with DHCP, it does work OK and udpcast works. However it would be better if the process could run without the user intervention at all of the clients. I know this isn't exactly a problem in udpcast itself, but in its surrounding environment, but I'm wondering if anyone might know if this is a problem specific to this client hardware and what might help the problem (other TFTP options to suspend, other kernel with newer driver that might avoid the issue)? There are only 2 machines on the network of my test, so there is no way another DHCP server might be answering the call. --Donald Teed
begin Thursday 08 April 2004 16:19, Donald Teed quote:
The udpcast part is working well enough. I made a nice cast-o-matic image for PXE boot with all of the network devices we may need to support. I think cast-o-matic is the neatest think about udpcast and its tools.
I'm using an initial menu in PXE boot to select how to launch udpcast (a solution suggested by Alain).
For that process, DHCP client fires up and gets an address OK. The client is using a Broadcom BCM5702 based ethernet device.
I select the menu item (say sender for whole disk) and then the linux kernel boots fine. DHCP gives out address 192.168.1.207
Now we get into the blue screen section and it shows:
Sending bootp request... Alert info, udhcpc V0.9.9_pre) started debug, sending discover... (3 times) info, No lease, failing
(OK)
At the time the bootp request goes out the dhcp server shows this in the log file:
in.tftpd[1902]: tftp: client does not accept options
As you correctly point out yourself later in your message, this is not actually a dhcp message, but a tftp message. However, if you already got this far (udpcast menu), it means that tftp did fine anyways (i.e. the message is not a fatal error, but merely a warning).
On the client, I use the OK button and see a prompt for Automatic Configuration. I select Yes for that, and I get another status screen showing that:
info, udhcpc 0.9.9_pre started debug, Sending discover debug, sending select for 192.168.1.210 info, lease of 192.168.1.210 obtained lease time 2800
Again the daemon log for dhcpd and in.tftpd shows that "client does not accept options"
It's weird that you get another in.tftpd message at this point. After booting of the initial images (kernel+initrd), the udpcast system no longer uses tftp. Are you positively sure that no other clients are using the system? Or maybe this is not some "old" message that has actually been logged earlyer (use tail -f to see messages appear in real-time, as soon as they are added to the log...) It might also be interesting to have a tcpdump running in another window, to display packets as they are received. tcpdump -s 1500 -ni eth0 port 67 or port 68 (if eth0 is the interface to which the clients are connected) This displays any packets involving port 67 or 68, i.e. DHCP packets. You can also add "or port 69" to display tftp traffic as well.
However everything is fine on this second bootp pass.
I read on Novell about the -r switch for in.tftpd to disable blksize for another Broadcom ethernet device in the same family.
I do not actually think that tftp is the problem. The kernel and initrd loaded fine, or else you would not even have seen the menu. IMHO, the problem is with dhcpd rather than tftpd. Regards, Alain
Thanks for the quick reply. I've verified the same server used with another client machine not using this Broadcom ethernet boots problem free. So this seems to be a DHCP problem with the Broadcom hardware or drivers. I found this item in a mailing list which seems to be about a similar problem: (Slow DHCP) http://bohnsack.com/lists/archives/xcat-user/2518.html Unless a newer driver for the Broadcom appears, I'll likely go with not using the menu based client and just have it automatically run receiver for when we are in full production. That should avoid the second DHCP request being run. Thanks for your help. Regards, --Donald Teed On Thu, 8 Apr 2004, Alain Knaff wrote:
begin Thursday 08 April 2004 16:19, Donald Teed quote:
The udpcast part is working well enough. I made a nice cast-o-matic image for PXE boot with all of the network devices we may need to support. I think cast-o-matic is the neatest think about udpcast and its tools.
I'm using an initial menu in PXE boot to select how to launch udpcast (a solution suggested by Alain).
For that process, DHCP client fires up and gets an address OK. The client is using a Broadcom BCM5702 based ethernet device.
I select the menu item (say sender for whole disk) and then the linux kernel boots fine. DHCP gives out address 192.168.1.207
Now we get into the blue screen section and it shows:
Sending bootp request... Alert info, udhcpc V0.9.9_pre) started debug, sending discover... (3 times) info, No lease, failing
(OK)
At the time the bootp request goes out the dhcp server shows this in the log file:
in.tftpd[1902]: tftp: client does not accept options
As you correctly point out yourself later in your message, this is not actually a dhcp message, but a tftp message. However, if you already got this far (udpcast menu), it means that tftp did fine anyways (i.e. the message is not a fatal error, but merely a warning).
On the client, I use the OK button and see a prompt for Automatic Configuration. I select Yes for that, and I get another status screen showing that:
info, udhcpc 0.9.9_pre started debug, Sending discover debug, sending select for 192.168.1.210 info, lease of 192.168.1.210 obtained lease time 2800
Again the daemon log for dhcpd and in.tftpd shows that "client does not accept options"
It's weird that you get another in.tftpd message at this point. After booting of the initial images (kernel+initrd), the udpcast system no longer uses tftp. Are you positively sure that no other clients are using the system? Or maybe this is not some "old" message that has actually been logged earlyer (use tail -f to see messages appear in real-time, as soon as they are added to the log...)
It might also be interesting to have a tcpdump running in another window, to display packets as they are received.
tcpdump -s 1500 -ni eth0 port 67 or port 68
(if eth0 is the interface to which the clients are connected)
This displays any packets involving port 67 or 68, i.e. DHCP packets.
You can also add "or port 69" to display tftp traffic as well.
However everything is fine on this second bootp pass.
I read on Novell about the -r switch for in.tftpd to disable blksize for another Broadcom ethernet device in the same family.
I do not actually think that tftp is the problem. The kernel and initrd loaded fine, or else you would not even have seen the menu.
IMHO, the problem is with dhcpd rather than tftpd.
Regards,
Alain
begin Thursday 08 April 2004 18:18, Donald Teed quote:
Thanks for the quick reply.
I've verified the same server used with another client machine not using this Broadcom ethernet boots problem free. So this seems to be a DHCP problem with the Broadcom hardware or drivers.
The first DHCP run by the PXE stack seems to run ok, and it's the second one, run by the menu system, which fails. I've made a new version 20040410 which is able to read the ip configuration read by the PXE stack. To enable it, just add "ipappend 1" on a line by itself to pxelinux.cfg/default This instructs Pxelinux to store the IP address, netmask, etc. that it got from the server into an environment variable that is then read by the udpcast menu system (today's version)
I found this item in a mailing list which seems to be about a similar problem:
(Slow DHCP) http://bohnsack.com/lists/archives/xcat-user/2518.html
Ok, so it is a timing issue. I've also added a new kernel parameter, "netwait", which instructs udpcast to wait this many seconds before attempting to do Dhcp. just add this to the append line in pxelinux.cfg/default : append ... netwait=10 (N.B. Should not be needed if you use ipappend). Alain
This solution sounds promising. I'm not sure where to find the updated version. Is it a newer kernel for PXE boot, udpcast version or something else that is updated? --Donald Teed On Sat, 10 Apr 2004, Alain Knaff wrote:
begin Thursday 08 April 2004 18:18, Donald Teed quote:
Thanks for the quick reply.
I've verified the same server used with another client machine not using this Broadcom ethernet boots problem free. So this seems to be a DHCP problem with the Broadcom hardware or drivers.
The first DHCP run by the PXE stack seems to run ok, and it's the second one, run by the menu system, which fails.
I've made a new version 20040410 which is able to read the ip configuration read by the PXE stack.
To enable it, just add "ipappend 1" on a line by itself to pxelinux.cfg/default
This instructs Pxelinux to store the IP address, netmask, etc. that it got from the server into an environment variable that is then read by the udpcast menu system (today's version)
I found this item in a mailing list which seems to be about a similar problem:
(Slow DHCP) http://bohnsack.com/lists/archives/xcat-user/2518.html
Ok, so it is a timing issue.
I've also added a new kernel parameter, "netwait", which instructs udpcast to wait this many seconds before attempting to do Dhcp.
just add this to the append line in pxelinux.cfg/default :
append ... netwait=10
(N.B. Should not be needed if you use ipappend).
Alain
On Monday 12 April 2004 04:53, Donald Teed wrote:
This solution sounds promising. I'm not sure where to find the updated version. Is it a newer kernel for PXE boot, udpcast version or something else that is updated?
The kernel is the same, it's the menu system which has changed There is an updated rpm on the Udpcast.linux.lu site (20040410), and updated sources for the menu system (udpcast-20040410.tar.gz. Forgot to update the pointer yesterday, but should be ok now), and all images (udpflop.img, udpcd.img,...) are updated as well. Cast-o-matic now generates images with the new feature as well. As the menu system has changed, rather than udpcast itself (the actual send/receive application), the version number displaed at the "Press any key to start" prompt has not changed. Regards, Alain
Hello, Thanks for the new kernel and initrd mechanisms. I've yet to try it with the broadcom dhcp client but will sometime soon. In the meantime I have another problem which is difficult to see what could be wrong. We have old IBM R31 models which are being sold, and they should be templated like they came from IBM - with a Windows XP Home partition and a hidden recovery partition. When the system comes up, one can use the F11 key which causes the BIOS to make a change in the partition table such that the second partition is visible and active. I created an image file for a full disk using udpcast in receive mode, and later sent it to two different machines with the same model of hard drive in them. (I used the updated linux kernel and initrd made from cast-o-matic today.) The udpcast session reports completion without error. Windows will boot fine with the udpcast restored images, but the F11 function is not being recognized. If we boot from Partition Magic CDROM and manually unhide the restore partition and make it active, then the second partition does boot. If we then hide it with Partition Magic, then the F11 feature is available and works to cause the second partition to be unhidden and bootable. This simple unhiding and re-hiding of the recovery partition (the second partition) is enough to repair whatever it is expecting to see for the recovery partition t o be in a bootable condition. We also have used an IBM utility diskette to restore the ability to see the F11 menu option, but this makes no difference. We double checked a drive copied by Symantec Ghost which had been used as the gold master in the udpcast process. On the same notebook, the F11 worked OK, so it is certain that we were working from a good copy of the image. It looks like there is a problem specific to the partition tables which is behind this. I'm very puzzled by this problem. If udpcast is basically doing a dd of the full drive, I can't see where we could be losing any information about the recovery partition. If you have any ideas where we could be losing the F11 restore capability in the copied drives, please let me know. If you don't have any suggestions, I think I'll try an image creation and restore using the Ghost for Unix method (dd, gzip, ftp from Knoppix) and see if it also has a problem with the hidden recovery partition or it can use it from F11 key. --Donald Teed On Mon, 12 Apr 2004, Alain Knaff wrote:
On Monday 12 April 2004 04:53, Donald Teed wrote:
This solution sounds promising. I'm not sure where to find the updated version. Is it a newer kernel for PXE boot, udpcast version or something else that is updated?
The kernel is the same, it's the menu system which has changed
There is an updated rpm on the Udpcast.linux.lu site (20040410), and updated sources for the menu system (udpcast-20040410.tar.gz. Forgot to update the pointer yesterday, but should be ok now), and all images (udpflop.img, udpcd.img,...) are updated as well.
Cast-o-matic now generates images with the new feature as well.
As the menu system has changed, rather than udpcast itself (the actual send/receive application), the version number displaed at the "Press any key to start" prompt has not changed.
Regards,
Alain
I suspect that the setting might be stored in the BIOS's CMOS memory rather than on disk itself. Indeed, if it was on disk, the whole-disk udpcast copy (/dev/hda) would have transferred it just fine. Alain
The same machine was booted with different hard drives, and the master would boot from F11 fine, while the copied disks failed in two instances. I know that the appearance of the F11 option can be toggled by the CMOS and we have a utility to switch it on if it had been disabled. If udpcast is reading from the entire drive then, as I said, I can't see where it might be missing any data. I will try the dd/gzip/ftp disk image method and see what happens. Thanks... --Donald Teed On Mon, 12 Apr 2004, Alain Knaff wrote:
I suspect that the setting might be stored in the BIOS's CMOS memory rather than on disk itself. Indeed, if it was on disk, the whole-disk udpcast copy (/dev/hda) would have transferred it just fine.
Alain
Hello, I'm going to test imaging using something similar to Ghost for Unix and verify if the problem is specific to udpcast, or may be a problem in common with other open source imaging solutions. We are also going to verify that when F11 is the very first thing done after imaging, we still see the failure. (I've wondered if aborting the Windows boot up caused something strange to happen with the F11 feature in CMOS - a Windows boot was usually the first thing the techs tested.) I expect you have your reasons for shutting down the mailing list from going out to other people, or to the archives on the web site, but it is unfortunate that I can't find out if other uspcast users might have experience with a specific problem like this one. Regards, --Donald Teed On Mon, 12 Apr 2004, Alain Knaff wrote:
I suspect that the setting might be stored in the BIOS's CMOS memory rather than on disk itself. Indeed, if it was on disk, the whole-disk udpcast copy (/dev/hda) would have transferred it just fine.
Alain
OK, I've run a test with the equivalent of ghost for unix (but with Knoppix), which does a dd of /dev/hda , piped to gzip and ftped to another machine. It restores with the opposite. The restored image boots and F11 works fine with this type of disk imaging. Here is what I did: 1. uploaddisk to FTP server 2. zero the drive on the notebook with dd if=/dev/zero from Knoppix 3. slurpdisk from FTP to restore to the zeroed drive So it looks like the problem is specific to using udpcast or something else associated with that method (the kernel, etc.). I'm going to do another test with udpcast and the same physical disk to see if there is something related to the same disk being used which is already known to have F11 working. --Donald Teed On Mon, 12 Apr 2004, Alain Knaff wrote:
I suspect that the setting might be stored in the BIOS's CMOS memory rather than on disk itself. Indeed, if it was on disk, the whole-disk udpcast copy (/dev/hda) would have transferred it just fine.
Alain
Hello, Another working theory is that the problem has nothing to do with udpcast itself but with the Linux server we are using on a Dell notebook. I've turned off acpid and the KDE arts sound server in case they might be leading to data corruption. I did notice a message on the server saying "Sound Server Failure error: cpu overload, aborting" This happened shortly after starting X, and before the image creation was done. We are using Xandros Linux as a server on the Dell machine. The client machine at this point is the IBM R31. I'm doing a fresh image creation under these new conditions and then I'll try the restore again. A restore I had done earlier today confirmed that while the BIOS believed that F11 was an option, it still failed to boot the F11 option, so it can't be a BIOS/CMOS thing that is getting in the way. The very same machine and drive had booted from F11 just prior to the restore with udpcast. I guess that given the nature of UDP we have no way of knowing whether data transferred is going through correctly. Is there something that can be done to improve on this situation? Regards, --Donald Teed
begin Tuesday 13 April 2004 20:11, Donald Teed quote:
I guess that given the nature of UDP we have no way of knowing whether data transferred is going through correctly. Is there something that can be done to improve on this situation?
One trick is to use compression (lzop). Although this doesn't prevent bit errors, it would make them very obvious: rather than having slightly different copies, it would abort with a CRC error. On the LLL project, we almost always use compression (it makes cast times much shorter), and in several years of operations, we only had a single case where a bit error occurred and made the compression abort. Alain
begin Tuesday 13 April 2004 14:26, Donald Teed quote:
I expect you have your reasons for shutting down the mailing list from going out to other people, or to the archives on the web site, but it is unfortunate that I can't find out if other uspcast users might have experience with a specific problem like this one.
I'm not really sure what you're refering to here. Our archive seems to be broken since mid January. However, this situation is certainly not intentional, and right now I'm investigating what is going on. Didn't notice it before, as the list is rather low-traffic and you're the first to mention that something is amiss. Stopping/starting mailman didn't help the situation (although our other lists are fine). I've no switched it to monthly volumes (rather than yearly), hoping that this will trigger some internal re-organization that might fix the issue. We'll see whether this messages shows up in the archive... Alain
Strange. Another mailing list I'm on - nSLUG (nslug.ns.ca) stopped with an unknown system error about one month ago. Anyway, hopefully it will be back up soon. --Donald Teed On Tue, 13 Apr 2004, Alain Knaff wrote:
begin Tuesday 13 April 2004 14:26, Donald Teed quote:
I expect you have your reasons for shutting down the mailing list from going out to other people, or to the archives on the web site, but it is unfortunate that I can't find out if other uspcast users might have experience with a specific problem like this one.
I'm not really sure what you're refering to here. Our archive seems to be broken since mid January. However, this situation is certainly not intentional, and right now I'm investigating what is going on. Didn't notice it before, as the list is rather low-traffic and you're the first to mention that something is amiss.
Stopping/starting mailman didn't help the situation (although our other lists are fine). I've no switched it to monthly volumes (rather than yearly), hoping that this will trigger some internal re-organization that might fix the issue. We'll see whether this messages shows up in the archive...
Alain
begin Tuesday 13 April 2004 22:02, Donald Teed quote:
Strange. Another mailing list I'm on - nSLUG (nslug.ns.ca) stopped with an unknown system error about one month ago.
Anyway, hopefully it will be back up soon.
--Donald Teed
On our site, it seems that the udpcast list is the only one affected. All others still archive just fine... And yes, the "Archive messages" options is set to Yes. Well, if nothing else helps, I'll just set up a second list (subscribed to the first) just for managing the archives... Alain
I read someone saying they had excellent performance with lzop, so I switched to an initrd with lzop as the default at the same time I picked up your latest updates on Monday. So I guess it can't be data corruption on that level. I did another test this afternoon - checked that the original did work with F11, sent it, restored to the same machine and hard drive and F11 did not work. This run was with the acpid stopped and KDE sound server not starting on the server machine. Given that a dd style of imaging should be working, where do you suggest I start looking for something that could be more robust? Should I replace busybox with another kernel that might provide better module support for the ethernet, etc.? Should I get rid of Xandros for the server and try another Linux distribution like Slackware for the TFTPD and DHCP platform (which again allows to substitute kernels for something that might be better)? I can only assume the F11 flaw is a symptom of data loss in this process, since the software sees the partition table information as simple 1's and 0's. If it is possible for the F11 feature to be busted, it might also be possible for other data to be wrong but superficially appear to be mostly OK. At this stage in our work we need to verify in the next few days that udpcast will work for us or look for another application or commercial product. Regards, --Donald Teed On Tue, 13 Apr 2004, Alain Knaff wrote:
begin Tuesday 13 April 2004 20:11, Donald Teed quote:
I guess that given the nature of UDP we have no way of knowing whether data transferred is going through correctly. Is there something that can be done to improve on this situation?
One trick is to use compression (lzop). Although this doesn't prevent bit errors, it would make them very obvious: rather than having slightly different copies, it would abort with a CRC error.
On the LLL project, we almost always use compression (it makes cast times much shorter), and in several years of operations, we only had a single case where a bit error occurred and made the compression abort.
Alain
begin Tuesday 13 April 2004 22:17, Donald Teed quote:
I read someone saying they had excellent performance with lzop, so I switched to an initrd with lzop as the default at the same time I picked up your latest updates on Monday. So I guess it can't be data corruption on that level. I did another test this afternoon - checked that the original did work with F11, sent it, restored to the same machine and hard drive and F11 did not work. This run was with the acpid stopped and KDE sound server not starting on the server machine.
Given that a dd style of imaging should be working, where do you suggest I start looking for something that could be more robust?
The first thing I'd check is to do a compare (using the cmp command) between an original (working) hard disk, and a udpcasted disk. In order to do this, build the two disks into a same machine one on the primary controller, one on the secondary, boot from a Knoppix CD, (or from an udpcast CD...) and run a compare: cmp -l /dev/hda /dev/hdc This not only shows whether the disks are equal, but also where exactly the differences lie, if there are any. It is important that between the udpcast and the compare operation, no boot was attempted from these disks (as it might have changed some bytes). Indeed, in Windows, even trivial activity such as moving some window changes the registry, which is enough to make the disk image different ;-) You may verify, after the compare, whether F11 is broken on the udpcasted disk. Maybe what's going on here is that the disk's serial number (can be displayed using hdparm -i /dev/hda) is stored somewhere in the partition table, maybe in some scrambled form. The F11 functionality would only works if the physical serial number, and the one in the partition table agree? It might be interesting to check what happens if you play back an image to the disk where it originally came from: Disk A ===> Disk B ===> Disk A If that works, then I think it might be a serial number issue. (Maybe overwrite Disk A with sth else between the two transfers, or manually switch off the F11 functionality) Another theory is that somehow the disk geometry as it appears to Linux is not the complete disk, and that one or more sectors near the end of the disk do not get copied, because for some reason the kernel (i.e. operating system binary) is mistaken about the exact size of the disk? If those sectors contain information that is critical for the F11 functionality, this might explain stuff. It's rather obvious that this is unrelated to UDP level corruption, or else lzop would have complained very loudly. It must be something that goes on locally in the machine. Maybe also the manufacturer of the machine might have some info how the F11 functionality works, unless he considers that a trade secret, of course ;-) Regards, Alain
On Tue, 13 Apr 2004, Alain Knaff wrote:
The first thing I'd check is to do a compare (using the cmp command) between an original (working) hard disk, and a udpcasted disk.
In order to do this, build the two disks into a same machine one on the primary controller, one on the secondary, boot from a Knoppix CD, (or from an udpcast CD...) and run a compare:
cmp -l /dev/hda /dev/hdc
I'm not sure if I can hook up 2 disks and a cdrom to the notebook at the same time, and I don't have a network bootable Linux I can use at the moment. I'll see if I can do something like this. Another possible test would be to upload an image made from ghost but for 2 different disks, and then do a cmp on the two image files on the server. That would at least verify whether there was something different for a unique serial number.
Maybe what's going on here is that the disk's serial number (can be displayed using hdparm -i /dev/hda) is stored somewhere in the partition table, maybe in some scrambled form. The F11 functionality would only works if the physical serial number, and the one in the partition table agree?
I thought about that, and my test with the equivalent of g4u (Ghost 4 Unix) would be consistant with that possibility of a checksum bit reflecting a serial number or something, but I also did an image and restore to the same physical disk with udpcast yesterday and that failed.
It might be interesting to check what happens if you play back an image to the disk where it originally came from:
Disk A ===> Disk B ===> Disk A
If that works, then I think it might be a serial number issue. (Maybe overwrite Disk A with sth else between the two transfers, or manually switch off the F11 functionality)
I did this with udpcast as the transfer method and I zeroed the drive with dd between the restore. Using that method, the g4u method of image and restore was able to boot with F11. Using the same approach and the same physical disk for the master and restored disk, udpcast failed to provide the F11 boot option (even with the CMOS flag forced to on from an IBM boot floppy).
Another theory is that somehow the disk geometry as it appears to Linux is not the complete disk, and that one or more sectors near the end of the disk do not get copied, because for some reason the kernel (i.e. operating system binary) is mistaken about the exact size of the disk? If those sectors contain information that is critical for the F11 functionality, this might explain stuff.
"The kernel" - would be the busybox operating system? I've been wondering if this is a variable I should replace since using even a 8 month old KNOPPIX I have no problems with the dd gzip ftp method (g4u). I have the feeling that the exercise with cmp would not offer me any more detail than what I know now: that somewhere in the process data is lost. So what I need to do is pinpoint what part of the pieces introduces the flaw.
It's rather obvious that this is unrelated to UDP level corruption, or else lzop would have complained very loudly.
It must be something that goes on locally in the machine.
Maybe also the manufacturer of the machine might have some info how the F11 functionality works, unless he considers that a trade secret, of course ;-)
I was thinking it could be a trade secret that Ghost and a few others know about, but since my other open source option works properly, and udpcast/busybox fails under the same method (keeping the same disk as source and target), I doubt it. Another interesting observation is that simply unhiding and rehiding the restore partition from Partition Magic corrects the F11 problem. Would that mean the error is in the beginning of the disk where the partition table is located? Regards, --Donald Teed
I was hoping to have a bootable USB floppy and such with something like tomsrtbt, but my first attempt didn't work (I don't have access to the hardware where I normally work). We did a test that confirmed data was being changed. udpcast received an image from a master disk. We named that first. Then it sent that image to a fresh disk and that disk was used to receive another image on the DHCP server ("second"). In summary, the first image had made one trip through the process, and the second one had made three transfers since the original (up, down, back up). When cmp -l was used to compare the compressed image files, they were mostly the same except for about 7 bytes around the beginning - I didn't let it finish but waited for about 15 minutes and there were no new differences reported. So that confirms what I suspected - that the partition table was getting corrupted. I'm still working on the theory that the tg3 versus bcm5700 ethernet modules has something to do with this, and it is a type of error specific to UDP that would not show up in something using TCP/IP. I'm hoping that the newer kernel will contain some bug fixes for the tg3 module. I'll switch to Slackware so that I'll have more control over using kernels straight from the man. I see on Dell's site that even Ghost 7.5 has been a problem with Broadcom's DOS ethernet drivers. --Donald Teed On Tue, 13 Apr 2004, Alain Knaff wrote:
begin Tuesday 13 April 2004 22:17, Donald Teed quote:
I read someone saying they had excellent performance with lzop, so I switched to an initrd with lzop as the default at the same time I picked up your latest updates on Monday. So I guess it can't be data corruption on that level. I did another test this afternoon - checked that the original did work with F11, sent it, restored to the same machine and hard drive and F11 did not work. This run was with the acpid stopped and KDE sound server not starting on the server machine.
Given that a dd style of imaging should be working, where do you suggest I start looking for something that could be more robust?
The first thing I'd check is to do a compare (using the cmp command) between an original (working) hard disk, and a udpcasted disk.
In order to do this, build the two disks into a same machine one on the primary controller, one on the secondary, boot from a Knoppix CD, (or from an udpcast CD...) and run a compare:
cmp -l /dev/hda /dev/hdc
This not only shows whether the disks are equal, but also where exactly the differences lie, if there are any.
It is important that between the udpcast and the compare operation, no boot was attempted from these disks (as it might have changed some bytes). Indeed, in Windows, even trivial activity such as moving some window changes the registry, which is enough to make the disk image different ;-)
You may verify, after the compare, whether F11 is broken on the udpcasted disk.
Maybe what's going on here is that the disk's serial number (can be displayed using hdparm -i /dev/hda) is stored somewhere in the partition table, maybe in some scrambled form. The F11 functionality would only works if the physical serial number, and the one in the partition table agree?
It might be interesting to check what happens if you play back an image to the disk where it originally came from:
Disk A ===> Disk B ===> Disk A
If that works, then I think it might be a serial number issue. (Maybe overwrite Disk A with sth else between the two transfers, or manually switch off the F11 functionality)
Another theory is that somehow the disk geometry as it appears to Linux is not the complete disk, and that one or more sectors near the end of the disk do not get copied, because for some reason the kernel (i.e. operating system binary) is mistaken about the exact size of the disk? If those sectors contain information that is critical for the F11 functionality, this might explain stuff.
It's rather obvious that this is unrelated to UDP level corruption, or else lzop would have complained very loudly.
It must be something that goes on locally in the machine.
Maybe also the manufacturer of the machine might have some info how the F11 functionality works, unless he considers that a trade secret, of course ;-)
Regards,
Alain
begin Friday 16 April 2004 04:27, Donald Teed quote:
When cmp -l was used to compare the compressed image files, they were mostly the same except for about 7 bytes around the beginning - I didn't let it finish but waited for about 15 minutes and there were no new differences reported. So that confirms what I suspected - that the partition table was getting corrupted.
This is very interesting. Now, if we could also have the exact output of cmp -l, it would be great! My theory about what happens is that on first boot, the BIOS changes these for some strange reason. Maybe, for some yet unknown reason, the BIOS thinks that the disk needs "initializing" and resets those bytes which might contain certain settings that it thinks are his. It might be interesting to find out _when_ exactly they change. One method is to run a "hexdump -n 512 /dev/hda" on the disk at various stages of the duplication. This prints out a hexdump of the first 512 bytes. Then just check whether the 7 bytes have their "good" state or the changed state. N.B. You can run this command right after the udp-receive without rebooting by pressing Alt-F2 and pressing return. This gives you a shell where you can run various commands. You can get back to the menu with Alt-F1 You may need to prefix the command with the word "busybox": busybox hexdump -n 512 /dev/hda You can also specify an offset: busybox hexdump -n 256 -s 256 /dev/hda This prints out the bytes 256 til 511 (practical to print out only around the offset where you expect the change, so you don't need to wade through 32 lines of output...). My gut feeling is that right after udp-receive, you find the 7 bytes in their "original" (good) state, but that they are change after first reboot. Or maybe, they only change for certain kinds of reboot (i.e. they remain untouched for network boot, but change when attempting to boot from the disk). I think we can reasonably rule out the following hypothesises: - Network transfer corruption: No, because then lzop compression would fail in a *very* obvious way (and corruption would be randomly spread accross the whole disk, rather than limited to 7 bytes somewhere in the MBR / partition table) - Hard disk read or write corruption: If that was the case, corruption would be spread out evenly accross the whole disk - Different interpretation of hard disk geometry: We would see corruption near the end of the disk, rather than near the beginning. Regards, Alain _______________________________________________ Udpcast mailing list Udpcast@udpcast.linux.lu http://udpcast.linux.lu/mailman/listinfo/udpcast
On Fri, 16 Apr 2004, Alain Knaff wrote:
One method is to run a "hexdump -n 512 /dev/hda" on the disk at various stages of the duplication. This prints out a hexdump of the first 512 bytes. Then just check whether the 7 bytes have their "good" state or the changed state.
N.B. You can run this command right after the udp-receive without rebooting by pressing Alt-F2 and pressing return. This gives you a shell where you can run various commands. You can get back to the menu with Alt-F1
I'm using a PXEboot initrd generated today and with the 2.4.26 linux kernel. I checked the box to include a built in shell. When activated it reports itself as Busybox 1.00-pre10 running msh. I have not had much luck with the built in shell. help works and it reveals only these commands are available: break cd continue eval exec exit export help login newgrp read readonly set shift times trap umask wait I tried the hexdump command you mention with or without busybox in front and it reports not found. Basic stuff like 'ls' is also not found. I get the feeling I'm doing something wrong.
In general, should a disk image be identical each time when compressed with lzop? I mean, does it make sense to verify the image by doing this: cmp image1.lzop image2.lzop where image2 was created by sending and receiving again image1? --Donald Teed On Fri, 16 Apr 2004, Alain Knaff wrote:
begin Friday 16 April 2004 04:27, Donald Teed quote:
When cmp -l was used to compare the compressed image files, they were mostly the same except for about 7 bytes around the beginning - I didn't let it finish but waited for about 15 minutes and there were no new differences reported. So that confirms what I suspected - that the partition table was getting corrupted.
This is very interesting. Now, if we could also have the exact output of cmp -l, it would be great!
My theory about what happens is that on first boot, the BIOS changes these for some strange reason. Maybe, for some yet unknown reason, the BIOS thinks that the disk needs "initializing" and resets those bytes which might contain certain settings that it thinks are his.
It might be interesting to find out _when_ exactly they change.
One method is to run a "hexdump -n 512 /dev/hda" on the disk at various stages of the duplication. This prints out a hexdump of the first 512 bytes. Then just check whether the 7 bytes have their "good" state or the changed state.
N.B. You can run this command right after the udp-receive without rebooting by pressing Alt-F2 and pressing return. This gives you a shell where you can run various commands. You can get back to the menu with Alt-F1
You may need to prefix the command with the word "busybox":
busybox hexdump -n 512 /dev/hda
You can also specify an offset:
busybox hexdump -n 256 -s 256 /dev/hda
This prints out the bytes 256 til 511 (practical to print out only around the offset where you expect the change, so you don't need to wade through 32 lines of output...).
My gut feeling is that right after udp-receive, you find the 7 bytes in their "original" (good) state, but that they are change after first reboot. Or maybe, they only change for certain kinds of reboot (i.e. they remain untouched for network boot, but change when attempting to boot from the disk).
I think we can reasonably rule out the following hypothesises:
- Network transfer corruption: No, because then lzop compression would fail in a *very* obvious way (and corruption would be randomly spread accross the whole disk, rather than limited to 7 bytes somewhere in the MBR / partition table) - Hard disk read or write corruption: If that was the case, corruption would be spread out evenly accross the whole disk - Different interpretation of hard disk geometry: We would see corruption near the end of the disk, rather than near the beginning.
Regards,
Alain
begin Monday 19 April 2004 16:45, Donald Teed quote:
I have not had much luck with the built in shell. help works and it reveals only these commands are available:
break cd continue eval exec exit export help login newgrp read readonly set shift times trap umask wait
Cast-o-matic had an error, which made it ignore the "full featured shell" option. I've uploaded a new version, which solves this problem Regards, Alain
begin Monday 19 April 2004 17:52, Donald Teed quote:
In general, should a disk image be identical each time when compressed with lzop? I mean, does it make sense to verify the image by doing this:
cmp image1.lzop image2.lzop
where image2 was created by sending and receiving again image1?
No, unfortunately this doesn't work out: lzop includes the date in the compressed file, so compressing twice the same device yields different results. Obviously, if you decompress these two different files again, you get the same contents however. Thus, for comparison purposes, best work on *un*compressed data. N.B. If you do the test on a file (rather than on a device), it will work, because in that case, the date used is the one from the file, rather than current time. Alain
OK, that lzop tip explains one of the sources of "error" I was seeing - it was just the timestamp in the lzop compressed file that made cmp of images differ. I've switched to gzip for testing of images being identical. I have good news... Switching to the 2.6.5 kernel for a better tg3 driver on the Broadcom 5702 ethernet has resolved the problem I had with F11 restore breaking. We are now getting identical duplications of the IBM R31 models. I can only guess that there was some timing issue with UDP multicast in the tg3 driver that was fixed since 2.4.24 (the kernel that originally was used on the udpcast server and had failed to produce accurate images). A big thanks to Alain for the kernel updates, fixes (busybox shell), lots of helpful information on DHCP options, and troubleshooting advice! I'm a fan. We need to get udpcast T-shirts or something. This must be twice as cool as Samba. --Donald Teed On Mon, 19 Apr 2004, Alain Knaff wrote:
begin Monday 19 April 2004 17:52, Donald Teed quote:
In general, should a disk image be identical each time when compressed with lzop? I mean, does it make sense to verify the image by doing this:
cmp image1.lzop image2.lzop
where image2 was created by sending and receiving again image1?
No, unfortunately this doesn't work out: lzop includes the date in the compressed file, so compressing twice the same device yields different results.
Obviously, if you decompress these two different files again, you get the same contents however.
Thus, for comparison purposes, best work on *un*compressed data.
N.B. If you do the test on a file (rather than on a device), it will work, because in that case, the date used is the one from the file, rather than current time.
Alain
participants (2)
-
Alain Knaff -
Donald Teed