That system is probably to much muscle for 98. Let's concentrate and just getting XP to work.
You are loading the correct SATA drive during setup right? Are you using the floppy that CAME with your motherboard? When XP looks on your floppy for a driver, how many drivers does it list and what are they?
After you have XP format the partition, make sure you leave the sata floppy driver in the case because XP may go back and read it again later during setup.
Download the hard drive tools from whatever manufacturer your HDD is. All the big names have a boot floppy you can make to test the drive out.
As for memtest86, once a valid, i.e. non corrupt, floppy is made, it just boots strait into the test and starts testing without you having to do anything. If you have the floppy in the drive and the PC just says to remove it, then it is not bootable. And thus, not correct. Try burning the ISO image instead. Or get a new floppy disk. Or do a FULL format of your floppy, maybe even a scandisk too, and then run the memtest install again.
I am curious to see how the x64 XP will install. I'm thinking your XP is having trouble loading or using either your hard drive (to large), or maybe the x800 PCI-X. Is your install disk XP Home or Pro? I think you said Home. What service pack is in it? If you can, try to get a XP Service Pack 2 CD.
cheers