Lost Circuits has posted an interesting article that should answer most of your questions whether moving to a massive amount of RAM can affect overall system performance negatively (there have been stories around for a while, you know), I'd definitely take a look to this article:
There are performance issues associated with higher system memory configurations; clock skew caused by higher load on the command and address bus necessitates relaxation of some chipset timing parameters, and potentially the reinsertion of the additional pipeline stages that were disabled as a function of enabling PAT.
The penalties in those cases are, however, in the order of 3-6% and not 40-60% that were shown to incur after adding two extra DIMMs. There is something wrong in the state of Denmark or was that the state of Benchmark here? We have some good ideas where the bugs are and have the data to back it up.