Of course that performance impact is negligible. You could see from the stratosphere that it's a deliberate flaw. As soon as AMD became popular, they had to introduce the same security hole as Intel had.
Just think of it, during normal execution a memory address is compared to the table of pages available to the process, and this is almost instant. Now, why not use the identical pure-hardware mechanism for speculative address check? Explanation: "because it's too slow". LOL. If it was so slow it would slow down normal program execution. It's the same freaking hardware module. It's almost instant.
On top of that, the speculative and out-of-order instruction execution has more time, since nothing yet depends on it. And sometimes the entire calculation is thrown away in the end, depending on the actual program flow. Which means, even less impact on the performance.
The only reason they want people to keep using memory access without a proper check is to keep the security hole open.