Method 1:
Play a video in windowed mode and move the window around with the mouse. If the video picture "falls behind" the window, you have an overlay.
Method 2:
Take a screenshot with Print Scrn, paste it to Paint. If you see the video image, then there is no overlay. If there is a black rectangle (or you can see the video from the player in the background if you left the player running when starting up Paint) then you have an overlay video.
WMP lets you turn off overlay (Tools -> Options -> Performance -> Advanced).
RealPlayer doesn't use it.
BTW not using overlays is a performance hit, so you should use it only for playing videos on TV.