Usb sockets, from the moment they are activated when you boot your computer, carry a constant 5 volt supply. For this reason, although not reccomended, if you plug a usb device in while the computer turned on, holding the connector loose at the front of the socket and slowly pushing it in, you will see small sparks on occasions if you look closely.
Whether or not it is particularly dangerous in terms of hardware damage to plug and unplug usb devices while the pc is on... well, I guess it depend on the device. Generally speaking, it's pretty safe, but I wouldn't like to guarentee that EVERY usb device is hot swappable.
Of course, I'm only speaking on theory from what I know and what I've seen, but personally, if I'd just spent £300 on some delicate piece of USB hardware, I'd try to avoid the power surge, however unlikely it was to damage it. If it's a gamepad or a mobile phone (cellphone) charger, I don't really care.
One thing I do know though, is that unplugging or replugging USB hardware while the computer is on running windows does sometimes crash the OS, producing a BSoD with a stop message mentioning (usually) USB related files. It doesn't seem to happen all that often though. I'm not 100% sure, but I believe it happens when a device is in use when you unplug it, and using the Safely Remove Hardware function ensures that the device is not in use so that it can be unplugged without crashing windows.
Hope this helps. Any little inaccuracies will be corrected I'm sure.