If you just want to pull it out then you can, just close all files and folders from the drive before you pull it out. Then it should do no significant harm.
I always use "Safely Remove Hardware" just to be safe. It doesn't have to be a tedious/lengthy task either, there's in fact a great time difference between left/right clicking the icon.
If you right click the icon, you need to select your device then click stop.
If you left click it, you just simply select it and it's much faster, so it only takes you about two seconds extra to "safely remove the hardware" than if you just pull it out.