It may be hard to believe, but it might just be the USB Flash Memory itself. I purchased two USB "sticks" - exact same brand, model and size - but one of them transfers files slower then the other. And that is using the same USB 2.0 port (and even tried the 3.0 ports I have also). It still works, but its something you just got to deal with.
What size files are we talking about transferring? What is the average transfer speed a few minutes in? At the end? Also what specific drive? Have you looked for benchmarks on it to see if you are significantly below what you should expect for the drive? These are important questions before we can really get into the cause. I have a pretty solid guide for getting the best possible responses to your questions here in my signature, it may be worth a look.
Most flash drives just plain suck for speed. If you thought you got a good deal for a large size stick then you almost certainly got a slow drive. Fast drives are generally a lot more pricey.
I agree with all the advice above. I'll just add the following:
1. You can use Device Manager to actually see how your hardware connects and verify which controller your device is connected to (and verify that it's USB 2.0). Skim THIS thread for more info about USB and how to View Devices by Connection
2. Also as mentioned, not all USB flash drives are created equal. Some are just faster then others. You can use something like Crystal Disk Mark to benchmark/compare your flash drives