This is a common issue.
What happens, is your computer is capable of pushing out 100fps when the textures and other game data are in your RAM.
Chances are, the game is polling your hard drive to load textures (as you look in a 360 or move to a new area with new objects in the game), which then has to push them into your RAM. Once they are in the RAM, the FPS normalizes at 100fps, but during the texture loading, it drops significantly because of how slow the hard drive is.
The simple fix is adding 1GB of ram.
I suspect if you hit Alt-Tab while gaming and inspected peak memory use in task manage for the .exe of the game that was running, you'd discover it exceeded 1GB.
You could also try turning texture quality down a notch to see if it will drop your paging file usage enough to reduce the fps drops.