I play Dota 2 and with my current video card GeForce 9400 GT its not going good .
No wonder! The GT-9400 is less potent than the integrated graphics in the new Intel "Ivy Bridge" CPUs. Most notably the CPUs with the HD-4000 IGP.
I don't see a reason not to buy the graphics card, and give it a shot. Hey, if it doesn't work out, you've got one less part to buy for your next build.
Your LGA-775 945 chipset board is ancient history. Possibly your PSU as well. IDK whether or not the new video card will over tax it. The Pentium D processors are power hounds, in and of themselves. With an old PSU, one of those, and a new graphics card, sounds like a recipe for disaster.
So, my best advice, try the card, but replace your PSU right along with it.
Without considering the video card; Per
this page you are at the minimum requirements for the game. You are well below the recommended specs.
Well, the Pentium D he has, is a bit punchier than the minimum recommended P-4 they're showing. His current video card is a turd, plain and simple.
This isn't a terribly high spec game, so a new graphics card, along with a PSU might keep him happy for a while.
The memory issue, can't be resolved without a 64 bit OS, so there's no point in worrying about it, or buying a copy of Win 7 or 8, to stuff in a PC this old.
Try it, abide with it, save for a new PC. In that order.
Plan on transferring any parts you buy from now on, to the new machine.