Here is a list of DX11 Graphics Cards.
Here is a hierarchy of how Graphics Cards compare.
Since you have a laptop, upgrades are not as straight forward as with Desktops.
While I have cracked the case on many laptops for various upgrades and fixes... it is still something that makes me a bit nervous as compared to desktops; to do it without damaging something requires some special tools that are not difficult to find, and a good step-by-step guide for your particular upgrade. I have replaced burned out screens, and have removed keyboards and bezels to access motherboard for processor & fan replacement, as well as swapping out oem hd for new ssd, etc ...
For your project... upgrading a video card...
In general, I would think "Probably not advisable" (my opinion, others may differ!)
Why?
a) Not really cost effective, even if you can find the cards... and if you can, they are probably used (refurbished? erm!) and do you really want to take a chance on something someone else is "parting out"?
Maybe someone knows of a source for new / guaranteed mobility cards? While Dell does sell various parts for the 17x, video card replacements are not listed on the site. I suspect the dell video cards are proprietary, not third party, but not sure.
b) Thermal and power concerns for the new cards
May not be adequately supplied by the old laptop & PSU. Will your laptop be able to dissipate the heat generated? How much more power will the new cards require?
c) cpu and gpu should be balanced... either could be bottlenecked by the other.
No point having a more current card if the cpu cannot keep up.
In your case... this is probably not an issue
as it appears both the 18x and the 17x generally use the same cpu.
Here is a report from someone who upgraded their 18x to GTX680 sli.
Here is another report on the 17x and the 680.
If money really is no object, why not just step up your Alienware Laptop to the 18x?
Specs... AM18XR2-10663BK
SLi ENABLED DUAL (2) 2048MB PCI-Express nVIDIA GTX 680M's (4096MB Total) w/ GDDR5 DX11 Video Cards
Heads up though: Not all AM18xR2 units are built the same. Some have different processors, (Slower) and some have different cards (some shipped with the GTX660).
Of course... if you just want to crack the case, do it yourself, and learn,
well, that's a whole 'nuther story.