It can matter but it may not, it really depends on the system. If you're talking about mixing PC6400 CAS 7 with PC6400 CAS 6 then it probably won't matter. The system will normally run at whatever the slower ram is. Even mixing different speeds can be okay in some cases.
Some motherboards can be picky about the RAM and mixing ram so it may not always work.