Bottom line: The restriction, of course, is due to speed. Microsoft wants optimized games to take full advantage of their new hardware, and specifically the 2.4 GB/s of raw throughput afforded by the custom NVMe SSD.
Three weeks out from the launch of Microsoft's next-gen gaming platform, we are still learning new bits of information about the upcoming systems.
As you may know, the Xbox Series X comes equipped with a custom 1TB NVMe solid-state drive. In a screenshot of the storage devices menu courtesy of Tom's Hardware, we see 781.0GB of available space (presumably after formatting, etc.). A video from YouTube user jackfrags, meanwhile, shows a bit more space at 802.0GB. I'm unsure why the two are different, maybe a different firmware version?
In any case, both sources reveal that games and apps on external drives must be stored on USB 3.0 devices with a capacity of at least 128GB. This restriction will eliminate the ability for some to repurpose older hard drives or lower-capacity flash drives.
Also worth noting is that storage drives must be formatted for the Xbox.
Tom's Hardware further clarifies that games which are optimized for the Xbox Series X |S can be stored on external USB 3.0 drives but can only be played from the internal drive or on an "internal" expansion card. Titles that aren't optimized, like original Xbox games, Xbox 360 titles and Xbox One games, can be installed and played from USB 3.0 devices.
Microsoft's Xbox One Series X and Series S launch on November 10. They are priced at $499 and $299 and come with 1TB and 512GB of internal storage, respectively.