The Defense Advanced Research Projects Agency (DARPA), the Defense Department's technology research arm, is hard at work on a project called Memex that, among other things, will make it easier to locate hidden services on the so-called Dark Web.
Memex program manager Dr. Chris White said the main issue they're trying to address is the one-size-fits-all approach to the Internet in which search results are based on consumer advertising and ranking.
As you may know, search engines like Bing and Google are only able to index roughly five percent of the Internet. Memex wants to scrape content from the millions of sites that get ignored by commercial search engines as well as the thousands of undocumented websites that operate on the TOR network.
DARPA is hoping that, by indexing hidden services sites, they'll be able to uncover patterns and relationships that could ultimately help law enforcement track illegal activity like what took place on the Silk Road.
Another aspect of the project is coming up with a better estimate of just how many hidden services sites there are on the TOR network. White said their best estimates before were in the single-digit thousands but now they believe there could be as many as 40,000 hidden services sites operating at any given time.
Memex has been in the works for the past year. As of now, there are eight partners involved in testing and deploying prototypes although the agency wouldn't reveal who they are.