Project Memex is Darpa's search engine for the Dark Web

Shawn Knight

Posts: 15,240   +192
Staff member

darpa developing search engine dark web search tor tor network deep web silk road memex hidden services indexing

The Defense Advanced Research Projects Agency (DARPA), the Defense Department’s technology research arm, is hard at work on a project called Memex that, among other things, will make it easier to locate hidden services on the so-called Dark Web.

Memex program manager Dr. Chris White said the main issue they’re trying to address is the one-size-fits-all approach to the Internet in which search results are based on consumer advertising and ranking.

As you may know, search engines like Bing and Google are only able to index roughly five percent of the Internet. Memex wants to scrape content from the millions of sites that get ignored by commercial search engines as well as the thousands of undocumented websites that operate on the TOR network.

DARPA is hoping that, by indexing hidden services sites, they’ll be able to uncover patterns and relationships that could ultimately help law enforcement track illegal activity like what took place on the Silk Road.

Another aspect of the project is coming up with a better estimate of just how many hidden services sites there are on the TOR network. White said their best estimates before were in the single-digit thousands but now they believe there could be as many as 40,000 hidden services sites operating at any given time.

Memex has been in the works for the past year. As of now, there are eight partners involved in testing and deploying prototypes although the agency wouldn’t reveal who they are.

Permalink to story.

 
I'd be interested in reading a paper on the technicals. Perhaps someone can correct me, but I was under the impression that hidden services were not inherently discoverable?

Also, Tor cannot be too pleased with this. As I understand it, I2P is now the bee's knees of anonymity and less than legal affairs. This would only put another nail in their coffin.
 
40 000 tor webs? maybe, but all the deepweb is not just tor, i2p or any other, there are also almost all intrapages from companies and foreign goverments, who are contected to the web but blocked or not indexed, so DARPA is trying to acces the North korea and russia intranet thats for sure, I doubt they will work with law enforcement to catch small child p**n sites, oh and remember local shared services like samba shared folders, can be indexed using that technology, because if they are going to index everything is using the ipv6 topology feature, so I hope everyone has their nude pics, on local encrypted drives with long really long passwords, btw linux and mac users need to fear it too.
 
Interestingly Memex is malay (or indonesian) crass words for Vagina
 
Unfortunately the Tor network has become notorious for child exploitation and not just your average torrent downloading.
 
Seems growth is their biggest enemy. Even with everyones bandwidth hobbled, their web ticks and trackers arent able to crawl through every orifice and siphon data fast enough. Must be a headache.
 
Back