Memex In Action: Watch DARPA Artificial Intelligence Search For Crime On The 'Dark Web'

Of late, DARPA has shown a growing interest in open sourcing its technology, even if its most terrifying creations, like army robot wildcats designed to reach speeds of 50Mph, are understandably kept private. In a week’s time, the wider world will be able to tinker with components of the military research body’s in-development search tool for the dark web. The Memex technology, named after an mechanical mnemonic dreamt up just as the Second World War was coming to a close, has already been put to use by a number of law enforcement agencies, who are looking to counter crime taking place on networks like Tor, where Hidden Services are protected by the privacy-enhancing, encrypted hosting, often for good, often for bad. In its first year, the focus at Memex has been on tracking human trafficking, but the project’s scope stretches considerably wider.

It’s likely that in the coming weeks many other law enforcement agencies will avail themselves of the search tools, which will land on DARPA’s Open Catalog next Friday (though DARPA told FORBES the release could be pushed back to the following Monday). FORBES got an exclusive look at the front end of one of the search technologies created by one of the Memex team, a group of self-proclaimed hackers called Hyperion Gray.

According to Alejandro Caceres, who heads up the Hyperion Gray team, a handful of his firm’s tools will be available, including “advanced web crawling and scraping technologies, with a dose of Artificial Intelligence and machine learning, with the goal of being able to retrieve virtually any content on the Internet in an automated way”. Its solution to the problem of finding crime on the so-called “dark web” (a term anathema to Tor’s supporters), is called SourcePin. It is trying to overcome one of the main barriers to modern search: crawlers can’t click or scroll like humans do and so often don’t collect “dynamic” content that appears upon an action by a user.

“Our approach to solving this problem is to build a system that sees the web more like a human user with a browser, and therefore actually behaves like a human user by using a browser to crawl the web, to the point of being able to scroll down a page, or even hover over an object on the page to reveal more content…. we are teaching the system how to act like a human and handle virtually any web page scenario. Eventually our system will be like an army of robot interns that can find stuff for you on the web, while you do important things like watch cat videos,” says Caceres...

http://www.forbes.com/sites/thomasbrewster/2015/04/10/darpa-memex-search-going-open-source-check-it-out/