The amount of untapped public information available on the web is quite astounding. But the majority of people have no idea how to access it, and even if they do, the sheer amount of data will probably get them nowhere.
This is where New York startup Enigma comes in. Founded by Hicham Oudghiri and Marc Dacosta, and helmed by CEO Jeremy Bronfmann, Enigma taps into over 100,000 public data sources from state and federal records to SEC filings, lists of frozen assets in the United Kingdom, and free tech company database Crunchbase. The result is an incredibly simplified and intuitive way to find hidden facts and connections in publicly available data.
Here's a small example:
TechCrunch did some extensive testing on Enigma, and according to their findings, looking up 'Google' brings up an extensive list of datasets to dive into, including a listing of the company's 2012 H-1B Visa applications. After jumping into that data set, the site run another search to drill down those Visa applications and view only software engineers, revealing that the U.S. Department of Labor certified 471 of Google's H-1B applications for software engineers and that the average wage was about $122,809.
How does the company get all this information? The data itself comes from a cornucopia of places, but most of Enigma's government data comes from issuing a Freedom of Information Act request to the U.S. General Services Administration for all the top level .gov domains. From there crawlers are used to download all the databases they can find, and algorithmically find connections between all those data points to create a sort of public knowledge graph. Whenever you search for a term on Enigma, the service actually searches around that term to figure out and display whatever applicable data sets it can find.
If you want to read up some more on Enigma, the entire article can be found here.