CivOmega Wants Your Open Data APIs To Help Fuel Search Engine

CivOmegaDevelopers are being invited to share their Open Data APIs with CivOmega to help drive the capacity of a natural language search engine.

CivOmega was created by Dan Schultz and colleagues as part of an entry in the recent Knight-Mozilla Open News Hack Day event. Civic hackathons – in which local developers and citizens compete to create visualizations and apps based on open data – have been growing in popularity this year. Janet Wagner reported for ProgrammableWeb in June that over 11,000 people participated in the United States' first annual National Day of Civic Hacking while Greg Bates reported that at the Maine Civic Hackathon, one team created a solution that had the potential to save the local city authority $17,000 compared with its' alternatives.

CivOmega aims to make government open data more accessible to end users – referred to as “outsiders” by the Open News Hack Day in order to describe how local citizens who don’t have any coding skills are unable to write complex queries to sort through the growing mountains of open data. The search engine interface lets users pose a question - such as how many times a lobbyist may have visited Government or how many residents of a certain age or ethnicity live in a particular area - and the CivOmega search engine queries open data sources (in these cases, lobbying records shared by the White House and non-profits like The Sunlight Foundation, and population statistics from US Census data) to respond with the best available information.

Speaking with ProgrammableWeb, Dan Schultz encouraged open data API developers to share their data via the CivOmega Github:

"The best way to contribute an API to CivOmega right now is to write Parser / Match pair and issue a pull request.

CivOmega is basically a collection of pattern matchers that know how to recognize certain types of question. If a pattern matcher recognizes a question it will return an object that knows how to answer it. The details are documented in the ReadMe, but the general idea is that if you have an API and you can think of example questions that your API knows how to deal with, you just need to write a little module that will be able to say to the system 'hey, I know how to answer that!' and issue a pull request with your code to the CivOmega repository."

For API developers wanting to promote their open data repository, when search results are displayed in CivOmega, a link is provided back to the original data source. Currently, data is sourced from the Sunlight Foundation, United States Census Bureau and the White House.

Dan Schultz explains that at present there is a payoff between having more opendata in the backend fuelling the answers and being able to decipher questions in the frontend to determine the best way to respond:

"Take a look at the modules that have been written so far to see how questions are being answered now. It isn't necessarily beautiful yet, but at least it works. We're going to be making this a more streamlined and powerful process eventually, but we can't wait for that to keep growing the project!"

Developers who want to contribute to the tool beyond plugging in their open data APIs are welcome to make contact. Some additional tasks have already been identified as requiring further developer input, including providing a JavaScript SDK and building a Library of parsers to work with open APIs.

Be sure to read the next Government article: The Recent Rise of Government Open Data APIs