Thoughts of an #OpenData Developer

31 Aug 2010

Figured a helpful post is a good one as any to start a blog on.  Part-thank-you and feedback for the  #OpenData government (and non-government) advocates here in Edmonton, and I hope it’ll be useful discussion material for the other open data movements  around the world, particularly because I see Statistics Edmonton as an app that I’d like to push beyond the competition and on with other cities.

Quick Context:

It’s 2010, and the City of Edmonton launches the Apps4Edmonton competition to promote the development of municipal applications using the data-sets from the Edmonton Open Data Catalogue.  It’s a pretty smart move, resulting in over 30 application submissions, including one from me.  I won’t go into the details of my app (leaving it for the next post), but suffice it to say that I had to run through a number of the datasets for it.

What really worked well:

There’s actually quite a bit, here are some highlights.

Grassroots-Level Interaction and Participation

Local software developer Mack D. Male writes down ideas during a principals for open data brainstorming session at City Hall on Saturday November 21, 2009. The City of Edmonton hosted the workshop to explore how to make more of its data available online in accessible formats, for the benefit of all Edmontonians.(photo by Ryan Jackson, copied from Edmonton Journal)

Before it really began, it started with a conversation.  I recall the dialogue and interest in working with local citizens, developers, and interested parties early on last year, and the efforts of the City of Edmonton staff (Devin Serink, James Rugge-Price, Chris Moore, and more I’ve missed) helped build an early community of people who understood the vision of Open Data in Edmonton.  It also helped that the Edmonton Journal wrote about it as well!

Ideas

80+ ideas, a number of which helped form the basis of applications that ultimately got developed.  That it was open to anyone to suggest applications meant that applications could be built for the everyday citizen.

Getting the word out

A blog post here, event invites sent out through the local tech people (@camlinke), and plenty of twitter messages meant that local developers would know about the competition and learn more about how to participate in it.

Prizes

These provided an incentive to entice a number of developers, myself included, to participate.  The different categories helped me explore potential applications to build (or not build), and I suspect more varieties of apps have come through because of this.

Minor Suggestion:

If a competition does run again, it would be interesting to see an idea entry officially by the City. In this scenario, the City would already have a need that it needs to have filled, and a number of local companies – or upstart entrepreneurs – could look to fill it.

Suggestions to Help Developers Build Sustainable, Local and Globally Transferable Open Data Apps

This is a broader, more “meta” level of thought as these apply to the general open data movement (thus to all cities/governments), and I’m hoping that these get discussed at conferences and solutions are being sought.

1) A Consistent Catalogue of Catalogues, Datasets

Take a peek into the data catalogues of San Francisco, New York City, City of Nanaimo, City of Toronto, and many more, and the consistent thing you’ll find is inconsistencies in the information available and  the format in which they’re made available in (for instance, zipped/unzipped, KML/none).  Not only will  developers need to find and familiarize themselves with different sites, designs, and methods of getting the data, they’ll also have to to adapt to each open data project’s data formats and content.  Even for data that updates on a yearly basis, that’s a lot of manual processing when one considers the number of cities undertaking their own open data project.

2) Automatic Updates

It’s currently up to the developer to manually check the data catalogue to see if (a) the current information is the accurate and (b) there isn’t a new dataset that supercedes the current one in scope, purpose, or other category.  Anything from neighbourhoods to garbage zones to political boundaries can change on a yearly basis, and historical data, which is particularly useful to identify trends, isn’t easily available for an app to identify or consume readily.

A Solution to Both?

I can almost hear someone yell “Why not an API?”  From a developer’s point of view, assuming there are APIs for specific types of data, that sounds like a wonderful idea.

That said, I’m not sure that’s the answer – at least not at the municipal level.  There’s enough work for each municipality to do, and creating/maintaining APIs shouldn’t be a core function.  Worse, I can imagine multiple cities having their own APIs, which wouldn’t solve the issue of non-consistent methods of getting information.

I think the answer probably lies in standardization efforts by an organization that would be comprised of member cities participating in the open data movement.  I’m not sure if the Open Data Foundation is the de facto organization for this, but I imagine such an organization gaining good initial traction if it could manage to standardize just one dataset to start off with.  Devin notes that among the leaders of standardization, there is openplans.org, who also run openmuni.org and open311.org.

In Summary…

… we looked at a few things that worked out well so far, as well as a number of things that developers would need some support on in order to push the cumulative potential of these applications further.  I’d be happy to answer any questions or chat further about this, but in the meantime, thanks to the people who made the competition possible so far, and I hope this helps organizers of future open data projects see a bit of a developer’s perspective.  Until later!

Update:

Added more people working on the project in the city, standardization leader.  Thanks to @dserink.

Also, if you liked my app, please vote for it!

2 Responses to Thoughts of an #OpenData Developer

Avatar

Gregg Coppen

September 1st, 2010 at 6:12 am

Very informative post. I am new to working with #opendata but think that you have raised some very relevant points here about standardisation and updates. Being fairly new to the city, it is also nice to know some more history about the apps competition and open data sources in general

This competition is a very progressive one and I have enjoyed the experience of participating in it. I am impressed by the standard of apps submitted and think that your submission for Statistics Edmonton is a really useful one.

I really like the beta feature of citizens being able to add their own datasets via an excel sheet submission. This is a good start towards building further on the already available sets and is a very innovative feature that lends value to the idea of #opendata.

And a big up to Devin, Chris, James & team for making this idea a reality and providing great data to work with in a variety of formats and to Mack, Cam and Edmontons tech community for helping to make it the success it already looks to be.

I look forward to joining the conversation and hopefully adding my own 2 cents

[WORDPRESS HASHCASH] The poster sent us ’0 which is not a hashcash value.

Avatar

Tweets that mention Thoughts of an #OpenData Developer - Ideas Worth Sharing -- Topsy.com

September 1st, 2010 at 2:26 pm

[...] This post was mentioned on Twitter by Mack D. Male, statsyeg and Eugene Chen, Eugene Chen. Eugene Chen said: Thoughts of an #OpenData Developer http://bit.ly/arZEUj #gov20c #opengov #gov20 open data #yeg [...]

[WORDPRESS HASHCASH] The comment’s server IP (208.74.66.43) doesn’t match the comment’s URL host IP (74.112.128.10) and so is spam.

Comment Form

About ideaOwl

Tech Entrepreneur, MBTI INFJ, random idea generator in the evenings. Been in the military, developed financial risk rating software, and worked in Waterloo, Singapore, Boston.

Photostream

My Tweets

Error: Twitter did not respond. Please wait a few minutes and refresh this page.