Open Data








Open Government

Before we discuss open data, we need to talk about open government.
Open government is the governing doctrine which holds that citizens have the right to access the documents and proceedings of the government to allow for effective public oversight (Lathrop, Daniel; Ruma, Laurel, eds. (February 2010). Open Government: Transparency, Collaboration and Participation in Practice).
The City of Edmonton is an Open City. 
An open city creates opportunity for diverse input and participation by inviting citizens to play a larger role in shaping their community and enabling social and economic growth.

Open City Principles


Inclusive

Edmontonians and City employees have access to information, programs and services and the democratic processes that shape municipal decisions. This means reducing socioeconomic, physical and technical barriers, creating accessible channels for delivery of programs and services. 

Innovative 

With the participation of Edmontonians and employees, the City envisions, creates and fosters new approaches, and efficient and sustainable practices.

Transparent 

The City is a steward of the information we manage and the services we provide on behalf of Edmontonians. Being transparent with Edmontonians and internally promotes trust and accountability. The City’s information is a public asset; consistent with privacy and protection legislation, it is disclosed in forms that Edmontonians can readily find and use. 

Participatory 

The City believes that a democracy values and respects public input and engages people in decision making. Community consultation provides valuable input into the decision-making process.

Collaborative 

The City actively engages Edmontonians, non-profit organizations, businesses and the community to design and deliver programs and services in an integrated and effective manner. 

Open City Framework

The Open City framework illustrates a sample of initiatives and programs supporting open city goals and objectives.
The 5 goals are supported by cultural transformation throughout the City.

What is Open Data?


Technically open 

The data must be available in a machine-readable format. In other words, it must have the ability to be easily read and processed by a computer.

Legally open 

The data must be licensed in a way that anyone can use it, without restrictions.

Technically Open

Data is easily accessible to its intended audience.



developers / programmers            data should have an API (application programming interface)
researchers in academia                 data should be available in a bulk download
average citizens                                data should be available without requiring software purchases


Legally Open

Legally open means it must be free for all users, universal participation.
It cannot be restricted to a specific use (e.g. university ONLY).


Many are uncomfortable with the aspect of unrestricted use. Governments are normally risk averse and have liability fears that never play out. 

It is simply a fear of the new way forward to a more open and transparent government.

Our Terms of Use cover the major concerns we have seen over the years.



Why Open Data?

Open data programs aim to proactively disclose government data, creating transparency and accountability. This strengthens democracy by enabling citizens to analyze data and become more involved with decisions. Over time, we could see open data pave the way to be a policy change agent.


 On April 14 2015, the Open City Policy was passed.
Let's take a closer look at this together.
Before we do, how many of you are aware of this policy?

Achievements 


Canada's Most Open City 2017
Canada's Most Open City 2016
Canada's Most Open City 2015

Open Data, Open Potential


Potential Benefits?

Opening our data should result in a decrease of FOIP requests as well as 311 calls.

Imagine if no person ever had to call 311 again to ask

"When does bus 1 arrive at bust stop 1123?"

311 Cost Reduction

311 averages 5900 incoming calls per day (2015), of which 26.95% are requesting Transit Bus information ("When does my bus arrive?").


Couple this with the average cost of $5.00 per call (2012) and the numbers are staggering!



Further Benefits

  • free analysis of City data can only improve it
    • could lead to improvements in IM / record keeping
  • enhance / improve internal processes
  • Citizens able to inform and engage more actively in municipal decision making (Council Memos in ODP)

Dataset Lifecycle

The Open Data Team has a strict process to ensure all things are considered before anything is made public on the portal.

How data becomes a dataset

  • A request for data is received (externally or internally)
    • via the portal or an email to opendata@edmonton.ca
  • ODT creates open data request (ODR) in POSSE and so begins the workflow


Workflow

A comprehensive workflow exists in POSSE for tracking open data requests.

Details

  • Open Data Team (ODT) evaluates the request by checking against existing requests
  • Open Data Advisory Group (ODAG) delegate is assigned to determine the Data Steward
  • Data Steward and ODT work together to complete required documentation and satisfy requirements
  • ODT builds their ETL process, using FME (including fully scheduled automation)
  • Dataset is built in a private staging area for the Business to review/ODT to tweak as needed
  • Once QA is complete, ODT meets with FOIP/Law for their advisement
    • if changes are needed, ODT and Business Partner work through the details
  • When FOIP/Law review passes, dataset is released publicly as per Business Partner's requirements
    • If desired, a PSA from Corporate Communications will be issued as well as tweets from @OpenDataEdm
  • ODT completes the workflow

Challenges

Prior to City Policy C581, it was often difficult to get the message across. Lately however, we see a shift where City areas are approaching us to host their data. 
Open data is a marathon, not a sprint.

No system? No problem!

Often we work with Business Partners who do not have their data systematized.
Thanks to Google Sheets, we can offer a quick solution that they fully own.

Types of Data

Structured data is easy to collect, organize and use while unstructured data, not so much.
This is important to keep in mind as we source data to make open.


Structured Data

Structured data is data that is easy to collect, organize and use. Typically it is stored in databases, web logs, financial applications etc.
Information is displayed in titled columns and rows which can easily be ordered and processed. 
The data simply exists without having to do much interpretation. 
The structure helps programs store, analyze and search the information.

Unstructured Data

Unstructured data typically refers to flat strings of text, examples of such include emails, text documents (Word docs, PDFs, etc.), social media posts, videos, audio files, and images. 
The major disadvantage of unstructured data is that it requires significant effort to interpret for any meaning. Also it doesn’t fit nicely into databases.
Unstructured data is rowdy and chaotic but if we can manipulate the data, we see that it is also incredibly valuable.

Things to Consider

Usability of Data
The most usable format for data is normally the one in which the dataset was first created.

Think Microsoft Excel, Oracle databases, Google Sheets.

For open data consumption, simple is best. If a .csv will do, why create a database table?

Of course, not all data is as simple as a spreadsheet nor can it all be represented as such. Aside from tabular data, there also exists two more data structures - hierarchical and network. Although these are more challenging to work with, it is only slightly.

Hierarchical
Hierarchically structured data shows the relationships between data points.

An example of hierarchical data is a simple organizational chart or family tree.

Network
Network structured data allows relationships to exist between any combination of data points in any direction.

A great example of a network data structure is any social network - LinkedIn, Facebook etc.



Management of Data
For some organizations, open data is now the main source for their data. A good example of this is an 'open' corporate data warehouse.

Using open data as a way for employees to access internal information has many potential benefits.
It increases knowledge of open data.
It increases sustainability and trust in the data, by making it a key component of the organisation’s operations.
It creates opportunities for innovation.

Access to Data
The format appropriate for each dataset may be different. A document (e.g. PDF) is easy for people to understand but this does not hold true for a machine.

For this reason, accessibility is often achieved by making the same data available in a range of different formats


For open data to be as accessible as possible, it must be in a format that
  • the user can understand
  • machines can read
  • supports easy reuse
  • doesn’t require expensive tools to access.

Geospatial Data

Geospatial data is typically more complex than tabular data. It details things like land parcels, road networks, or rooflines using points, polygons and more.

When publishing this type of data in open data, we typically use Esri shapefiles from GeoMedia.

These are then downloadable from our portal in formats like KML, KMZ, Shapefile, Original and GeoJSON.

Statistics on Use

Our open data portal uses Google Analytics to track usage. Here below are some interesting facts.

2016

155,142 users / 676,671 page views of which 7,248 were City employees

2017 (to Oct 17)

176,333 users / 1,019,007 page views of which 6,359 were City employees 

Election Night 2017

On election night, we saw a record number of concurrent users at 2,939 for the near-real-time election results (datasets and charts).


92,237,658 downloads in 5 years

City of Edmonton went live with Socrata on Aug 23 2011

Storytelling

Data is like paint. It does no one any good if it is sitting in a closed can.
It is only good if it is applied.
So then, how do we apply data?
The first step is contextualizing it.

Perspectives

Imagine not having to contact a web author to put your story out there. 
Imagine if you could do that right here in the open data portal.
A recent enhancement to our portal gives us (you) the ability to do just that - tell stories with and about data!

Sample Stories

"I find the Open Data team to be enthusiastic, flexible, innovative and creative. They are skilled at getting to the root of what's required and providing it succinctly and on time. Their thorough understanding of all facets and platforms available for open data make it easy to build the right presentation to the public as well as generate valuable internal reporting. In a world swamped with data, they have really helped create some clarity."
-Tara (Analyst) / Holly (Manager)  : 311 & Inside Information