An API Framework for Digital Government

We live in the era of COVID-19 and climate change. Now more than ever, it is absolutely essential that governments can communicate with citizens, community groups, and businesses digitally to provide services, inform the public, and engage with communities. Digital services and data have become life-saving resources, that if shared (in real-time and with the right partners and access rights), can aid with decision-making and emergency responsiveness, allow research insights, and foster innovation by businesses and non-profits alike.

The technology exists to make all of this happen. Application programming interfaces (APIs) are a mature, general purpose technology that can be leveraged by governments to create digital connections between systems. APIs can share data in real-time. APIs can allow a digital service, such as filling in a form or confirming your identity, to be reused by all government departments (and industry partners) so that resources are saved and duplication is minimised. APIs can create a pipeline for large, anonymised datasets so that researchers can sift through for clues to public health needs, or so that machine learning and artificial intelligence programs can ingest and automate insights and analysis in real-time. APIs enable emergency alert systems. APIs allow external stakeholders to share data with governments while preserving commercial-in-confidence data. APIs can feed data into dashboards that allow monitoring, analysis and instant responsiveness of population challenges in real-time.

Across governments, these benefits of APIs are recognised. There are hundreds of uses currently in place across Europe, Canada, the U.S., UK, Singapore, Japan, Latin America, Australia and New Zealand that allow systems to connect and share data and service functionality by API. But the adoption of APIs is not yet making a major difference to policy outcomes. APIs are not yet adding value to government’s digital efforts in a measurable way.

The challenge of generating benefits from government APIs

Outside some innovative, isolated examples, APIs have not enabled better data sharing arrangements to respond to today’s wicked problems. APIs have not helped governments address climate crisis challenges. APIs have not allowed better insights into city, regional, national or regional responses to the COVID19 pandemic. APIs are only just beginning to enable better financial management for citizens and businesses who can connect their bank accounts to digital financial products. APIs are not enabling the interoperability of health records across borders so that patients do not have to repeat their symptoms or health histories to each new health professional that is added to their team.

So what is holding APIs back from being a key enabling technology that can help governments manage digital engagement or solve today’s wicked problems? Private industry have asked a similar question. For over ten years, private businesses have moved towards IT infrastructures that are composable and modularised. For business, this means an entire IT system does not need to be taken offline when one component needs to be rebuilt, upgraded, or have new features added. In modernised IT infrastructures, business found that APIs could help them integrate components from external systems or from other internal systems. Individual business units within a larger company or startup would begin using APIs to connect data. Services built with APIs (called ‘shared services’, such as a login component or a payments component), could be reused in multiple products: an iOS app, an Android app, a website, an embeddable form in a newsletter…

Lessons from private industry

Businesses were able to connect API components in multiple ways to speed up product development and onboarding with partners. After these benefits became clearer, other lines of business in the same company would also look for opportunities to use APIs. But that new line of business would also build a login component, a payments component… Quite quickly, one line of business in the company would expose a dataset as an API. Then another line of business would do something similar, but use a different data model that named the datasets in a different way. So complexity began to increase again. Duplicative efforts began to waste resources across the business again.

This same type of complexity and duplication is what is currently happening with APIs in government environments. APIs are seen as a beneficial technology by technologists and developer teams working within one government department. But that government department does not have insight into how their API could be of benefit to another department. They do not realise that another department already built a payments API or a login API. One department needs to communicate with schools, so they create a dataset of all the schools they speak with, and then they build an API that integrates that data with their email newsletter system. Another department also has a list of schools. They also build their own API. Meanwhile the education department has a master list of schools. They are worried about opening it with an API because it has confidential information, such as the principal’s personal phone number in it.

Best practices are available to governments to build high quality APIs. Resources are available and have been adopted by forward thinking governments around the globe. These resources explain how to make APIs in a way that they are secure, only share the right data with the right access permission levels, and apply world-class standards from industry and the maturing API technology sector. But applying these best practices does not reduce the risk of duplication. Applying these best practices does not reduce the complexity of multiple departments building similar APIs within their department. It does not stop a department from creating their own datasets or shared services, and then building an API to connect it instead of looking to see if any other departments have a ready-made API solution.

When private industry faced this problem, they moved to an API-first model. An API-first model steps back and looks at the overall business goal and asks: Can APIs help us solve this problem? This is what is now needed in governments. But moving to such an approach fundamentally alters the way government is organised. Departments must be able to collaborate across silos and share budgets and resources to reuse existing APIs. Data custodians need to manage ‘single source of truth’ datasets and offer APIs to all departments so that one single master education list, for example, is available to all departments (with access permission restrictions in place to stop departmental staff from seeing the principal’s phone number if they don’t need it). Governments must work with industry, researchers, and community groups to identify priority use cases for APIs. New metrics systems are needed that can measure the value that APIs are creating, and whether they are helping achieve the intended policy goals. And government decision-makers and policy developers need to be more opinionated around when to use APIs as an enabling technology.

I recently worked on a European Commission study into the use of APIs for digital government. During this study, government stakeholders were asked what they needed. Overwhelmingly, they asked for best practices to guide their work. But while best practices are readily available for technical implementations, there are less evidence-based examples from government settings that address how to move to an API-first approach across a whole-of-government operation. Some stakeholder interviews and workshops described the challenges that this created for API practitioners.

The challenge when policy avoids technological discussions

The disconnect between policy making and API implementations is a key gap that remains unsolved. This disconnect is apparent from the literature review of best practices and in stakeholder liaison around the globe. Policy makers and government decision makers do not need to know the technical aspects of creating APIs. But, in my opinion, there does need to be more clear signposting that APIs are an enabling technology for policy. I will give two examples from Europe, where I am based: The European Commission Green Deal and the European Data Strategy. Both are high-level policy documents that aim to drive innovation and whole-of-Commission activity in the years ahead to solve key global challenges. At present, neither document mentions APIs specifically as a general purpose, policy enabling technology.

The European Green Deal is a high-level European Commission document that comprehensively outlines key strategies to address four key policy goals, to:

• Become climate-neutral;
• Reduce pollution;
• Encourage innovation in clean technologies; and
• Ensure a just transition to a more sustainable economic society.

This work cannot wait. The Green Deal will need a technological connective tissue that enables participation of a wider range of players - from individual citizens and households, to small and medium businesses, to academic and research institutions, governments, and large enterprises. This technological connective tissue will enable us to share resources, communicate research and experiment findings, collaborate on new ideas around the world, automate menial tasks, and enable data insights. Application Programming Interfaces (APIs) can be that technological connective tissue that enables us to have a global platform for collaborative action. However, the Green Deal does not specifically mention APIs as being the technological enabler. Therefore, for each activity or action, each department responsible for a strategic action area of the Green Deal may use a solution that meets their departmental needs. Other departments may later have difficulties linking into these initiatives because APIs were not available. A department could move to release datasets online. But researchers are unable to plug these datasets into their analysis automatically if APIs aren’t available.

The European Data Strategy aims to support Europe in becoming a leading role model for a society empowered by data to make better decisions – in business and the public sector. The Strategy recognises that APIs will be built for high value datasets identified through the Open data and public sector information Directive. Additional domain-driven data spaces are also identified to encourage ecosystem partnerships around sharing data in areas such as finance, health, the European Green Deal, agriculture, manufacturing and others. The strategy notes challenges in data availability, interoperability, and data infrastructure development. While it is expected that domain-driven data spaces will use APIs to enable interoperability, this is not stated and so higher level policy stakeholders for individual strategic actions may be unclear that APIs can be used as a foundational technology to help solve these challenges.

The API Framework for Digital Government

As part of my research when helping contribute to the writing of the API Framework for Digital Government, I conducted a comprehensive literature review to analyse how governments and private industry are adopting API best practices. This review looked at examples from a whole-of-government strategic and policy level through to an implementation level. Over 340 documents were reviewed. Interviews and workshops were also held with key informants across the European Union and around the globe. Key approaches were distilled from the reports and categorised together into common themes.

From this work, a cohesive, robust API Framework comprised of 12 proposals was created. This API Framework acknowledges that governments are already taking a range of actions in digital government and API implementation that must continue to be supported. But, in my opinion, the Framework approach also seeks to help governments shift to a more cohesive approach in which the risks of duplication and complexity are reduced. The proposed API Framework helps governments reorient towards an API-First approach in much the same way that private industry did so, while also acknowledging that governments have unique roles unlike private industry. For example, I believe governments must ensure equity of access and opportunity for all citizens and businesses. Governments must ensure that data and digital service availability do not create imbalances of power. Governments must ensure that solutions are sustainable and use available resources responsibly.

There is an organisational law of technology known as Conway’s Law. It states that software systems can only ever be designed that will reflect the organising structure of the creator. In government, based on today’s configurations around the world, that means that software will be siloed, that existing budgets and resources will be protected, and that collaboration will be minimal. Therefore, I believe, the success of an API Framework that achieves cohesion, interoperability, and collaboration will require fundamental changes to the organisational structure of governments. Hopefully, the proposed API Framework will encourage such discussions.

The proposed API Framework is made up of 12 proposals, as organised in the diagram, below. These proposals aim to be implemented at a strategic (policy support) level, at a tactical level (such as by a Department where resource allocation decisions are being made), and at an implementation level (when APIs are being operationalised).

At each level, action is needed to ensure APIs align with policy, build effective platforms and ecosystems, involve people, and harness best practice processes:

— Governments, representative bodies, and policy makers need to act at a strategic level. Without the following foundational elements in place, governments will only ever create ad hoc APIs that will eventually generate complexity, reduce interoperability, and reinforce existing siloes. This strategic work can be done before APIs are created, or while current API activities continue. At an API strategic level, governments need to:

1. Align APIs with policy goals.

2. Define the government API platform vision.

3. Create API governance structures.

4. Form guiding principles for API processes.

—   Once there is an understanding of APIs as technological enabler that facilitates the achievement of government policy goals, governments can then better set actionable targets and allocate resources. At an API tactical level, governments need to:

5. Design metrics and prioritise APIs by policy areas.

6. Harmonise data models and other platform and ecosystem assets.

7. Establish cross-competency teams.

8. Follow an API product approach.

—   Finally, with policy alignment and departmental resources allocated, governments can work on the technical and day-to-day operational elements of adopting and managing APIs. At an operational/implementation level, governments need to:

9. Measure policy impacts of APIs.

10.   Build API platform components.

11.   Appoint API product managers.

12.   Adopt an API lifecycle approach.
 

Who should use this Framework

I believe the Framework is useful to 4 key personas:

Policy Lead

What value they can get from the report...

Policy leaders and key government decision-makers need to understand how APIs can be the technological enabler for many of their policies and strategies. While these readers do not need to understand technical aspects of APIs, they do need to consider the broader ramifications of how APIs stimulate platform models in order to be able to guide policy that makes maximum use of the potential of platform approaches. Understanding that APIs enable governments and external parties to share data and functionalities and build products and services together raises a whole range of new policy questions: how should governments facilitate ecosystems, who owns the intellectual property (and data) from these shared platforms, what rights do citizens and local businesses have to the products and services that are built, and so on.

Digital Government Lead

What value they can get from the report...

While individual government organisational structures may vary, there tends to be someone with oversight responsibilities for managing the government’s digital agenda. (This may be shared by multiple people, as well.) The Digital Government Lead will be the key implementer of the API Framework as a whole, and would be expected to know where the government is at in terms of maturing along the Framework model. (Checklists are provided in the Framework to help the Digital Government Lead assess what actions their government could next take.)

Product Manager Lead

What value they can get from the report...

Staff responsible for an entire API activity area or in charge of an API product will gain particular value from the Framework. I believe it will help them make the case for a more coordinated approach to APIs across the whole of government, will help them in thinking through the IT architecture and best practices in managing and measuring the impact of APIs, and will guide their day-to-day management of API products under their management.

API architects and developers

What value they can get from the report...

Governments have already released a number of best practice technical documents on how to design, secure, build and version APIs within government. The best practices for API lifecycle approaches are collected in the Framework and provide developers with key links to best practices. The Framework also gives them a better understanding of the context in which their individual APIs will operate and hopefully encourages them to take on new skills beyond the technical realm, including in design thinking, impact assessment, metrics collection, and collaboration which will be necessary for all developers when working with API product managers to create high value government APIs.