Designing digital services that are accountable, understood, and trusted (OSCON 2016 talk)

These are the speaker notes and slides from my talk at OSCON 2016 last month.

Power, design and possibilities

Hello.

Welcome to this session about power and importance of designing digital services that are understandable, accountable and trusted.

I’m hoping to convince you that designing digital services that are understandable, accountable and trusted is now a commercial as well as a moral imperative, and that building an open society in the digital age is about more than open code.

And that, whether you like it or not, if you work in software or design in 2016 you also work in politics.

Drawing of Whitehall

My name is Richard Pope. Quick bit about me.

I worked for the Government Digital Service for 5 years, part of the team that delivered GOV.UK. I then went on to work with teams across different government departments and policy areas.

Before that, I setup the labs team at Consumer Focus, the UK’s statutory consumer rights organisation, building tools to empower consumers. I've worked at various commercial startups including moo.com and ScraperWiki.

I co-founded the Rewired State series of hackdays back in 2007 that aimed to get developers and designers interested in making government better.

'Government as a platform' diagram

The last piece of work I did in government was on a conceptual framework for the idea of Government as a Platform. Government as a Platform is the idea of treating government like a software stack to make it possible to build well-designed services for people.

The work involved sketching some ideas out in code, not to try and solve them upfront, but to try and identify where some of the the hard design problems were going to be.

Mock-up of app to help people move house

Things like: what might be required to enable an end-to-end commercial service for buying a house?

Mock-up of Borchester City Council website

Or what would it take for local authorities to be able to quickly spin-up a new service for providing parking permits?

'Government as a platform' diagram

With this kind of thinking, you rapidly get into questions of power.

What should the structure of government be?

Should there be a minister responsible for online payment? Secretary of State for open standards?

What does it do to people’s understanding of their government?

How and where should you build in opportunities for recourse for when things go wrong (which does happen from time-to-time).

This is the Arthur Dent problem. In Hichhickers guide to the When he has just worked up to the surprise that the council are going to bulldoze his house:

And that, at the heart of it, is the subject of this talk.

But the plans were on display

How do we build stuff that people can understand and trust, and is accountable when things go wrong?
How do we design for recourse?

I’m hoping to convince you that, whether you like it or not, you also work in politics. That software is politics.

Power diagram from Anthony Sampson book

This image is from a book called ‘The New Anatomy of Britain’ the journalist Anthony Sampson.

He published a book approximately every 10 years from the 60’s on the subject of power in the UK. Each book included a diagram - this is the one from 1962.

The size of the bubble denotes how much power he thought that group had. In this one, the Aristocracy' is still significant enough for a mention.

Power diagram from Anthony Sampson book

This is the one from the early 80’s. The Civil Service looms large (This was the age of 'Yes Minister'. Trade Unions and nationalised industries also feature.

Power diagram from Anthony Sampson book

In 2004 the centre of government, in the form of the prime minister is shown as a lot more powerful. As is what we’d now call 'traditional media’ (Fleet Street / TV). Despite being the age of 'Web 2.0', digital doesn't get a singled out.

I think these diagrams are useful because I think they demonstrate a couple of things really clearly.

Firstly, they reduce things down to an abstract landscape of power. There is only power - be it commercial or political, accountable or unaccountable.

Secondly, they show that power is mutable: it changes over time, and it can be change, and that can happen quite rapidly too (as the changing power of unions illustrates).

Politics is about the distribution of power in society

But, fundamentally, they show this: that politics is about the distribution of power in society.

In the second decade of the 21st century, digital services - code and design - are changing how power is distributed.

To illustrate how, I’m to going to give a couple of quick examples, from different sectors:

Ars Technica article about VW emissions scandal

Society’s ability to regulate industries effectively is limited by its ability to access and understand code, as we saw with the VW emissions scandal.

Carnegie Mellon University report on targeting of online job adverts

Research from Carnegie Mellon University found that men were far more likely to see Google adverts for high paying executive positions than women.

Citizens Advice report on quality of online job adverts

The decisions developers make about how they model adverts on job publishing platforms have an effect on people's ability to find work.

City A.M. article on Deliveroo protests

Debates about workers rights are increasingly debates about code. The employee tracking and conditions that are coded into the platforms of Deliveroo and Uber are changing what it means to be employed.

Bloomberg article on 'Flash Crash' of the pound

The other day, there was the suggestion that the slump in the pound was the result of algorithms feeding on news stories from social media. (Possibly some of these could have been generated by bots. It's bots all the way down!)

BBC News article on Facebook new bias

Facebook’s dilemmas around news algorithms are pretty well documented. The decisions they are faced with about censorship and quasi-regulation are the things that, historically, only nation states have had to deal with (but ofcourse Facebook lacks the opportunities for recourse that nation states have in place - at least in the western tradition).

Screen-shot of Google Now requesting access to a Nest thermostat

Connected devices are redefining what privacy - a fundamental human right - means. And we are, in turn, being asked to trust opaque machine learning systems with that data. This is a screen-grab of Google Now requesting access to a Nest thermostat.

2 news articles about Uber's legal battles

Uber is winning the battle for the future of public transport through a combination of legal brute force and amazing service design.
But, long-term, could we see them replacing democratically accountable civic transport networks?

Labour party election leaflet promising new 'Hopper ticket'

And that is not the only area the ability of our elected officials to make transport policy is being impaired.

Back in May, there was an election in London for a new Mayor.

One of the flagship policies of the Labour candidate was a 'hopper ticket’ - bus users would be able to change busses as many times as they liekd with an hour.

This is a big deal for people on low incomes on the outskirts of London. Clear policy to try and address that.

The policy actually launched a couple of weeks ago, but in a slightly in the different form.

Just after the election, Transport for London released a statement saying that, in the short-term, they could only implement the policy for up to two bus rides.

Reading between the lines, it sounds like TfL couldn't fully deliver on an electoral promise because they couldn't change the code, and the candidate and the press couldn't know this during the election.

I imagine that was an interesting first meeting.

Government policy is increasingly expressed in code like this.

Software is politics because software is power

So, if politics is about this distribution of power in society, software is now politics.

The decisions of designers and developers are a political force of their own. So this is what I mean when I say I think you work in politics.

And we are asking users to trust us with more data, to allow code to make more decisions for them all the time.

Power diagram featuring large representation of 'digital'

You are in this bubble!

Now, it’s possible to see all this in a negative light. Run for the hills and disconnect! Reality or nothing! But I think we have reached a really interesting point from a design point of view.

And it’s this:

It’s time to stop designing digital services to be easy to use and to start designing them to be understandable,accountable and trusted and easy to use.

If you want your users to trust you with more data, and make more decisions on their behalf.

If you want users to start trusting their data in your machine learning system.

Or if you want users to trust your device in their house.

This is the fundamental design decision facing people building digital services right now.

It’s time to stop designing digital services to be easy to use and to start designing them to be understandable, accountable and trusted and easy to use.

'It just works' is not good enough anymore

Or to put it another way: ‘It just works’ is not good enough anymore.

It's time for some new design patterns

So, we need to figure out how to wire these attributes - understandable, accountable, trusted, into the services and institutions of the digital age.

It’s time for some new design patterns.

Incidentally, we’ve been here before: the manufacturing and industrial revolutions led to new institutions and practices that took the work of entrepreneurs and put it to work for society.

Accountability at the point of use, Expose the rules, Make permissions understood, Digital tools for digital consumer rights

There are four areas I think deserve our attention:

1. Accountability at the point of use

Can we make accountability and transparency part of the design of services?

Mock-up of UK government service for starting a company

This is a hypothetical service for starting and managing a company. Now, obviously the service should be designed around user needs, and users should not have to understand the structure of government to use it.

But that is not the same as obfuscating how government works.

Mock-up showing meta-data about the service

So, what if you could understand why that service is the way it is directly from the service? For example which government ministers are responsible for its safe running?

Mock-up showing performance of service

How the service is performing?

Mock-up showing legislation that underpins the service

What is the underlying legislation for the service? Why does it exist and how can it be changed?

Mock-up showing 'register data' behind the service

What definitive data sources does it use?

Mock-up showing meta-data about the service

This is a government example, but what would a commercial equivalent be?

Maybe making it clear exactly how much was paid to an uber driver and if it meets living wage levels, directly on the email receipt?

Or making it possible to understand supply chains and environmental impact when buying a product from Amazon?

Or display government food safety inspection data next to a delivery order from Deliveroo?

Could they do all this in the app, at the point of use?

In fact, Google has started showing some work in this direction with Google News explicitly marking fact-checked articles.

Photograph of an Amazon Echo

This is only going to get more important as superficially good design abstracts how things actually work further away from the user.

So maybe you should be able to ask Alexa “did the people who assembled you have the right to paid parental leave?”

2. Expose the rules

As we saw with the bus-fare example earlier, more government and non-government services are expressed in code. The code is the definitive article and wanting to change policy requires an understanding of how the code works.

So, what mechanisms do we have to understand how the code works?

One way is to obviously examine the source code directly.

Screen-shot of the Digital by Default Service standard (highlighting point 8)

Luckily the UK government increasingly does open its code.

I was part of the team that drafted this a few years ago - it's the Digital By Default Service Standard, and it includes a requirement to open up the source code of new projects.

Other governments have since made similar commitments.

Now, obviously not everyone can read code - although journalists and consumer rights organisations probably do need to get better at it - and there are many circumstances where organisations will not want to release their source code.

Screen-shot of a Jenkins dashboard

Maybe we can look at other tools of the software development tool-chain to help expose the rules.

If services published their tests, would it help people understand how they work? After all, Gherkin syntax is designed to be understandable by non-coders.

Here are a couple of examples that explain the rules about free prescriptions in the UK:

Given a user who is 60 or older, When they collect a prescription, Then it should be free

Given a user who is pregnant, And has a valid, MedEx certificate, When they collect a prescription, Then it should be free

Given a user who receives, Universal Credit, When they collect a prescription, Then it should be free

Could the publishing of software tests help politicians make better promises?

Photograph of Siri running on an Apple Watch

As software agents of one sort or another (bots, digital assistants, news feed algorithms) start to make more and more decisions for us could publishing of software tests be useful for making bots more transparent?

You can’t ‘view source’ on Siri or Google Now.

WHEN a user is outside, AND more than 1 km from home, THEN display nearby bus stops

So, might you get things like this.

Screen-shot of Habitat

A couple of years ago, I built a proof of concept bot called Habitat to test this idea — it used Gherkin tests as the user interface.

It worked surprisingly well, at least for exposing simple, deterministic decisions.

But tests are only one way of exposing how code works. We could also see the emergence of software deposits organisations for holding private code and that consumer rights organisations or government inspectorates have the right to audit. Some of this already happens in the gambling industry.

For bots using machine learning, the public publishing of training sets will probably also become important.

3. Permission understood

The next area is 'permissions'.

Take a minute to think about permissions. You might think that this is already solved,

Probably the most important research question in digital product design today.

We are now pretty used to apps asking for permission to use our cameras or access our location. But we are not yet used to the idea of different services exchanging data about us, beyond maybe email address.

I’d like to show you an example of how that could work in a government context.

Government as a Platform permissions demo

That is not how government works at the moment. And that is only a prototype.

But we are going to have to figure out design patterns for exchanging new types of data in a way people understand.

The other reason this is a hard problem: it is also a moving target because of more data and more devices. We are struggling today with a few devices and data points.

“everything more durable than a bar of soap and with a retail value of over $5 probably has as much computing power as your laptop today”  - Charlie Stross

I think this quote from Charlie Stross sums up the problem we face.

Screen-shot of Yahoo homepage from 1997

And I think there is a parallel here to how the tools we needed to organise the web had to change over time. Back in the late 90's, a hierarchical list of categories was a perfectly acceptable way to organise the web for people, but as it got bigger we needed better and better search; personalised search; machine learning assisted search.

When it comes to permission systems, we are currently the equivalent of the Yahoo homepage.

Mock-up of UK Government transparency log

The argument here is not just transparency.

Services providing an access history to users, should become an accepted standard because putting users in control of their data is the best protection organisations have against fraud and that users have against misuse.

This is a mockup of a government transparency log, but something like it could equally apply to a commercial service.

Again, I'm not sure we have a clue what design patterns will work here - especially as the amount of data expands.

Screen-shot of the Projects By If design catalogue

There is no single answer and this subject needs serious investment by governments and tech sector.

I think Facebook have gone on record to say they want data sharing to become more of a design problem and less of a policy problem, which is encouraging.

4. Digital tools for digital consumer rights

Finally, for users to really trust stuff in the digital world, we need trusted 3rd parties to do some of the hard work for them.

And this means giving some elbow room to some new digital watchdogs.

We are all familiar with the idea that new technology resulting in regulatory institutions.

Photographs of the covers of 'Unsafe at any speed' and 'Which? Magazine'

It took this book in the US to change the law around car safety.

In the UK, The Consumers Association was set up to test the products of the manufacturing revolution.

What will the watchdogs of the digital age look like?

Can the tools that we use to develop software become the tools of consumer watchdogs?

GIVEN a company with license R409, WHEN it abstracts > 100L in a 30 day period, THEN there has been a breach of license R409

Environmental campaign groups might start automatically checking open government data for breaches of regulations.

Screen-shots of openregister.org

We could see third parties actively verifying datasets and checking facts.

The approach the GDS is taking to open registers is intended to enable this - they using signed Merkle tree’s so others can verify the integrity of a government register.

So a legal watchdog could independently and automatically verify the integrity of the Land Registry.

Or an app on your phone might verify the food safety rating of the takeaway you just walked in to.

Screen-shot of a job advert validator

Data validators become advocacy tools when they are checking for compliance with data standards that have an effect on equality in society.

This is a validator for the jobs advert standard that the government has adopted. Try it out on your company.

Screen-shots of 'Open Diversity Data' and 'Two Factor Auth'

Single purpose monitoring services like twofactorauth.org and OpenDiversityData seems to be another emerging pattern.

Debian reproducible builds

The work going on around software supply chains is also really interesting.

The effort towards reproducible builds of Debian for example - being able to say exactly what set of code is running on a device.

Could using the code scanning tools currently used by companies to identify open source licence restrictions in their codebase could be repurposed to identify if vulnerabilities?

This is going to be increasingly important in safety critical systems - you want to know if the system operating the breaks on your car is the one it is supposed to be. There is an excellent talk from Fosdem on Safety Critical FOSS - Jeremiah Foster, I recommend looking it up.

In a world with one taxi company and one shop, do we need a global consumer rights organisation?

We need to ask ourselves this question. In a world of Amazon, Facebook and Uber - do we need a global consumer rights org? Who is going to explain all this to users?

Accountability at the point of use, Expose the rules, Make permissions understood, Digital tools for digital consumer rights

So, these are some important areas we need some new design patterns for.

Power diagram featuring large representation of 'digital'

And if you work in the digital industry, you are in this bubble.

Like it or not, you work in politics.

I could go on about ultimate power corrupting and all that. And the moral angle is very important.

Photograph of a crowd

But questions of accountability, understanding and trust are only going to become louder.

These issues are only going to get harder to solve as we ask users for more data and to trust code to make decisions for them.

The organisations that understand this and start thinking about how to make services that are accountable, understandable and trusted will have the advantage.

Thank you