Table of Contents
- Introduction
- What an Algorithm is
- What We Mean by Ethics
- Where the Book Shines
- Where it Could Do Better
- Differential Privacy
- Defining Fairness
- Pareto Frontier
- Quantifying & Injecting Diversity
- GANs for Fairness & Privacy
- Transparency / Interpretability Experiments
- Counterfactual Explanations
- ACM FAccT Papers
Introduction
Micheal Kearns & Aaron Roth’s The Ethical Algorithm is a must-read for anyone working in data, social science, and policy, or simply anyone interested in solving the ethical challenges around algorithms. And asking the hard questions we need to ask to do so. Here’s why I’m buzzing about it:
- Quantifying social values means we can embed them into our algorithms. Tiptoeing around this doesn’t solve social challenges or make them go away, it only obscures them. If we want to reap the rewards of algorithmic decision-making, while still improving society, it’s important to explicitly define and discuss ethical goals, standards, and trade-offs in specific, quantitative ways (e.g., Pareto frontier).
- Embedding ethics into data workflows forces us to be explicit about the impact of our algorithms and how that could differ between groups, decisions we make based on those algorithms, and 2nd and 3rd order consequences of those decisions.
- I believe we’re at an inflection point. Companies that succeed and generate higher trade multiples in the future will over-index on algorithmic ethics. Consumers, shareholders, voting publics, and regulatory bodies demand it. Corporate leaders who don’t embrace it will miss a significant medium-term, demand-led revenue opportunity and a chance to get in front of regulation.
In this post, I’ll discuss where the book shines, where it could do better, and highlight a few of the book’s key concepts, including differential privacy, defining fairness, Pareto frontier, a slider quantifying and injecting diversity into Facebook, generative adversarial networks (GANs) for fairness and privacy, transparency / interpretability experiments, and counterfactual explanations.
This just scratches the surface. The book contains visuals, quantitative examples, and a lot more detail on those topics and on reproducibility / p-hacking, game theory, 2-sided markets, and the singularity (read: read the book!). Before we dive in, it’s important to mention the fundamental premises of what an algorithm is and what we mean by ethics.
What an Algorithm Is
An algorithm is, at its heart, an optimization procedure. It optimizes on an objective function (e.g., predictive accuracy) and it does so at all costs. The problem is not that algorithms don’t do what we want them to do, but that they do exactly what we tell them to do. We shouldn’t find it surprising when an algorithm produces a model that has a different false negative or false positive rate for different demographic groups. When maximizing accuracy across multiple demographic groups, an algorithm will naturally optimize better for the majority population at the expense of the minority population because, by definition, there are more people from the majority group so they contribute more to overall accuracy.
Optimization procedures won’t satisfy constraints that weren’t explicitly encoded in their design. We shouldn’t expect to get for free anything we didn’t explicitly state in the objective. And we shouldn’t expect algorithms to avoid any behavior that we didn’t specify should be explicitly avoided. Machine learning (ML) requires that we be explicit about everything we want and don’t want. In this thinking, many of the daily algorithmic bias headlines are examples of optimization gone awry.
Additionally, while algorithms, like hammers, are human-designed artifacts, they’re also fundamentally different. A hammer is designed to do one thing – hit a nail. But, if you use it to maliciously hit someone’s hand, that’s unethical. On the other hand, moral character can’t always be ascribed to the designer of the algorithm. In ML, the model that results (e.g., a loan default classifier) isn’t explicitly written by a programmer. Instead, it’s solved for automatically by optimizing a specified objective function, which can lead to unanticipated behavior with ethical consequences. With that in mind, Kearns & Roth don’t view the problem in terms of inputs or objective functions, but constraints. Instead of purely optimizing, how can we optimize given privacy, fairness, and transparency constraints?
What We Mean by Ethics
The book includes issues of privacy, fairness, transparency / interpretability, accountability, and morality. Algorithmic privacy and fairness are the most mature areas with their own literature, theories, and experimental methodologies. Privacy is on the firmest scientific foundation and farthest along. It also has general agreement on definitions that are intuitive, precise, and manageable and newer solutions, like differential privacy, are already deployed for iPhone app usage and the 2020 U.S. Census.
Fairness is a tougher nut to crack. Unlike privacy, it has less agreement on definitions and less understanding of trade-offs between different demographic groups. For example, by enforcing racial fairness, there’s no guarantee that we don’t unintentionally magnify gender discrimination. Still, there are concrete proposals and methodologies around fairness, even if they are simplifications, that lay the foundation for theory and practice.
Transparency / interpretability is even farther behind and is difficult to formalize in a quantitative way. Interpretable to whom? Interpretability varies by math / computation literacy of the observer. A layperson with no quantitative education may not consider a model interpretable. Someone familiar with classical statistical modeling might understand a linear model, but not a multilayer neural network because feature correlations cannot be easily extracted. A deep learning expert will consider a deep neural network interpretable and that higher order correlations explain the model’s behavior. Additionally, what do we want to be interpretable – the data, algorithm, model, or decision made? Simple algorithms and datasets can still lead to complex output with opaque relationships between variables. Fortunately, research has put forth promising solutions, like counterfactual explanations.
Accountability and morality are farthest behind. There are some norms for which we may not want to use algorithms at all. For example, should algorithms be used in automated warfare to make an unassisted decision to kill a human being, even if they do make fair, more accurate decisions than humans? The thinking here is that such a decision should only be made by someone who understands the moral agency, responsibility, and consequences involved.
Where the Book Shines
While previous books like Weapons of Math Destruction and daily headlines on algorithmic bias are important in drawing spotlight on the serious societal effects of algorithms making consequential personal decisions on recidivism, prison sentencing, hiring & firing, college admissions, loan disbursement, healthcare allocation, and immigration / deportation at population-level scale, they provide little in the way of solutions. When they do, they focus more on legislation, regulation, and watchdog oversight, which are necessary, but not sufficient, to solve such problems. Institutional solutions like these don’t scale and can’t keep pace with the velocity of algorithmic decision-making. Moreover, as we’ve seen, well-intended regulations like barring use of race as a variable, rather than requiring unbiased output, actually end up having the opposite effect of intensifying bias.
The Ethical Algorithm, on the other hand, focuses on technical solutions and how to implement them in practice with quantitative examples. Kearns & Roth are computer scientists at UPENN specializing in the design and core principles of algorithms. They propose that we literally embed social values, including privacy, fairness, and transparency / interpretability, directly into the Python code of algorithms.
The authors go about this in a systematic way – thinking about the social properties we want algorithms to have, how we could define them, how they can be implemented in algorithms, and assessing trade-offs to accuracy. On privacy, their answer is to use differential privacy instead of today’s more widely used standard of anonymization, which is deeply flawed (see below). Fascinatingly, the mechanism of differential privacy – adding noise to individual observations – has wider applications beyond privacy (e.g., Maxwell 2.0 equilibrium). On fairness, they propose optimizing on an objective function, but with a fairness constraint. How we define fairness in those constraints is up to us humans. The book details several ways we could define fairness and how they could be applied in different situations.
Kearns & Roth also delineate where the work of algorithms and data scientists ends and where that of humans begins. They’re clear upfront that they’re scientists, not policymakers or stakeholders. Algorithmic ethics can only quantify the trade-off between fairness and accuracy and nothing more. The book does a great job in visualizing this interface as a Pareto frontier or a fairness vs. accuracy curve (see below). It’s a range of options that could be thought of as a knob that we as a society can turn to different levels depending on the stakes involved.
In this sense, algorithmic ethics delivers just the starting point for dialogue with policymakers and society. It’s entirely silent on the right definitions, groups, and model selection. These decisions are up to stakeholders and society to make. At the end, it will not be algorithms, but humans that make the most important decisions of which groups merit protection, which definition of fairness is best, and which model on the Pareto frontier is best.
Where It Could Do Better
A few areas where I thought the book could do better:
- It glosses over the idea of “fairness gerrymandering” and balancing fairness for multiple groups. It would be great to see more detail on how we could think about and address this. I came away from this chapter with the concern that if ensuring fairness for multiple groups is “mathematically impossible” then how could fairness be implemented at all?
- Quantifying trade-offs between fairness and accuracy can feel uncomfortable. While Kearns & Roth briefly touch on this, the book could go further in alleviating the general public’s concerns about adopting algorithmic solutions to fairness.
- Applying fairness constraints on an algorithm means real monetary costs to companies in terms of accuracy trade-offs or the need to collect more data. It would be great to see examples, even if hypothetical, around costs as well as potential monetary opportunities.
- It doesn’t address talent in data. It could highlight the diversity shortage in data and leadership, quantitative talent shortage in regulatory bodies and government, and the need to include arts and humanities in tackling challenges. This could do a great deal in driving algorithmic ethics higher up on the agenda and developing better solutions.
I’m excited for the next edition to see if the authors address some of these issues and to see real-life implementations of their ideas.
Differential Privacy
Of the ethical dilemmas, privacy has progressed the most. But, today’s prevailing mode of privacy is still k-anonymity / anonymization, or removing personal identifiable information (PII) through redaction and coarsening. Unfortunately, anonymization is conceptually and technically flawed. Even if you anonymize someone’s data, say a medical record, someone can do a join of multiple anonymized medical records to re-identify that person (linkage analysis). Anonymization’s fundamental, unfixable flaw is that it pretends that the dataset in front of you is the only dataset in the world and the only dataset that will ever exist in the future. Moreover, it’s been shown that even your seemingly innocuous “digital exhaust” (e.g., Facebook likes) can be used to infer sensitive information about you. These sorts of privacy collapses have been demonstrated with, for example, genome-wide association studies (GWAS) and the Netflix Prize.
Differential privacy is a more effective framework. It was first introduced in 2005 and is already in use by Apple, Google, and the U.S. Census. The goal of differential privacy: nothing about an individual should be learnable from a dataset that cannot be learned from the same dataset with that individual’s data removed. In doing so, it protects you against arbitrary harms. Differential privacy is grounded in a counterfactual thought experiment. Consider a study causally linking smoking to lung cancer. Let’s say there are two imaginary world’s – one in which you’re one of n participants, and one in which you’re not a participant (so n-1 participants). If the study finds that smoking causes lung cancer, that result would have occurred whether your data was included or not. That smoking causes lung cancer is a fundamental fact about the world. Removing your individual data record doesn’t change the probability of that outcome by much. In a Bayesian framework: an observer’s posterior belief, no matter what their initial belief was, is close to what it would have been had they observed the output without your data.
The mechanism for differential privacy is a randomized algorithm where noise (zero-mean) is added to each observation. This yields relatively accurate aggregate data, but makes it impossible to reverse-engineer anyone’s individual data. The privacy-accuracy trade-off can be visualized as a knob. You can increase noise to each observation, but this reduces aggregate accuracy, and vice versa. Where we set the knob depends on what’s at stake – if we’re dealing with very sensitive information like medical records or census data, we can set the knob high on noise. But, if we’re working with lower stakes data like app usage, we can turn the knob lower. Additionally, any statistical analysis and even ML algorithms like back-propagation can be made differentially private (albeit with the need for more data). Differential privacy doesn’t protect against data that is networked, for example, the GEDMatch DNA genealogy database used to catch the Golden State Killer in 2018 (as one person’s DNA is tied to family members) or Strava fitness data that was used to expose secret U.S. military base locations. For these, we’ll need game theory approaches.
Defining Fairness
It’s tempting to think that we can solve fairness problems by restricting the types of information we feed to models. However, simply restricting inputs or features like race or gender doesn’t work nowadays as sensitive information like race, gender, sexuality can be inferred from seemingly irrelevant information like purchases, Facebook likes, political party, income, zip code, movie preferences, check-ins, etc. Moreover, restricting reference to race and gender inputs doesn’t guarantee that the model is unbiased.
A more effective way to define fairness is through outputs. Consider the example of predicting creditworthiness among two groups – circles and squares:
- Statistical parity: fraction of circles and squares granted loans is the same (equality of output). This definition is weak because it ignores features (x). It’s blind to the properties of individuals so it can be satisfied with random lending. It also doesn’t specify the goal of the prediction, but is simply a constraint on the prediction itself. While statistical parity wouldn’t be useful for lending, it could be useful if we purely want to distribute an opportunity and don’t have a specific notion of merit (e.g., free tickets to a show).
- Equality of false negatives: false rejection (false negatives) rate is the same for circles and squares. This evenly distributes “mistakes,” or harm, rather than outcomes. Harm comes to those who are creditworthy, but are denied loans. This constraint requires that a random creditworthy circle and square have the same probability of being harmed. In a realistic framing, your race doesn’t affect the probability that the algorithm will harm you.
- Equality of false positives: in the case of tax audits, the false audit (false positive) rate is the same for circles and squares (evenly distributing harm). This constraint requires that a random tax-compliant circle and square have the same probability of being audited.
- Equality of positive predictive value (PPV): among people the algorithm deems creditworthy, the repayment rate across circles and squares is roughly the same. Without this, the ultimate human decision-makers will have a rational incentive to let race influence decisions because doing so will make the bank more money.
We can’t achieve all of these definitions of fairness simultaneously, meaning even at the starting point of defining fairness, we face trade-offs. Keep in mind that our definitions of fairness protect groups, not specific individuals in those groups.
it will be up to stakeholders and society, not algorithms, to decide which groups to protect, whether they are based on gender, race, age, disability status, wealth, nationality, sexual orientation, etc. As we know, algorithms are good at doing what we ask and don’t ask them to do and nothing else. If we ask them for accuracy and not fairness, we won’t get fairness. If we ask them for fairness for one group, but not another, that’s exactly what it will produce. We need to specify everything we want and don’t want.
Pareto Frontier
Once we choose a definition of fairness and the groups meriting protection, we can derive a Pareto frontier, which quantifies “good” solutions to any optimization with multiple competing criteria. This would be akin to portfolio management’s efficient frontier, quantifying the trade-off between returns and volatility. In this case, it would be a reasonable set of choices for trade-offs between fairness and accuracy. To construct the Pareto frontier, we can create a numerical objective, like a penalty, that’s a weighted combination of the error and unfairness score and plot weights. We end up with a plot with error on the x-axis and unfairness on the y-axis.
While the Pareto frontier provides a set of optimal options, it’s wholly silent on which point to choose. That’s up to us as a society to judge.
The curve is a knob or interface between the algorithm and society. And, similar to differential privacy, the stakes matter. We, as a society, might turn the knob to a higher level of fairness for policing and lending than we would for ad targeting. At the end, society will decide on the definition of fairness, groups to protect, and which model on the Pareto frontier fits best.
Quantifying & Injecting Diversity
Game theory moves beyond the above examples to more complex interactions that can lead to a suboptimal collective equilibrium, as in Facebook, Google Maps, and Waze. Facebook’s collaborative filtering algorithm produces a selfish equilibrium (Nash). Specifically, it produces an echo chamber equilibrium, which we all pretty much agree is a bad equilibrium. One interesting solution is to increase diversity and exploration in recommendations / feeds. On Facebook, this would mean deliberately exposing people to articles and products outside of their comfort zone. Mechanically, this could be a slider that can be adjusted by the user that algorithmically adjusts K-cluster distance.
GANs for Fairness & Privacy
Over the past few years, self-play algorithms like GANs have advanced and drawn media spotlight with applications like TD-Gammon and AlphaGo. An interesting GAN application is one designed for non-game goals like fairness and privacy. In an area like fairness in predicting creditworthiness, a Learner whose goal is to maximize accuracy (minimize error) would play against a Regulator whose goal is maximizing fairness and would continually confront the Learner with new subgroups suffering discrimination (fairness gerrymandering). Similarly, for differential privacy, a GAN can be used to generate hyper-realistic synthetic medical records for research purposes. A Generator whose goal is to keep a synthetic dataset as close to a real dataset as possible plays against a differentially private Discriminator that points out differences.
Transparency / Interpretability Experiments
We highlighted the challenges in interpretability based on an observer’s numeric literacy and unclarity on what to make interpretable (e.g., data, algorithm, model, decision). One approach to formalize interpretability is to run behavioral experiments on different groups testing understanding of models of varying complexity. We could then develop an observer-specific definition of interpretability and study the design and limitation of algorithms obeying those definitions.
Counterfactual Explanations
All decision-making, including the human kind, is algorithmic. The only difference is that human decision-making is based on a logic we can’t quantitatively elaborate yet. So the real choice is not whether or not to use algorithms, but whether or not to use precisely specified algorithms. Humans are great at coming up with self-reported post-hoc rationalizations rather than real insights into their decisions.
Reasoning about and exploring counterfactuals is a crucial benefit of ML algorithms. For example, if an algorithm denied you credit, it can explain what is the smallest change in your application that could have changed the decision to accepted, like being in your current job an extra 6 months or having an additional $10k in savings.
ACM FAccT Papers
If you’d like to explore more, the ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), previously called FAT ML, includes a ton of scientific papers by year. Here are links: 2020, 2019, 2018, 2017, 2016, 2015, 2014.
What do you think? What are the promises and pitfalls of this approach? Is there something we missed?
Thanks for reading and sharing,
Rish
