The Black Box Society: The Secret Algorithms That Control Money and Information

by Frank Pasquale

Overview

Every day, corporations are connecting the dots about our personal behavior—silently scrutinizing clues left behind by our work habits and Internet use. The data compiled and portraits created are incredibly detailed, to the point of being invasive. But who connects the dots about what firms are doing with this information? The Black Box Society argues that we all need to be able to do so—and to set limits on how big data affects our lives.

Hidden algorithms can make (or ruin) reputations, decide the destiny of entrepreneurs, or even devastate an entire economy. Shrouded in secrecy and complexity, decisions at major Silicon Valley and Wall Street firms were long assumed to be neutral and technical. But leaks, whistleblowers, and legal disputes have shed new light on automated judgment. Self-serving and reckless behavior is surprisingly common, and easy to hide in code protected by legal and real secrecy. Even after billions of dollars of fines have been levied, underfunded regulators may have only scratched the surface of this troubling behavior.

Frank Pasquale exposes how powerful interests abuse secrecy for profit and explains ways to rein them in. Demanding transparency is only the first step. An intelligible society would assure that key decisions of its most important firms are fair, nondiscriminatory, and open to criticism. Silicon Valley and Wall Street need to accept as much accountability as they impose on others.

Product Details

ISBN-13:	9780674970847
Publisher:	Harvard University Press
Publication date:	08/29/2016
Pages:	320
Product dimensions:	6.00(w) x 9.20(h) x 0.90(d)

About the Author

Frank Pasquale is Professor of Law at Cornell Law School and Cornell Tech and the author of The Black Box Society: The Secret Algorithms That Control Money and Information. His work has appeared in The Atlantic, the New York Times, the Los Angeles Times, the Guardian, and other outlets.

Table of Contents

1 Introduction-The Need to Know 1

2 Digital Reputation in an Era of Runaway Data 19

3 The Hidden Logics of Search 59

4 Finance's Algorithms: The Emperor's New Codes 101

5 Watching (and Improving) the Watchers 140

6 Toward an Intelligible Society 189

Notes 221

Acknowledgments 305

Index 307

Editorial Reviews

A timely and important book about the algorithmic processes that play such central roles in our emerging information society. Pasquale explores the abuses that have resulted from insufficient transparency and exposes the inability of either markets or regulators to instill appropriate levels of accountability. He is not a reflexive technology-basher, however, but instead offers judicious reform proposals.

Julie E. Cohen

The algorithmic control that law scholar Frank Pasquale eloquently and intelligently details and analyzes goes beyond money information and into almost every aspect of our lives. For this reason, although it might appear merely to be a book about technology and finance, The Black Box Society, ultimately, is a radical and political work that deserves wide attention… The Black Box Society includes, for example, a fine explanation of the way that corporate and government surveillance work in concert and why we should be concerned about both… [Pasquale’s] brutal on the subject of the NSA, but devastating in his critique of Facebook, Twitter and Google and the myths that continue to surround them: myths of neutrality, myth about the ephemeral nature of their power and more. His analysis of search is pointed and poignant, underlining that we need to understand it better and treat search results more critically and sceptically… Pasquale’s detailed analyses, and his recipes not just for transparency but also for accountability, for more rigour in regulation and harder-hitting enforcement, deserve a careful read—and then action.

Times Higher Education - Paul Bernal

Required reading in many law school and computer science courses… Details how secret databases and little-known applications of AI algorithms have had harmful effects on finance, business, education, and politics. These opaque practices violate personal privacy, lower credit ratings, and make unfair or biased decisions on parole, mortgage, and job applications.

Issues in Science and Technology - Ben Shneiderman

An exhilarating read, brimming with passion. Pasquale’s bold and ambitious book lifts the lid on the ‘black box society’ by tackling a wide array of issues, from secrecy in finance to credit scoring, from search engines to automated decision-making, from institutional transparency to the relationship between government and big corporations. Writing with urgency and utter conviction, he paints a compelling—and devastating—picture of the world that we are building.

Daniel J. Solove

Frank Pasquale’s new book on the secret algorithms that motor the monstrous heart of big data is a timely work of non-fiction, a ‘true conspiracy’ about regulatory weakness in the face of technological hubris and greed.

Flavorwire - Jonathon Sturgeon

Frank Pasquale’s notable new book, The Black Box Society, tries to come to grips with the dangers of ‘runaway data’ and ‘black box algorithms’ more comprehensively than any other book to date… It’s an important read for anyone who is interested in the hidden pitfalls of ‘big data’ and who wants to understand just how quantified our lives have become without our knowledge.

Slate - David Auerbach

Everyone who uses the Internet for entertainment, education, news or commerce is implicated in a web of data collection whose breadth surpasses ordinary awareness… As [Pasquale’s] exposé…shows, this is a society in which basic functions are performed in deliberate obscurity through the collection and algorithmic manipulation of personal data… In The Black Box Society, Pasquale finds reason to believe that even some of the most secretive and unresponsive institutions can be held to account. Elucidating the problem is a first step.

Nature - Steven Aftergood

The Black Box Society is a frightening portrait of the ever more powerful shadowy world that blocks light from reaching our everyday lives. It is also a call to action, with a range of suggestions that inevitably pale in comparison to the gargantuan task at hand. But small steps sometimes have outsize consequences. Just ask the folks who control what we see, influence what we buy, and determine whether we can pay for it.

Reading the Markets - Brenda Jubin

If you are a person in America, then there are equations trying to learn more about you… Some of these equations work for private companies and some of them work for the government, but they all generate correlations based on your behavior… Frank Pasquale’s new book The Black Box Society is a tour of how computational intelligence has come to dominate three important parts of American life: reputation, search, and finance.

New Republic - Malcolm Harris

Recognized internationally as a landmark work of law and political economy.

Lawrence Joseph

The Black Box Society offers a good dose of fresh thinking and may advance our debates over privacy. It also helps greatly that it is a good read, not just for those who are curious about privacy but also for those who are already familiar with the privacy literature.

Science - Viktor Mayer-Schönberger

The Black Box Society offers a good dose of fresh thinking and may advance our debates over privacy. It also helps greatly that it is a good read, not just for those who are curious about privacy but also for those who are already familiar with the privacy literature.

Science - Viktor Mayer-Schönberger

Recognized internationally as a landmark work of law and political economy.
— Lawrence Joseph Commonweal

An exhilarating read, brimming with passion. Pasquale’s bold and ambitious book lifts the lid on the ‘black box society’ by tackling a wide array of issues, from secrecy in finance to credit scoring, from search engines to automated decision-making, from institutional transparency to the relationship between government and big corporations. Writing with urgency and utter conviction, he paints a compelling—and devastating—picture of the world that we are building.
— Daniel J. Solove, author of Nothing to Hide: The False Tradeoff between Privacy and Security

A timely and important book about the algorithmic processes that play such central roles in our emerging information society. Pasquale explores the abuses that have resulted from insufficient transparency and exposes the inability of either markets or regulators to instill appropriate levels of accountability. He is not a reflexive technology-basher, however, but instead offers judicious reform proposals.
— Julie E. Cohen, author of Configuring the Networked Self: Law, Code, and the Play of Everyday Practice

Frank Pasquale’s notable new book, The Black Box Society, tries to come to grips with the dangers of ‘runaway data’ and ‘black box algorithms’ more comprehensively than any other book to date…It’s an important read for anyone who is interested in the hidden pitfalls of ‘big data’ and who wants to understand just how quantified our lives have become without our knowledge.
— David Auerbach Slate

Everyone who uses the Internet for entertainment, education, news or commerce is implicated in a web of data collection whose breadth surpasses ordinary awareness…As [Pasquale’s] exposé…shows, this is a society in which basic functions are performed in deliberate obscurity through the collection and algorithmic manipulation of personal data…In The Black Box Society, Pasquale finds reason to believe that even some of the most secretive and unresponsive institutions can be held to account. Elucidating the problem is a first step.
— Steven Aftergood Nature

The Black Box Society is a frightening portrait of the ever more powerful shadowy world that blocks light from reaching our everyday lives. It is also a call to action, with a range of suggestions that inevitably pale in comparison to the gargantuan task at hand. But small steps sometimes have outsize consequences. Just ask the folks who control what we see, influence what we buy, and determine whether we can pay for it.
— Brenda Jubin Reading the Markets

The Black Box Society offers a good dose of fresh thinking and may advance our debates over privacy. It also helps greatly that it is a good read, not just for those who are curious about privacy but also for those who are already familiar with the privacy literature.
— Viktor Mayer-Schönberger Science

If you are a person in America, then there are equations trying to learn more about you…Some of these equations work for private companies and some of them work for the government, but they all generate correlations based on your behavior…Frank Pasquale’s new book The Black Box Society is a tour of how computational intelligence has come to dominate three important parts of American life: reputation, search, and finance.
— Malcolm Harris New Republic

Required reading in many law school and computer science courses…Details how secret databases and little-known applications of AI algorithms have had harmful effects on finance, business, education, and politics. These opaque practices violate personal privacy, lower credit ratings, and make unfair or biased decisions on parole, mortgage, and job applications.
— Ben Shneiderman Issues in Science and Technology

This book by Pasquale is disturbing. The premise is that corporate and public unchecked use of computer algorithms to collect and analyze data harms the public…Pasquale calls out Google, Facebook, and the financial industry for unchecked use of data to make profits and broken promises of privacy protection.
— Harry Charles Library Journal

Frank Pasquale’s new book on the secret algorithms that motor the monstrous heart of big data is a timely work of non-fiction, a ‘true conspiracy’ about regulatory weakness in the face of technological hubris and greed.
— Jonathon Sturgeon Flavorwire

The algorithmic control that law scholar Frank Pasquale eloquently and intelligently details and analyzes goes beyond money information and into almost every aspect of our lives. For this reason, although it might appear merely to be a book about technology and finance, The Black Box Society, ultimately, is a radical and political work that deserves wide attention… The Black Box Society includes, for example, a fine explanation of the way that corporate and government surveillance work in concert and why we should be concerned about both… [Pasquale’s] brutal on the subject of the NSA, but devastating in his critique of Facebook, Twitter and Google and the myths that continue to surround them: myths of neutrality, myth about the ephemeral nature of their power and more. His analysis of search is pointed and poignant, underlining that we need to understand it better and treat search results more critically and sceptically… Pasquale’s detailed analyses, and his recipes not just for transparency but also for accountability, for more rigour in regulation and harder-hitting enforcement, deserve a careful read—and then action.
— Paul Bernal Times Higher Education

Pasquale has emerged as one of the go-to thinkers on Big Data and the algorithmic economy, and The Black Box Society (along with his Twitter feed) is a great—if discomfiting—place to start. You’ll come away overwhelmed by the speed and recklessness of data compilation and its uses and abuses.
— Jonathon Sturgeon Flavorwire

By carefully breaking down ‘the business practices of leading Internet and finance companies, focusing on their use of proprietary reputation, search, and finance technologies,’ Pasquale pulls off an amazing feat of explanation, simultaneously and seamlessly explaining how and why black boxes exist, as well as what they can control and what happens when society entrusts black box technol­ogy with consequential decisions and hands immense power to the black box firms of Silicon Valley and Wall Street…The Black Box Society is a first-rate work of synthesis, combin­ing ideas from law and economics, interpretive social science, science studies, and the philosophy of technology into an essential study of the political economy of information.
— Thomas Drueke Law Library Journal

From the Publisher