Beautiful Data: The Stories Behind Elegant Data Solutions

by Toby Segaran, Jeff Hammerbacher

Overview

In this insightful book, you'll learn from the best data practitioners in the field just how wide-ranging — and beautiful — working with data can be. Join 39 contributors as they explain how they developed simple and elegant solutions on projects ranging from the Mars lander to a Radiohead video.

With Beautiful Data, you will:

• Explore the opportunities and challenges involved in working with the vast number of datasets made available by the Web
• Learn how to visualize trends in urban crime, using maps and data mashups
• Discover the challenges of designing a data processing system that works within the constraints of space travel
• Learn how crowdsourcing and transparency have combined to advance the state of drug research
• Understand how new data can automatically trigger alerts when it matches or overlaps pre-existing data
• Learn about the massive infrastructure required to create, capture, and process DNA data

That's only small sample of what you'll find in Beautiful Data. For anyone who handles data, this is a truly fascinating book. Contributors include:

Nathan Yau

Jonathan Follett and Matt Holm

J.M. Hughes

Raghu Ramakrishnan, Brian Cooper, and Utkarsh Srivastava

Jeff Hammerbacher

Jason Dykes and Jo Wood

Jeff Jonas and Lisa Sokol

Jud Valeski

Alon Halevy and Jayant Madhavan

Aaron Koblin with Valdean Klump

Michal Migurski

Jeff Heer

Coco Krumme

Peter Norvig

Matt Wood and Ben Blackburne

Jean-Claude Bradley, Rajarshi Guha, Andrew Lang, Pierre Lindenbaum, Cameron Neylon, Antony Williams, and Egon Willighagen

Lukas Biewald and Brendan O'Connor

Hadley Wickham, Deborah Swayne, and David Poole

Andrew Gelman, Jonathan P. Kastellec, and Yair Ghitza

Toby Segaran

Product Details

ISBN-13:	9780596157111
Publisher:	O'Reilly Media, Incorporated
Publication date:	08/15/2009
Pages:	364
Product dimensions:	7.00(w) x 9.10(h) x 1.00(d)

About the Author

Toby Segaran is the author of Programming Collective Intelligence, a very popular O'Reilly title. He was the founder of Incellico, a biotech software company later acquired by Genstruct. He currently holds the title of Data Magnate at Metaweb Technologies and is a frequent speaker at technology conferences.

Jeff Hammerbacher is the Vice President of Products and Chief Scientist at Cloudera. Jeff was an Entrepreneur in Residence at Accel Partners immediately prior to joining Cloudera. Before Accel, he conceived, built, and led the Data team at Facebook. The Data team was responsible for driving many of the statistics and machine learning applications at Facebook, as well as building out the infrastructure to support these tasks for massive data sets. The team produced several academic papers and two open source projects: Hive, a system for offline analysis built above Hadoop, and Cassandra, a structured storage system on a P2P network. Before joining Facebook, Jeff was a quantitative analyst on Wall Street. Jeff earned his Bachelor's Degree in Mathematics from Harvard University.

Table of Contents

Dedication; Preface; How This Book Is Organized; Conventions Used in This Book; Using Code Examples; How to Contact Us; Safari® Books Online; Chapter 1: Seeing Your Life in Data; 1.1 Personal Environmental Impact Report (PEIR); 1.2 your.flowingdata (YFD); 1.3 Personal Data Collection; 1.4 Data Storage; 1.5 Data Processing; 1.6 Data Visualization; 1.7 The Point; 1.8 How to Participate; Chapter 2: The Beautiful People: Keeping Users in Mind When Designing Data Collection Methods; 2.1 Introduction: User Empathy Is the New Black; 2.2 The Project: Surveying Customers About a New Luxury Product; 2.3 Specific Challenges to Data Collection; 2.4 Designing Our Solution; 2.5 Results and Reflection; Chapter 3: Embedded Image Data Processing on Mars; 3.1 Abstract; 3.2 Introduction; 3.3 Some Background; 3.4 To Pack or Not to Pack; 3.5 The Three Tasks; 3.6 Slotting the Images; 3.7 Passing the Image: Communication Among the Three Tasks; 3.8 Getting the Picture: Image Download and Processing; 3.9 Image Compression; 3.10 Downlink, or, It's All Downhill from Here; 3.11 Conclusion; Chapter 4: Cloud Storage Design in a PNUTShell; 4.1 Introduction; 4.2 Updating Data; 4.3 Complex Queries; 4.4 Comparison with Other Systems; 4.5 Conclusion; 4.6 Acknowledgments; 4.7 References; Chapter 5: Information Platforms and the Rise of the Data Scientist; 5.1 Libraries and Brains; 5.2 Facebook Becomes Self-Aware; 5.3 A Business Intelligence System; 5.4 The Death and Rebirth of a Data Warehouse; 5.5 Beyond the Data Warehouse; 5.6 The Cheetah and the Elephant; 5.7 The Unreasonable Effectiveness of Data; 5.8 New Tools and Applied Research; 5.9 MAD Skills and Cosmos; 5.10 Information Platforms As Dataspaces; 5.11 The Data Scientist; 5.12 Conclusion; Chapter 6: The Geographic Beauty of a Photographic Archive; 6.1 Beauty in Data: Geograph; 6.2 Visualization, Beauty, and Treemaps; 6.3 A Geographic Perspective on Geograph Term Use; 6.4 Beauty in Discovery; 6.5 Reflection and Conclusion; 6.6 Acknowledgments; 6.7 References; Chapter 7: Data Finds Data; 7.1 Introduction; 7.2 The Benefits of Just-in-Time Discovery; 7.3 Corruption at the Roulette Wheel; 7.4 Enterprise Discoverability; 7.5 Federated Search Ain't All That; 7.6 Directories: Priceless; 7.7 Relevance: What Matters and to Whom?; 7.8 Components and Special Considerations; 7.9 Privacy Considerations; 7.10 Conclusion; Chapter 8: Portable Data in Real Time; 8.1 Introduction; 8.2 The State of the Art; 8.3 Social Data Normalization; 8.4 Conclusion: Mediation via Gnip; Chapter 9: Surfacing the Deep Web; 9.1 What Is the Deep Web?; 9.2 Alternatives to Offering Deep-Web Access; 9.3 Conclusion and Future Work; 9.4 References; Chapter 10: Building Radiohead's House of Cards; 10.1 How It All Started; 10.2 The Data Capture Equipment; 10.3 The Advantages of Two Data Capture Systems; 10.4 The Data; 10.5 Capturing the Data, aka "The Shoot"; 10.6 Processing the Data; 10.7 Post-Processing the Data; 10.8 Launching the Video; 10.9 Conclusion; Chapter 11: Visualizing Urban Data; 11.1 Introduction; 11.2 Background; 11.3 Cracking the Nut; 11.4 Making It Public; 11.5 Revisiting; 11.6 Conclusion; Chapter 12: The Design of Sense.us; 12.1 Visualization and Social Data Analysis; 12.2 Data; 12.3 Visualization; 12.4 Collaboration; 12.5 Voyagers and Voyeurs; 12.6 Conclusion; 12.7 References; Chapter 13: What Data Doesn't Do; 13.1 When Doesn't Data Drive?; 13.2 Conclusion; 13.3 References; Chapter 14: Natural Language Corpus Data; 14.1 Word Segmentation; 14.2 Secret Codes; 14.3 Spelling Correction; 14.4 Other Tasks; 14.5 Discussion and Conclusion; 14.6 Acknowledgments; Chapter 15: Life in Data: The Story of DNA; 15.1 DNA As a Data Store; 15.2 DNA As a Data Source; 15.3 Fighting the Data Deluge; 15.4 The Future of DNA; 15.5 Acknowledgments; Chapter 16: Beautifying Data in the Real World; 16.1 The Problem with Real Data; 16.2 Providing the Raw Data Back to the Notebook; 16.3 Validating Crowdsourced Data; 16.4 Representing the Data Online; 16.5 Closing the Loop: Visualizations to Suggest New Experiments; 16.6 Building a Data Web from Open Data and Free Services; 16.7 Acknowledgments; 16.8 References; Chapter 17: Superficial Data Analysis: Exploring Millions of Social Stereotypes; 17.1 Introduction; 17.2 Preprocessing the Data; 17.3 Exploring the Data; 17.4 Age, Attractiveness, and Gender; 17.5 Looking at Tags; 17.6 Which Words Are Gendered?; 17.7 Clustering; 17.8 Conclusion; 17.9 Acknowledgments; 17.10 References; Chapter 18: Bay Area Blues: The Effect of the Housing Crisis; 18.1 Introduction; 18.2 How Did We Get the Data?; 18.3 Geocoding; 18.4 Data Checking; 18.5 Analysis; 18.6 The Influence of Inflation; 18.7 The Rich Get Richer and the Poor Get Poorer; 18.8 Geographic Differences; 18.9 Census Information; 18.10 Exploring San Francisco; 18.11 Conclusion; 18.12 References; Chapter 19: Beautiful Political Data; 19.1 Example 1: Redistricting and Partisan Bias; 19.2 Example 2: Time Series of Estimates; 19.3 Example 3: Age and Voting; 19.4 Example 4: Public Opinion and Senate Voting on Supreme Court Nominees; 19.5 Example 5: Localized Partisanship in Pennsylvania; 19.6 Conclusion; 19.7 References; Chapter 20: Connecting Data; 20.1 What Public Data Is There, Really?; 20.2 The Possibilities of Connected Data; 20.3 Within Companies; 20.4 Impediments to Connecting Data; 20.5 Possible Solutions; 20.6 Conclusion; Contributors; COLOPHON;