Gift Guide

Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work [NOOK Book]


What is bad data? Some people consider it a technical phenomenon, like missing values or malformed records, but bad data includes a lot more. In this handbook, data expert Q. Ethan McCallum has gathered 19 colleagues from every corner of the data arena to reveal how they’ve recovered from nasty data problems.

From cranky storage to poor representation to misguided policy, there are many paths to bad data. Bottom line? Bad data is data that gets...

See more details below
Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work

Available on NOOK devices and apps  
  • NOOK Devices
  • Samsung Galaxy Tab 4 NOOK 7.0
  • Samsung Galaxy Tab 4 NOOK 10.1
  • NOOK HD Tablet
  • NOOK HD+ Tablet
  • NOOK eReaders
  • NOOK Color
  • NOOK Tablet
  • Tablet/Phone
  • NOOK for Windows 8 Tablet
  • NOOK for iOS
  • NOOK for Android
  • NOOK Kids for iPad
  • PC/Mac
  • NOOK for Windows 8
  • NOOK for PC
  • NOOK for Mac

Want a NOOK? Explore Now

NOOK Book (eBook)
$17.99 price
(Save 43%)$31.99 List Price


What is bad data? Some people consider it a technical phenomenon, like missing values or malformed records, but bad data includes a lot more. In this handbook, data expert Q. Ethan McCallum has gathered 19 colleagues from every corner of the data arena to reveal how they’ve recovered from nasty data problems.

From cranky storage to poor representation to misguided policy, there are many paths to bad data. Bottom line? Bad data is data that gets in the way. This book explains effective ways to get around it.

Among the many topics covered, you’ll discover how to:

  • Test drive your data to see if it’s ready for analysis
  • Work spreadsheet data into a usable form
  • Handle encoding problems that lurk in text data
  • Develop a successful web-scraping effort
  • Use NLP tools to reveal the real sentiment of online reviews
  • Address cloud computing issues that can impact your analysis effort
  • Avoid policies that create data analysis roadblocks
  • Take a systematic approach to data quality analysis
Read More Show Less

Product Details

  • ISBN-13: 9781449324971
  • Publisher: O'Reilly Media, Incorporated
  • Publication date: 11/7/2012
  • Sold by: Barnes & Noble
  • Format: eBook
  • Edition number: 1
  • Pages: 264
  • File size: 6 MB

Meet the Author

Q Ethan McCallum is a consultant, writer, and technology enthusiast, though perhaps not in that order. His work has appeared online on The O’Reilly Network and, and also in print publications such as C/C++ Users Journal, Doctor Dobb’s Journal, and Linux Magazine. In his professional roles, he helps companies to make smart decisions about data and technology.

Read More Show Less

Table of Contents

About the Authors;
Conventions Used in This Book;
Using Code Examples;
Safari® Books Online;
How to Contact Us;
Chapter 1: Setting the Pace: What Is Bad Data?;
Chapter 2: Is It Just Me, or Does This Data Smell Funny?;
2.1 Understand the Data Structure;
2.2 Field Validation;
2.3 Value Validation;
2.4 Physical Interpretation of Simple Statistics;
2.5 Visualization;
2.6 Keyword PPC Example;
2.7 Search Referral Example;
2.8 Recommendation Analysis;
2.9 Time Series Data;
2.10 Conclusion;
Chapter 3: Data Intended for Human Consumption, Not Machine Consumption;
3.1 The Data;
3.2 The Problem: Data Formatted for Human Consumption;
3.3 The Solution: Writing Code;
3.4 Postscript;
3.5 Other Formats;
3.6 Summary;
Chapter 4: Bad Data Lurking in Plain Text;
4.1 Which Plain Text Encoding?;
4.2 Guessing Text Encoding;
4.3 Normalizing Text;
4.4 Problem: Application-Specific Characters Leaking into Plain Text;
4.5 Text Processing with Python;
4.6 Exercises;
Chapter 5: (Re)Organizing the Web’s Data;
5.1 Can You Get That?;
5.2 General Workflow Example;
5.3 The Real Difficulties;
5.4 The Dark Side;
5.5 Conclusion;
Chapter 6: Detecting Liars and the Confused in Contradictory Online Reviews;
6.1 Weotta;
6.2 Getting Reviews;
6.3 Sentiment Classification;
6.4 Polarized Language;
6.5 Corpus Creation;
6.6 Training a Classifier;
6.7 Validating the Classifier;
6.8 Designing with Data;
6.9 Lessons Learned;
6.10 Summary;
6.11 Resources;
Chapter 7: Will the Bad Data Please Stand Up?;
7.1 Example 1: Defect Reduction in Manufacturing;
7.2 Example 2: Who’s Calling?;
7.3 Example 3: When “Typical” Does Not Mean “Average”;
7.4 Lessons Learned;
7.5 Will This Be on the Test?;
Chapter 8: Blood, Sweat, and Urine;
8.1 A Very Nerdy Body Swap Comedy;
8.2 How Chemists Make Up Numbers;
8.3 All Your Database Are Belong to Us;
8.4 Check, Please;
8.5 Live Fast, Die Young, and Leave a Good-Looking Corpse Code Repository;
8.6 Rehab for Chemists (and Other Spreadsheet Abusers);
8.7 tl;dr;
Chapter 9: When Data and Reality Don’t Match;
9.1 Whose Ticker Is It Anyway?;
9.2 Splits, Dividends, and Rescaling;
9.3 Bad Reality;
9.4 Conclusion;
Chapter 10: Subtle Sources of Bias and Error;
10.1 Imputation Bias: General Issues;
10.2 Reporting Errors: General Issues;
10.3 Other Sources of Bias;
10.4 Conclusions;
10.5 References;
Chapter 11: Don’t Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad?;
11.1 But First, Let’s Reflect on Graduate School …;
11.2 Moving On to the Professional World;
11.3 Moving into Government Work;
11.4 Government Data Is Very Real;
11.5 Service Call Data as an Applied Example;
11.6 Moving Forward;
11.7 Lessons Learned and Looking Ahead;
Chapter 12: When Databases Attack: A Guide for When to Stick to Files;
12.1 History;
12.2 Consider Files as Your Datastore;
12.3 File Concepts;
12.4 A Web Framework Backed by Files;
12.5 Reflections;
Chapter 13: Crouching Table, Hidden Network;
13.1 A Relational Cost Allocations Model;
13.2 The Delicate Sound of a Combinatorial Explosion…;
13.3 The Hidden Network Emerges;
13.4 Storing the Graph;
13.5 Navigating the Graph with Gremlin;
13.6 Finding Value in Network Properties;
13.7 Think in Terms of Multiple Data Models and Use the Right Tool for the Job;
13.8 Acknowledgments;
Chapter 14: Myths of Cloud Computing;
14.1 Introduction to the Cloud;
14.2 What Is “The Cloud”?;
14.3 The Cloud and Big Data;
14.4 Introducing Fred;
14.5 At First Everything Is Great;
14.6 They Put 100% of Their Infrastructure in the Cloud;
14.7 As Things Grow, They Scale Easily at First;
14.8 Then Things Start Having Trouble;
14.9 They Need to Improve Performance;
14.10 Higher IO Becomes Critical;
14.11 A Major Regional Outage Causes Massive Downtime;
14.12 Higher IO Comes with a Cost;
14.13 Data Sizes Increase;
14.14 Geo Redundancy Becomes a Priority;
14.15 Horizontal Scale Isn’t as Easy as They Hoped;
14.16 Costs Increase Dramatically;
14.17 Fred’s Follies;
14.18 Myth 1: Cloud Is a Great Solution for All Infrastructure Components;
14.19 Myth 2: Cloud Will Save Us Money;
14.20 Myth 3: Cloud IO Performance Can Be Improved to Acceptable Levels Through Software RAID;
14.21 Myth 4: Cloud Computing Makes Horizontal Scaling Easy;
14.22 Conclusion and Recommendations;
Chapter 15: The Dark Side of Data Science;
15.1 Avoid These Pitfalls;
15.2 Know Nothing About Thy Data;
15.3 Thou Shalt Provide Your Data Scientists with a Single Tool for All Tasks;
15.4 Thou Shalt Analyze for Analysis’ Sake Only;
15.5 Thou Shalt Compartmentalize Learnings;
15.6 Thou Shalt Expect Omnipotence from Data Scientists;
15.7 Final Thoughts;
Chapter 16: How to Feed and Care for Your Machine-Learning Experts;
16.1 Define the Problem;
16.2 Fake It Before You Make It;
16.3 Create a Training Set;
16.4 Pick the Features;
16.5 Encode the Data;
16.6 Split Into Training, Test, and Solution Sets;
16.7 Describe the Problem;
16.8 Respond to Questions;
16.9 Integrate the Solutions;
16.10 Conclusion;
Chapter 17: Data Traceability;
17.1 Why?;
17.2 Personal Experience;
17.3 Immutability: Borrowing an Idea from Functional Programming;
17.4 An Example;
17.5 Conclusion;
Chapter 18: Social Media: Erasable Ink?;
18.1 Social Media: Whose Data Is This Anyway?;
18.2 Control;
18.3 Commercial Resyndication;
18.4 Expectations Around Communication and Expression;
18.5 Technical Implications of New End User Expectations;
18.6 What Does the Industry Do?;
18.7 What Should End Users Do?;
18.8 How Do We Work Together?;
Chapter 19: Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough;
19.1 Framework Introduction: The Four Cs of Data Quality Analysis;
19.2 Complete;
19.3 Coherent;
19.4 Correct;
19.5 aCcountable;
19.6 Conclusion;

Read More Show Less

Customer Reviews

Be the first to write a review
( 0 )
Rating Distribution

5 Star


4 Star


3 Star


2 Star


1 Star


Your Rating:

Your Name: Create a Pen Name or

Barnes & Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation


  • - By submitting a review, you grant to Barnes & and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Terms of Use.
  • - Barnes & reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously

    If you find inappropriate content, please report it to Barnes & Noble
    Why is this product inappropriate?
    Comments (optional)