Data Profiling

by Ziawasch Abedjan, Lukasz Golab, Felix Naumann, Thorsten Papenbrock

Overview

Data profiling refers to the activity of collecting data about data, {i.e.}, metadata. Most IT professionals and researchers who work with data have engaged in data profiling, at least informally, to understand and explore an unfamiliar dataset or to determine whether a new dataset is appropriate for a particular task at hand. Data profiling results are also important in a variety of other situations, including query optimization, data integration, and data cleaning. Simple metadata are statistics, such as the number of rows and columns, schema and datatype information, the number of distinct values, statistical value distributions, and the number of null or empty values in each column. More complex types of metadata are statements about multiple columns and their correlation, such as candidate keys, functional dependencies, and other types of dependencies.

This book provides a classification of the various types of profilable metadata, discusses popular data profiling tasks,and surveys state-of-the-art profiling algorithms. While most of the book focuses on tasks and algorithms for relational data profiling, we also briefly discuss systems and techniques for profiling non-relational data such as graphs and text. We conclude with a discussion of data profiling challenges and directions for future work in this area.

Product Details

ISBN-13:	9783031007378
Publisher:	Springer International Publishing
Publication date:	11/08/2018
Series:	Synthesis Lectures on Data Management
Pages:	136
Product dimensions:	7.52(w) x 9.25(h) x (d)

About the Author

Ziawasch Abedjan is Assistant Professor and Head of the ""Big Data Management"" (BigDaMa) Group at the Technische Universitat Berlin. Before Ziawasch was a postdoc at the ""Computer Science and Artificial Intelligence Laboratory"" at MIT working on various data integration topics. Ziawasch received his Ph.D. from the Hasso Plattner Institute in Potsdam, Germany. His research interests include, data mining, data integration, and data profiling.
Lukasz Golab is an Associate Professor at the University of Waterloo and a Canada Research Chair. Prior to joining Waterloo, he was a Senior Member of Research Staff at AT&T Labs in Florham Park, NJ, USA. He holds a B.Sc. in Computer Science (with High Distinction) from the University of Toronto and a Ph.D. in Computer Science (with Alumni Gold Medal) from the University of Waterloo. His publications span several research areas within data management and data analytics, including data stream management, data profiling, data quality, data science for social good, and educational data mining.
Felix Naumann studied mathematics, economy, and computer sciences at the University of Technology in Berlin. After receiving his diploma in 1997 he joined the graduate school ""Distributed Information Systems"" at Humboldt University of Berlin. He completed his Ph.D. thesis on ""Quality-driven Query Answering"" in 2000. In 2001 and 2002 he worked at the IBM Almaden Research Center on topics around data integration. From 2003-2006 he was an assistant professor of information integration at the Humboldt University of Berlin. Since 2006 he has held the chair for information systems at the Hasso Plattner Institute at the University of Potsdam in Germany. He is Editor-in-Chief of the Information Systems journal. His research interests are in the areas of information integration, data quality, data cleansing, text extraction, and-of course-data profiling. He has given numerous invited talks and tutorials on the topic of the book.
Thorsten Papenbrock is a researcher and lecturer at the Hasso Plattner Institute at the University of Potsdam in Germany. He received his M.Sc. in IT-Systems Engineering in 2014 and his Ph.D. in Computer Science in 2017. His thesis on ""Data Profiling-Efficient Discovery of Dependencies"" inspired many sections of this book. In research, his main interests are data profiling, data cleaning, distributed and parallel computing, database systems, and data analytics.

Table of Contents

Preface.- Acknowledgments.- Discovering Metadata.- Data Profiling Tasks.- Single-Column Analysis.- Dependency Discovery.- Relaxed and Other Dependencies.- Use Cases.- Profiling Non-Relational Data.- Data Profiling Tools.- Data Profiling Challenges.- Conclusions.- Bibliography.- Authors' Biographies .