Scientific Data Management: Challenges, Technology, and Deployment

by Arie Shoshani (Editor), Doron Rotem (Editor)

Overview

Dealing with the volume, complexity, and diversity of data currently being generated by scientific experiments and simulations often causes scientists to waste productive time. Scientific Data Management: Challenges, Technology, and Deployment describes cutting-edge technologies and solutions for managing and analyzing vast amounts of data, helping scientists focus on their scientific goals.

The book begins with coverage of efficient storage systems, discussing how to write and read large volumes of data without slowing the simulation, analysis, or visualization processes. It then focuses on the efficient data movement and management of storage spaces and explores emerging database systems for scientific data. The book also addresses how to best organize data for analysis purposes, how to effectively conduct searches over large datasets, how to successfully automate multistep scientific process workflows, and how to automatically collect metadata and lineage information.

This book provides a comprehensive understanding of the latest techniques for managing data during scientific exploration processes, from data generation to data analysis. Enhanced by numerous detailed color images, it includes real-world examples of applications drawn from biology, ecology, geology, climatology, and more.

Check out Dr. Shoshani discuss the book during an interview with International Science Grid This Week (iSGTW): http://www.isgtw.org/?pid=1002259

Product Details

ISBN-13:	9780367384760
Publisher:	Taylor & Francis
Publication date:	10/18/2019
Edition description:	Reprint
Pages:	592
Product dimensions:	6.12(w) x 9.19(h) x (d)

About the Author

Arie Shoshani is a senior staff scientist at Lawrence Berkeley National Laboratory, where he heads the Scientific Data Management Research Group. Dr. Shoshani is also the director of the Scientific Data Management Center, one of several large computer science centers supported by the SciDAC program of the U.S. Department of Energy.

Doron Rotem is a senior staff scientist at Lawrence Berkeley National Laboratory, where he heads the research program on scientific data management.

Table of Contents

List of Figures ix

List of Tables xvii

Acknowledgments xix

Contributors xxi

Introduction xxvii

I Storage Technology and Efficient Storage Access

1 Storage Technology Jason Hick John Shalf 3

2 Parallel Data Storage and Access Robert Ross Alok Choudhary Garth Gibson Wei-keng Liao 35

3 Dynamic Storage Management Arie Shoshani Flavia Donna Junmin Gu Jason Hick Maarten Litmaath Alex Sim 73

II Data Transfer and Scheduling

4 Coordination of Access to Large-Scale Datasets in Distributed Environments Tevfik Kosar Andrei Hutanu Jon McLaren Douglas Thain 115

5 High-Throughput Data Movement Scott Klasky Hasan Abbasi Viraj Bhat Ciprian Docan Steve Hodson Chen Jin Jay Lofstead Manish Parashar Karsten Schwan Matthew Wolf 151

III Specialized Retrieval Techniques and Database Systems

6 Accelerating Queries on Very Large Datasets Ekow Otoo Kesheng Wu 183

7 Emerging Database Systems in Support of Scientific Data Per Svensson Peter Boncz Milena Ivanovo Martin Kersten Niels Nes Doron Rotem 235

IV Data Analysis, Integration, and Visualization Methods

8 Scientific Data Analysis Chandrika Kamath Nikil Wale George Karypis Gaurav Pandey Vipin Kumar Krishna Rajan Nagiza F. Samatova Paul Breimyer Guruprasad Kara Chongle Pan Srikanth Yoginath 281

9 Scientific Data Management Challenges in High-Performance Visual Data Analysis E. Wes Bethel Prabhat Hank Childs Ajith Mascarenhas Valerio Pascucci 325

10 Interoperability and Data Integration in the Geosciences Michael Gertz Carlos Rueda Jianting Zhang 369

11 Analyzing Data Streams in Scientific Applications Tore Risch Samuel Madden Hari Balakrishan Lewis Girod Ryan Newton Milena Ivanovo Erik Zeitler Johannes Gehrke Biswanath Panda Mirek Riedewald 399

V Scientific Process Management

12 Metadata and Provenance Management Ewa Deelman Bruce Berriman Ann Chervenak Oscar Corcho Paul Groth Luc Moreau 433

13 Scientific Process Automation and Workflow Management Bertram Ludäscher Ilkay Altintas Shawn Bowers Julian Cummings Terence Critchlow Ewa Deelman David De Roure Juliana Freire Carole Goble Matthew Jones Scott Klasky Timothy McPhillips Norbert Podhorszki Claudio Silva Ian Taylor Mladen Vouk 467

Conclusions and Future Outlook Arte Shoshani Doron Rotem 509

Index 515

What People are Saying About This

From the Publisher

"… Each chapter contains insights and experience gleaned by experts and luminaries in storage who are confronting and managing the data tsunami that has now inundated the leading-edge scientific and supercomputing centers around the world. Individuals in a variety of scientific and commercial areas who are struggling to manage large amounts of data should find this book both educational and useful."
—Ron Farber, Scientific Computing, 2010

Editorial Reviews

"… Each chapter contains insights and experience gleaned by experts and luminaries in storage who are confronting and managing the data tsunami that has now inundated the leading-edge scientific and supercomputing centers around the world. Individuals in a variety of scientific and commercial areas who are struggling to manage large amounts of data should find this book both educational and useful."
—Ron Farber, Scientific Computing, 2010

From the Publisher