Parallel R: Data Analysis in the Distributed World

by Q. Ethan McCallum, Stephen Weston

Overview

It’s tough to argue with R as a high-quality, cross-platform, open source statistical software product—unless you’re in the business of crunching Big Data. This concise book introduces you to several strategies for using R to analyze large datasets, including three chapters on using R and Hadoop together. You’ll learn the basics of Snow, Multicore, Parallel, Segue, RHIPE, and Hadoop Streaming, including how to find them, how to use them, when they work well, and when they don’t.

With these packages, you can overcome R’s single-threaded nature by spreading work across multiple CPUs, or offloading work to multiple machines to address R’s memory barrier.

• Snow: works well in a traditional cluster environment
• Multicore: popular for multiprocessor and multicore computers
• Parallel: part of the upcoming R 2.14.0 release
• R+Hadoop: provides low-level access to a popular form of cluster computing
• RHIPE: uses Hadoop’s power with R’s language and interactive shell
• Segue: lets you use Elastic MapReduce as a backend for lapply-style operations

Product Details

ISBN-13:	9781449320331
Publisher:	O'Reilly Media, Incorporated
Publication date:	10/21/2011
Sold by:	Barnes & Noble
Format:	eBook
Pages:	126
File size:	4 MB

About the Author

Q Ethan McCallum is a consultant, writer, and technology enthusiast, though perhaps not in that order. His work has appeared online on The O’Reilly Network and Java.net, and also in print publications such as C/C++ Users Journal, Doctor Dobb’s Journal, and Linux Magazine. In his professional roles, he helps companies to make smart decisions about data and technology.

Stephen Weston has been working in high performance and parallelcomputing for over 25 years. He was employed at Scientific Computing Associates in the 90's, working on the Linda programming system, invented by David Gelernter. He was also a founder of Revolution Computing, leading the development of parallel computing packages for R, including nws, foreach, doSNOW, and doMC. He works at Yale University as an HPC Specialist.

Table of Contents

Preface; Conventions Used in This Book; Using Code Examples; Safari® Books Online; How to Contact Us; Acknowledgments; Chapter 1: Getting Started; 1.1 Why R?; 1.2 Why Not R?; 1.3 The Solution: Parallel Execution; 1.4 A Road Map for This Book; 1.5 In a Hurry?; 1.6 Summary; Chapter 2: snow; 2.1 Quick Look; 2.2 How It Works; 2.3 Setting Up; 2.4 Working with It; 2.5 When It Works…; 2.6 …And When It Doesn’t; 2.7 The Wrap-up; Chapter 3: multicore; 3.1 Quick Look; 3.2 How It Works; 3.3 Setting Up; 3.4 Working with It; 3.5 When It Works…; 3.6 …And When It Doesn’t; 3.7 The Wrap-up; Chapter 4: parallel; 4.1 Quick Look; 4.2 How It Works; 4.3 Setting Up; 4.4 Working with It; 4.5 Summary of Differences; 4.6 When It Works…; 4.7 …And When It Doesn’t; 4.8 The Wrap-up; Chapter 5: A Primer on MapReduce and Hadoop; 5.1 Hadoop at Cruising Altitude; 5.2 A MapReduce Primer; 5.3 Thinking in MapReduce: Some Pseudocode Examples; 5.4 Binary and Whole-File Data: SequenceFiles; 5.5 No Cluster? No Problem! Look to the Clouds…; 5.6 The Wrap-up; Chapter 6: R+Hadoop; 6.1 Quick Look; 6.2 How It Works; 6.3 Setting Up; 6.4 Working with It; 6.5 When It Works…; 6.6 …And When It Doesn’t; 6.7 The Wrap-up; Chapter 7: RHIPE; 7.1 Quick Look; 7.2 How It Works; 7.3 Setting Up; 7.4 Working with It; 7.5 When It Works…; 7.6 …And When It Doesn’t; 7.7 The Wrap-up; Chapter 8: Segue; 8.1 Quick Look; 8.2 How It Works; 8.3 Setting Up; 8.4 Working with It; 8.5 When It Works…; 8.6 …And When It Doesn’t; 8.7 The Wrap-up; Chapter 9: New and Upcoming; 9.1 doRedis; 9.2 RevoScale R and RevoConnectR (RHadoop); 9.3 cloudNumbers.com;