Want a NOOK? Explore Now
Master the Linux Tools That Will Make You a More Productive, Effective Programmer
The Linux Programmer's Toolbox helps you tap into the vast collection of open source tools available for GNU/Linux. Author John Fusco systematically describes the most useful tools available on most GNU/Linux distributions using concise examples that you can easily modify to meet your needs.
You'll start by learning the basics of downloading, building, and installing open source projects. You'll then learn how open source tools are distributed, and what to look for to avoid wasting time on projects that aren't ready for you. Next, you'll learn the ins and outs of building your own projects. Fusco also demonstrates what to look for in a text editor, and may even show you a few new tricks in your favorite text editor.
You'll enhance your knowledge of the Linux kernel by learning how it interacts with your software. Fusco walks you through the fundamentals of the Linux kernel with simple, thought-provoking examples that illustrate the principles behind the operating system. Then he shows you how to put this knowledge to use with more advanced tools. He focuses on how to interpret output from tools like sar, vmstat, valgrind, strace, and apply it to your application; how to take advantage of various programming APIs to develop your own tools; and how to write code that monitors itself.
Next, Fusco covers tools that help you enhance the performance of your software. He explains the principles behind today's multicore CPUs and demonstrates how to squeeze the most performance from these systems. Finally, you'll learn tools and techniques to debug your code under anycircumstances.
Coverage includes
Linux has no shortage of tools. Many are inherited from Unix, with cryptic two-letter names that conjure up images of developers trying to preserve space on a punch card. Happily, those days are long gone, but the legacy remains.
Many of those old tools are still quite useful. Most are highly specialized. Each may do only one thing but does it very well. Highly specialized tools often have many options that can make them intimidating to use. Consider the first time you used grepand learned what a regular expression was. Perhaps you haven't mastered regular expression syntax yet (don't worry; no one else has, either). That's not important, because you don't need to be a master of regular expressions to put grep to good use.
If there's one thing that I hope you learn from this book, it's that there are many tools out there that you can use without having to master them. You don't need to invest an enormous amount of time reading manuals before you can be productive. I hope you will discover new tools that you may not have been familiar with. Some of the tools this book looks at are quite old and some are new. All of them are useful. As you learn more about each tool, you will find more uses for it.
I use the term tool loosely in this book. To me, creating tools is as important as using tools, so I have included various APIs that are not usually covered in much detail in other books. In addition, this book provides some background on the internal workings of the Linux kernel that are necessary to understand what some tools are trying to tell you. I present a unique perspective on the kernel: the user's point of view. You will find enough information to allow you tounderstand the ground rules that the kernel sets for every process, and I promise you will not have to read a single line of kernel source code.
What you will not find in this book is reconstituted man pages or other documentation stitched into the text. The GNU and Linux developers have done a great job of documenting their work, but that documentation can be hard to find for the inexperienced user. Rather than reprint documentation that will be out of date by the time you read this, I show you some ingenious ways to find the most up-to-date documentation.
GNU/Linux documentation is abundant, but it's not always easy to read. You can read a 10,000-word document for a tool and still not have a clue what the tool does or how to use it. This is where I have tried to fill in the missing pieces. I have tried to explain not just how to use each tool, but also why you would want to use it. Wherever possible, I have provided simple, brief examples that you can type and modify yourself to enhance your understanding of the tools and Linux itself.
What all the tools in this book have in common is that they are available at no cost. Most come with standard Linux distributions, and for those that may not, I have included URLs so that you can download them yourself.
As much as possible, I tried to keep the material interesting and fun. Who Should Read This Book
This book is written for intermediate to advanced Linux programmers who wish to become more productive and gain a better understanding of the Linux programming environment. If you're an experienced Windows programmer who feels like a fish out of water in the Linux environment, then this book is for you, too.
Non-programmers should also find this book useful because many of the tools and topics I cover have applications beyond programming. If you are a system administrator, or just a Linux enthusiast, then there's something for you in this book, too. The Purpose of This Book
I wrote this book as a follow-up to an article I wrote for the Linux Journal entitled "Ten Commands Every Linux Developer Should Know." The inspiration for this article came from my own experience as a Linux programmer. In my daily work I make it a point to invest some of my time in learning something new, even if it means a temporary lull in progress on my project. Invariably this strategy has paid off. I have always been amazed at how many times I learned about a tool or feature that I concluded would not be useful, only to find a use for it shortly afterward. This has always been a powerful motivation for me to keep learning. I hope that by reading this book, you will follow my example and enhance your skills on a regular basis.
It's also just plain fun to learn about this stuff. If you are like me, you enjoy working with Linux. Motivating yourself to learn more has never been a problem. Because Linux is open source, you have the opportunity to understand all of its inner workings, which is not possible with closed source environments like Windows. In this book I present several freely available resources available to help you learn more. How to Read This Book
The chapters are presented such that each chapter can stand on its own. Later chapters require some background knowledge that is presented in the earlier chapters. Wherever possible, I have cross-referenced the material to help you find the necessary background information.
I believe the best way to learn is by example, so I have tried to provide simple examples wherever possible. I encourage the reader to try the examples and experiment. How This Book Is Organized
Chapter 1, Downloading and Installing Open Source Tools, covers the mechanisms used to distribute open source code. I discuss the various package formats used by different distributions and the advantages and disadvantages of each. I present several tools used to maintain packages and how to use them.
Chapter 2, Building from Source, covers the basics of building an open source project. I present some of the tools used to build software and alternatives that are emerging. There are several tips and tricks in this chapter that you can use to master your use of make. I also show you how to configure projects that are distributed with GNU's autoconf tools so that you can customize them to meet your needs. Finally, I cover the stages of the build that are often misunderstood by many programmers. I look at some of the errors and warnings you are likely to encounter and how to interpret them.
Chapter 3, Finding Help, looks at the various documentation formats tucked away in your Linux distribution that you may not know about. I look at the tools used to read these formats and discuss effective ways to use them.
Chapter 4, Editing and Maintaining Source Files, discusses the various text editors available for programmers as well as the advantages and disadvantages of each. I present a set of features that every programmer should look for in an editor and measure each editor against these. This chapter also covers the basics of revision control, which is vital for software project management.
Chapter 5, What Every Developer Should Know about the Kernel, looks at the kernel from a user's perspective. In this chapter you will find the necessary background information required to understand the workings of a Linux system. I introduce several tools that allow you to see how your code interacts with the kernel.
Chapter 6, Understanding Processes, focuses on processes, their characteristics, and how to manage them. I cover a good deal of background required to introduce the tools in this chapter and understand why they are useful. In addition, this chapter introduces several programming APIs that you can use to create your own tools.
Chapter 7, Communication Between Processes, introduces the concepts behind inter-process communication (IPC). This chapter contains mostly background information required for Chapter 8. Along with each IPC mechanism, I introduce the APIs required to use it along with a working example.
Chapter 8, Debugging IPC with Shell Commands, presents several tools available to debug applications that use IPC. It builds on the information from Chapter 7 to help you interpret the output of these tools, which can be difficult to understand.
Chapter 9, Performance Tuning, introduces tools to measure the performance of your system as well as the performance of individual applications. I present several examples to illustrate how programming can impact performance. I also discuss some of the performance issues that are unique to multi-core processors.
Chapter 10, Debugging, presents several tools and techniques that you can use to debug applications. I look at some open source memory debugging tools including Valgrind and Electric Fence. I also take an in-depth look at the capabilities of gdb, and how to use it effectively.
Foreword xvii Preface xix Acknowledgments xxiii About the Author xxv Chapter 1 Downloading and Installing Open Source Tools 1
1.1 Introduction 1
1.2 What Is Open Source? 2
1.3 What Does Open Source Mean to You? 2
1.4 An Introduction to Archive Files 4
1.5 Know Your Package Manager 12
1.6 Some Words about Security and Packages 17
1.7 Inspecting Package Contents 27
1.8 Keeping Packages up to Date 33
1.9 Summary 39 Chapter 2 Building from Source 41
2.1 Introduction 41
2.2 Build Tools 41
2.3 The Build Process 74
2.4 Understanding Errors and Warnings 78
2.5 Summary 100 Chapter 3 Finding Help 103
3.1 Introduction 103
3.2 Online Help Tools 103
3.3 Other Places to Look 120
3.4 Documentation Formats 124
3.5 Internet Sources of Information 131
3.6 Finding Information about the Linux Kernel 134
3.7 Summary 138 Chapter 4 Editing and Maintaining Source Files 141
4.1 Introduction 141
4.2 The Text Editor 142
4.3 Revision Control 189
4.4 Source Code Beautifiers and Browsers 203
4.5 Summary 216 Chapter 5 What Every Developer Should Know about the Kernel 221
5.1 Introduction 221
5.2 User Mode versus Kernel Mode 222
5.3 The Process Scheduler 226
5.4 Understanding Devices and Device Drivers 257
5.5 The I/O Scheduler 282
5.6 Memory Management in User Space 286
5.7 Summary 315 Chapter 6 Understanding Processes 317
6.1 Introduction 317
6.2 Where Processes Come From 317
6.3 The exec Functions 320
6.4 Process Synchronization with wait 327
6.5 The Process Footprint 329
6.6 Setting Process Limits 340
6.7 Processes and procfs 343
6.8 Tools for Managing Processes 346
6.9 Summary 355Chapter 7 Communication between Processes 357
7.1 Introduction 357
7.2 IPC Using Plain Files 358
7.3 Shared Memory 363
7.4 Signals 370
7.5 Pipes 381
7.6 Sockets 382
7.7 Message Queues 393
7.8 Semaphores 402
7.9 Summary 412 Chapter 8 Debugging IPC with Shell Commands 415
8.1 Introduction 415
8.2 Tools for Working with Open Files 415
8.3 Dumping Data from a File 420
8.4 Shell Tools for System V IPC 426
8.5 Tools for Working with POSIX IPC 431
8.6 Tools for Working with Signals 434
8.7 Tools for Working with Pipes and Sockets 437
8.8 Using Inodes to Identify Files and IPC Objects 440
8.9 Summary 442 Chapter 9 Performance Tuning 445
9.1 Introduction 445
9.2 System Performance 445
9.3 Application Performance 475
9.4 Multiprocessor Performance 501
9.5 Summary 509 Chapter 10 Debugging 513
10.1 Introduction 513
10.2 The Most Basic Debugging Tool: printf 514
10.3 Getting Comfortable with the GNU Debugger: gdb 529
10.4 Debugging Shared Objects 561
10.5 Looking for Memory Issues 569
10.6 Unconventional Techniques 583
10.7 Summary 594 Index 597
Linux has no shortage of tools. Many are inherited from Unix, with cryptic two-letter names that conjure up images of developers trying to preserve space on a punch card. Happily, those days are long gone, but the legacy remains.
Many of those old tools are still quite useful. Most are highly specialized. Each may do only one thing but does it very well. Highly specialized tools often have many options that can make them intimidating to use. Consider the first time you used grepand learned what a regular expression was. Perhaps you haven't mastered regular expression syntax yet (don't worry; no one else has, either). That's not important, because you don't need to be a master of regular expressions to put grep to good use.
If there's one thing that I hope you learn from this book, it's that there are many tools out there that you can use without having to master them. You don't need to invest an enormous amount of time reading manuals before you can be productive. I hope you will discover new tools that you may not have been familiar with. Some of the tools this book looks at are quite old and some are new. All of them are useful. As you learn more about each tool, you will find more uses for it.
I use the term tool loosely in this book. To me, creating tools is as important as using tools, so I have included various APIs that are not usually covered in much detail in other books. In addition, this book provides some background on the internal workings of the Linux kernel that are necessary to understand what some tools are trying to tell you. I present a unique perspective on the kernel: the user's point of view. You will find enough information to allow you to understand the ground rules that the kernel sets for every process, and I promise you will not have to read a single line of kernel source code.
What you will not find in this book is reconstituted man pages or other documentation stitched into the text. The GNU and Linux developers have done a great job of documenting their work, but that documentation can be hard to find for the inexperienced user. Rather than reprint documentation that will be out of date by the time you read this, I show you some ingenious ways to find the most up-to-date documentation.
GNU/Linux documentation is abundant, but it's not always easy to read. You can read a 10,000-word document for a tool and still not have a clue what the tool does or how to use it. This is where I have tried to fill in the missing pieces. I have tried to explain not just how to use each tool, but also why you would want to use it. Wherever possible, I have provided simple, brief examples that you can type and modify yourself to enhance your understanding of the tools and Linux itself.
What all the tools in this book have in common is that they are available at no cost. Most come with standard Linux distributions, and for those that may not, I have included URLs so that you can download them yourself.
As much as possible, I tried to keep the material interesting and fun.
This book is written for intermediate to advanced Linux programmers who wish to become more productive and gain a better understanding of the Linux programming environment. If you're an experienced Windows programmer who feels like a fish out of water in the Linux environment, then this book is for you, too.
Non-programmers should also find this book useful because many of the tools and topics I cover have applications beyond programming. If you are a system administrator, or just a Linux enthusiast, then there's something for you in this book, too.
I wrote this book as a follow-up to an article I wrote for the Linux Journal entitled "Ten Commands Every Linux Developer Should Know." The inspiration for this article came from my own experience as a Linux programmer. In my daily work I make it a point to invest some of my time in learning something new, even if it means a temporary lull in progress on my project. Invariably this strategy has paid off. I have always been amazed at how many times I learned about a tool or feature that I concluded would not be useful, only to find a use for it shortly afterward. This has always been a powerful motivation for me to keep learning. I hope that by reading this book, you will follow my example and enhance your skills on a regular basis.
It's also just plain fun to learn about this stuff. If you are like me, you enjoy working with Linux. Motivating yourself to learn more has never been a problem. Because Linux is open source, you have the opportunity to understand all of its inner workings, which is not possible with closed source environments like Windows. In this book I present several freely available resources available to help you learn more.
The chapters are presented such that each chapter can stand on its own. Later chapters require some background knowledge that is presented in the earlier chapters. Wherever possible, I have cross-referenced the material to help you find the necessary background information.
I believe the best way to learn is by example, so I have tried to provide simple examples wherever possible. I encourage the reader to try the examples and experiment.
Chapter 1, Downloading and Installing Open Source Tools, covers the mechanisms used to distribute open source code. I discuss the various package formats used by different distributions and the advantages and disadvantages of each. I present several tools used to maintain packages and how to use them.
Chapter 2, Building from Source, covers the basics of building an open source project. I present some of the tools used to build software and alternatives that are emerging. There are several tips and tricks in this chapter that you can use to master your use of make. I also show you how to configure projects that are distributed with GNU's autoconf tools so that you can customize them to meet your needs. Finally, I cover the stages of the build that are often misunderstood by many programmers. I look at some of the errors and warnings you are likely to encounter and how to interpret them.
Chapter 3, Finding Help, looks at the various documentation formats tucked away in your Linux distribution that you may not know about. I look at the tools used to read these formats and discuss effective ways to use them.
Chapter 4, Editing and Maintaining Source Files, discusses the various text editors available for programmers as well as the advantages and disadvantages of each. I present a set of features that every programmer should look for in an editor and measure each editor against these. This chapter also covers the basics of revision control, which is vital for software project management.
Chapter 5, What Every Developer Should Know about the Kernel, looks at the kernel from a user's perspective. In this chapter you will find the necessary background information required to understand the workings of a Linux system. I introduce several tools that allow you to see how your code interacts with the kernel.
Chapter 6, Understanding Processes, focuses on processes, their characteristics, and how to manage them. I cover a good deal of background required to introduce the tools in this chapter and understand why they are useful. In addition, this chapter introduces several programming APIs that you can use to create your own tools.
Chapter 7, Communication Between Processes, introduces the concepts behind inter-process communication (IPC). This chapter contains mostly background information required for Chapter 8. Along with each IPC mechanism, I introduce the APIs required to use it along with a working example.
Chapter 8, Debugging IPC with Shell Commands, presents several tools available to debug applications that use IPC. It builds on the information from Chapter 7 to help you interpret the output of these tools, which can be difficult to understand.
Chapter 9, Performance Tuning, introduces tools to measure the performance of your system as well as the performance of individual applications. I present several examples to illustrate how programming can impact performance. I also discuss some of the performance issues that are unique to multi-core processors.
Chapter 10, Debugging, presents several tools and techniques that you can use to debug applications. I look at some open source memory debugging tools including Valgrind and Electric Fence. I also take an in-depth look at the capabilities of gdb, and how to use it effectively.
Anonymous
Posted March 16, 2007
Fusco gathers into one easy to read book the many open source tools available under linux. Cumulatively written by hundreds (if not thousands) of contributors. You might pause a moment when reading the text, to reflect on the amazing amount of code that is freely available under linux. The tools are meant for two types of readers. The system administrator. The programmer. For the sysadmin, there are tools for install packages. Very practical, since updated packages often have bug fixes or new functionalities. This includes rpm, which is used by Red Hat, Suse and other linux distributions. But dpkg is also explained. This is used by Debian and Ubuntu. For the programmer, tools include ways to share memory between processes, as well as communicating between them via semaphores or message queues. The book reflects a general approach taken by Fusco. Tools are described across the popular linux distributions. The book can be used by you, regardless of which distribution you favour. There is a stylistic difference between the material in the book and a corresponding text on Microsoft's offerings. The latter often has a rich graphical framework, like Visual Studio. In contrast, significant portions of this book refer to tools used at the command line. Reflecting linux's heritage in unix.
Was this review helpful? Yes NoThank you for your feedback. Report this reviewThank you, this review has been flagged.
Overview
Master the Linux Tools That Will Make You a More Productive, Effective Programmer
The Linux Programmer's Toolbox helps you tap into the vast collection of open source tools available for GNU/Linux. Author John Fusco systematically describes the most useful tools available on most GNU/Linux distributions using concise examples that you can easily modify to meet your needs.
You'll start by learning the basics of downloading, building, and ...