Psychological Investigations of Competence in Decision Making

Read an Excerpt

Introduction: What Does It Mean to Be Competent?

Kip Smith, James Shanteau, and Paul Johnson

This book departs from the more traditional topics of judgment and decision-making research. It emphasizes neither the deficiencies of human cognition, nor heuristics and biases, nor behavior in carrying out tasks to which no one is particularly well adapted. Rather, it introduces two task-general sources of competent decision making in a wide variety of professional domains. In this introductory note, we summarize the genesis of this project and define what we mean by competence.

Recent summaries of work on expertise have focused on the measurement of expert (optimal) performance, its replication in laboratory settings, the mediating mechanisms for such performance, and the role of deliberate practice in achieving it. Clear examples of expertise in this sense occur in games, in athletic and musical performance, and in certain types of work.

The premise of this book is that much activity in everyday life and work is not of this sort. Many of the situations we encounter are novel, infrequent in our experience, or variable with respect to presenting conditions and the action to be taken. Such tasks require decisions to be made and actions taken in the face of ambiguous and/or incomplete information. Time pressure is frequently great, and the penalties for failure are often severe.

Examples of such situations include investing in a market, controlling an industrial accident, and detecting fraud. These are all environments that defy a definition of optimal performance. Practice may be beneficial but is unlikely to be the sole foundation for skilled performance. Indeed, the idea of optimal performance often does not apply, yet the benefits of successful decision making are considerable. Typically, in these domains there are also individuals who perform better than others.

In these and other, less dramatic situations like weather forecasting and probability estimation, individuals may use knowledge that is quite different from that of someone whose work is based on practice in achieving a known or computable standard (e.g., scheduling airline reservations). We refer to domains without criteria for optimal performance as competency-based, and we describe the behavior of individuals who work in them by the term competence.

In competency-based domains, we expect the mechanisms that govern actions taken to achieve goals to invoke task-general cognitive processes that redefine the role of the agent or the task constraints. For example, representational structures may be used to redefine the task or to distribute task components so that the agent is no longer taxed beyond her capacity. Alternatively, the task at hand may be recast as an instance of a broad class of tasks for which evolution has provided an adaptive metacognitive process.

We identify a pair of metacognitive processes that give structure to otherwise ill-structured tasks. The first we call metacognition-self, the second metacognition-others. Metacognition is thinking about the kind of thinking that a task requires. Metacognition-self is an introspective reevaluation of ongoing or planned cognitive activity and behavior. By taking this internal stepping-back, an agent may put herself within the situation and become able to identify herself as a source of task constraint. This task-general process may lead the agent to recast her role or to redefine the task, which may, in turn, facilitate performance. The chapter by Dominguez et al. illustrates the power of using metacognition-self in the operating room. Pliske et al. discuss how metacognition-self provides the flexibility needed when making a weather forecast.

The second metacognitive process is metacognition-others, thinking about others' thinking or, at the least, thinking about how others ought to be thinking. By jumping into others' minds, by taking a normative stance about how they should be processing information, an agent may be able to predict their thoughts, decision making, and behavior. This task-general process may lead the agent to update her goals or refocus her attention to critical cues, which may, in turn, facilitate performance. The chapter by Skriver et al. presents a study of a daunting situation in which metacognition-others saved many lives. Jones shows how metacognition-others is a key element in the successful design of human-computer systems. Grazioli et al. present empirical work in pair a of domains, fraud detection and spot currency trading, in which metacognition-others is the sole basis for agent differentiation and superior performance.

In addition to addressing the two central competence enablers - metacognition-self and metacognition-others - each chapter in this book enriches our understanding of the basis of skill and of success in the performance of decision-making tasks generally and in specific domains of work and society.

Jones and Pliske et al. discuss how addressing domain-specific constraints is a cornerstone of competent performance. The domains are quite different, but the approach is similar. Grazioli et al. and Kurz et al. identify sources of power in performance. The sources are neither processing speed, nor precision, nor the ability to remember large amounts of information. Rather, all four chapters illustrate that competency is an issue of adaptation and fit to task demands. As Simon argued in Sciences of the Artificial, once adapted, the agent simply does what the task requires. If we wish to understand the basis for an agent's success, we need to understand the structure of the task in which the behavior occurs. We must understand what the invariants are and how successful behavior is explained in terms of them.

A related issue is the problem of generativity. The chapters by Hardman and Ayton, Skriver et al., and Dominguez et al. address how far performance on familiar tasks can be extended when dealing with novelty. Task-general cognitive processes that redefine the role of the agent or the constraints of the task prevent performance from deteriorating dramatically as one moves away from the normal day-to-day routine.

The final three chapters in this volume address a more traditional competence enabler - representation that fits the demands of the task and the bounded rationality of the decision maker. Hardman and Ayton argue that argumentation provides fitting representations that support competent decision making under ambiguity and ignorance. In addition, argumentation explains several systematic deviations from the prescriptions of expected utility theory. The chapter by Kurz et al. addresses a topic that has been a mainstay of research in judgment and decision making: the representation of probabilistic information. They offer alternative accounts for procedures that reformulate the task and that markedly improve performance. Weiss and Shanteau address the issue of evaluating competence in domains in which not all agents working on a given task behave alike. Presumably, the learning history of each individual results in a specific adaptation to the task. However, each individual's adaptation results in unique performance. The authors show how we can understand competence in the face of variability.

As the editors of this book, our goal has been to open a new direction for judgment and decision-making research. Academic research generally and our society particularly have largely neglected the fact that sound judgment and decision making are the crux of many professions. By understanding and communicating what professional decision makers do and how they do it well, we make valuable contributions both to our field and to the professional community at large.

Part I

Metacognition-Self

1 The Conversion Decision in Laparoscopic Surgery: Knowing Your Limits and Limiting Your Risks

Cynthia O. Dominguez, John M. Flach, Patricia L. McDermott, Daniel M. McKellar, and Margaret Dunn

Not in the past 100 years has such an upheaval in medicine occurred: The " discipline of surgery" is joining the technologic revolution and advancing the state of the art with laparoscopic surgery. This represents a radical shift in the concept of surgical practice. The "great leap of faith" has occurred; for the first time in history, surgeons are performing surgical procedures without physically seeing or touching the organs they are removing or repairing. (Satava, 1993, p. 111)

Day after day, all around the world, patients are wheeled into operating rooms to undergo procedures that they hope will restore them to better health. Over the past 20 years, and especially in the past decade, innovations in surgical technology and accompanying techniques have led to a reduced level of access trauma (damage to healthy tissue from the incision) for patients. Smaller incisions are made possible through the use of tubular fiberoptic cameras. Images of the operative area are displayed to surgeons on an external TV-like monitor, and the patients' internal structures are manipulated with long-stemmed instruments. As a group, these advances are called minimally invasive surgery and the procedure itself laparoscopy.

Although patients and insurance companies are generally thrilled about reduced hospital stays, quicker recoveries, and the greater convenience of minimally invasive procedures, there is a cost. Minimally invasive surgery introduces significant challenges to surgeons. The new surgeon-patient interface adds a barrier between the surgeon and the work environment so that perceptual information surgeons need is degraded, and the motor skills required are more technically demanding (Cuschieri, 1995). When a surgical procedure is especially challenging, involving a patient with unusual anatomy and/or acute inflammation of tissues, persisting with a minimally invasive approach may increase the risk of major injury to a nearby structure. The surgeon has to evaluate the benefits and risks between continuing laparoscopically and converting to an open procedure. This ongoing evaluation is the conversion decision, to open or not to open.

General surgeons identified this evaluation as a critical decision that could be examined and better understood through cognitive field research. Benefits of opening include increased exposure, direct view and feel of the operative area, and often time efficiency whereby the operation can be done more quickly. Generally, 4% to 5% of laparoscopic cho- lecystectomy (gallbladder removal) cases are converted to open procedures (Schrenk & Woisetschlager, 1995; Southern Surgeons Club, 1991). This conversion decision, to open or not to open, was the focus of our research. However, it is important to understand that this "decision" is not typically presented to the surgeon as a distinct event, but happens over time in the context of a dynamically evolving situation.

The complexity of this decision cannot be overstated. It is not made by evaluating a static set of alternatives at just one point in time; on the contrary, it is extended in time, and it involves the integration of changing goals and information from many sources. Assessing the risk of unintended injury involves knowledge of one's own capabilities and those of other members of the surgical team. Further, it is clear that there is no consensus upon which to establish a normative or "right" decision. Twenty surgeons might describe 20 differing courses of thought and action in projecting how they would act in the best interests of a specific patient. Also, the outcome of the procedure is not a reliable measure of decision quality. In many cases, satisfactory outcomes follow despite questionable decisions, and occasionally negative outcomes may result despite reasonable choices. The current standards recommend a conservative approach, whereby surgeons convert to an open procedure whenever complications arise. However, there is great concern that a surgeon on any given case might not follow this standard - that a surgeon might persist in using minimally invasive procedures to the point where patient safety becomes compromised (Greene, 1995).

In this research, we used an exploratory approach to look at expertise in a commonly performed minimally invasive surgical procedure, laparoscopic cholecystectomy. Goals of this research were (1) to understand the decision to open during laparoscopic surgery and ultimately to develop a training intervention based on that understanding; (2) to understand perceptual expertise, including resident-staff differences and how both groups stay within the boundaries of safe performance; and (3) to understand how metacognition interacts with expertise in this context. In this chapter, we will first discuss laparoscopic surgery in general, in the context of a particular challenging case, and in terms of analogous concepts that help us see relationships between surgery and other domains involving similar risks and complexities. Next, we will explain methods used in this research and findings related to metacognition. Finally, we will review other current writings and research pertaining to metacognition and suggest a new conceptualization that incorporates the findings of this research.

Laparoscopic Surgery: Challenges and a Case Study

Clearly, surgery is a profession well suited to the observation and study of risk, decision making, and various aspects of expertise. Surgeons must bring extensive medical knowledge, perceptual-motor skill, understanding of tools and their uses, and (last but certainly not least) good judgment to bear when they operate. To study the expertise of surgeons, we have found it useful to focus on a particular case, gallbladder removal in an 80-year-old woman. Gallbladder removal is a common bread-and-butter type of operation for general surgeons. This woman's case was chosen because it was a difficult one in which there was a wide range of opinion as to the appropriate course of surgical action. We used the background information and videotape from this patient's laparoscopic procedure to examine expertise in laparoscopic surgery and the role of metacognition in that expertise.

The Case

It was clear that this woman had an acutely infected gallbladder; the surgeon noted a palpable mass in the gallbladder's location when he physically examined her. She had a 2-day history of fever, pain in the right upper quadrant of her abdomen, and a high white blood cell count (leukocytosis). The ultrasound exam confirmed that she had a distended gallbladder with a thickened wall and gallstones. Pericholecystic fluid, indicating inflammation of gallbladder tissues, was also noted preoperatively.

Surgeons we interviewed had different opinions as to whether this woman's surgery should begin as an open or a laparoscopic procedure. Some surgeons reasoned that it should be done open because the patient was old; her aged lungs and heart would be less likely to withstand the pressure from insufflation¹ needed in laparoscopy and she certainly didn't need to return to work quickly. Ironically, other surgeons reasoned that the procedure should be done laparoscopically because the patient was old; she was more likely to develop pneumonia after an open procedure, and the bed rest needed to recover from the large incisions might decrease her likelihood of ever returning to full functioning. The conflicting opinions are difficult to reconcile, which is precisely why this case is a good one to study in the tradition of examining critical incidents (Klein, Calderwood, & MacGregor, 1989) to understand expertise and decision making.

Although these trade-offs were discussed by many of the 20 surgeons we interviewed, only 2 of them indicated that they would not begin the procedure laparoscopically; all of the others would "at least take a look" with the laparoscope. The procedure began with making the incisions and inserting the tubular ports through which instruments are inserted in the body. The surgeon first inserted the laparoscope in the umbilical port to survey the anatomy in the operative area. Three other ports were also placed and secured in the same manner, located roughly in a diagonal line from above the navel port toward the right hip. In general, placement of these incisions and ports depends upon where the surgeon believes the patient's biliary anatomy is located and how it can best be accessed. Port placement is quite important, because it may or may not afford proper visualization of the back side of an instrument when structures are clipped or cut.

When the laparoscope brought this patient's gallbladder into view, it was clear that the gallbladder was diseased. It was reddish-purple, with splotches of green and black. The greenish color was referred to as classic dead tissue. The distention of the gallbladder presented a problem. Grasping the gallbladder and retracting it would be impossible without somehow relieving the pressure (removing the gallbladder requires grasping it and pulling it up and out of the way while structures are identified and connective tissues are severed). An accepted method for dealing with this distention is to drain the fluid with a needle.

A strong concern of surgeons was whether the dead tissue on the wall of the gallbladder might cause the gallbladder to break apart, spilling infected bile into the abdomen; just how much tension might tear this tissue apart could be known with certainty only by trial and error. The risk of gallbladder tearing was treated as an acceptable one by some (but not all) surgeons:

I know this is going to be very friable tissue, it's going to fall apart very easily in my forceps and [the procedure] may be very difficult to complete laparoscopically. But this is still one that I would give a fighting chance to, because I know her recovery will be that much more quick, and I can diminish her mortality from things like postoperative lung problems and so on. (Staff surgeon)

The alternative, but minority, viewpoint:

It's going to shred. The gallbladder wall is dying. You're going to find yourself flailing. You're going to pull on the gallbladder to give yourself exposure to the cystic duct, and it's going to tear . . . you have torn the gallbladder, you've exposed their belly to everything the gallbladder has in it, you increase their risk of abdominal infection, increase their risk of a wound infection. The gallbladder is gangrenous; it's probably so adherent to the surrounding tissue that you can easily just cut through something and not even know it, because the surrounding tissues are going to be just that inflamed. And again, the laparoscopic procedure is done to shorten the person's hospital stay. But this person has a sick gallbladder. Their concern is not just getting back to work in six days, this person could die from this disease. Your concern is doing what's best for the patient, not what leaves a minimal scar. (Fifth-year resident surgeon)

Even at this early point in the operation, the conflict about which approach would inflict the least harm on the patient is apparent. The first surgeon felt that laparoscopy was best; the second surgeon felt that the potential harm of laparoscopy for this patient outweighed the problem of scarring, which was treated as a cosmetic issue.