Public Service Performance: Perspectives on Measurement and Management

Cambridge University Press
978-0-521-85991-2 - Public Service Performance - Perspectives on Measurement and Management - Edited by George A. Boyne, Kenneth J. Meier, Laurence J. O’Toole Jr. and Richard M. Walker
Excerpt

1     Introduction

George A. Boyne, Kenneth J. Meier, Laurence J. O’Toole Jr. and Richard M. Walker

Introduction

The performance of public organizations around the globe is constantly under scrutiny by a variety of stakeholders including politicians, citizens, service users and government inspectors. In the UK, the Labour administration has placed public service improvement at the centre of its domestic agenda and recent events such as the major terrorist attacks of 9/11 in the US, the response of public agencies to Hurricane Katrina and the SARS outbreak in the Far East clearly demonstrate the pressure on public organizations to perform well. Knowledge of routes to higher levels of performance in public organizations is, therefore, of critical importance.

   However, there is only limited evidence on the determinants of performance in public organizations (Boyne 2003; O’Toole and Meier 1999). A range of important questions persist about the performance of public bodies. This edited volume seeks to contribute new knowledge on the issues of performance measurement and management in public organizations by focusing upon three questions:

• What approaches should be adopted tomeasure the performance of public agencies?
• What aspects of management influence the performance of public agencies?
• As the world globalizes, what are the key international issues in performance measurement and management?

   In focusing upon these fundamental questions, the contributors to this book debate methodological and technical issues in the measurement of performance in public organizations and provide empirical analyses of the determinants of performance. The book also provides some important groundbreaking work by considering the international dimensions of these issues. Prior to considering these issues in more detail, and discussing the chapter contributions, we describe the meeting from which the papers presented here emerged.

The determinants of performance in public organizations seminar

In May 2004 the editors of this book organized a major international seminar on the ‘Determinants of Performance in Public Organizations’ at Cardiff University. The purposes of the seminar were to bring together scholars working at the leading edge of research on the performance of public organizations, give greater prominence to this area of academic inquiry, and delineate an agenda for further research. The conference participants are collectively mapping the research frontier on the determinants of organizational performance – especially on the role of management in public service improvement. The publication of this book completes the process of publishing all the papers presented at the seminar. Seven of the papers can be found in a symposium edition of the Journal of Public Management Research and Theory edited by Boyne and Walker (2005) and published in October 2005.¹

   The seminar was sponsored by the Advanced Institute for Management Research (www.aimresearch.org) and the Public Management Research Association (www.pmranet.org). The Advanced Institute for Management Research (AIM) is a £17 million flagship initiative that was launched in 2003 by the UK’s Economic and Social Research Council (ESRC) and the Engineering and Physical Sciences Research Council (EPSRC). The AIM seeks to build capacity for effective management research that addresses academic, business, public service and policy audiences. The AIM has four objectives, to:

• conduct research that will identify actions to enhance the UK’s international competitiveness;
• raise the scientific quality and international standing of UK research on management;
• expand the size and capacity of the active base for UK research on management;
• develop the engagement of that capacity with world-class research outside the UK and with practitioners as co-producers of knowledge about management and other users of research within the UK.

   The public services formed one theme within the AIM rubric. A dozen AIM research fellows were appointed to work on the public services. These fellows, as a group, have undertaken systematic reviews of existing work and new theoretical and empirical research. They have also laid out research agendas in their areas of work and identified capacity building requirements in the UK social science community. This book is edited by two of those Fellows (Boyne and Walker) and two AIM International Visiting Fellows (Meier and O’Toole). George A. Boyne’s work focused on public service failure and turnaround and Richard M. Walker examined the relationship between innovation and organizational performance. As International Visiting Fellows, Kenneth J. Meier and Laurence J. O’Toole brought their expertise on public management and performance and research methodologies to a UK audience.

   The AIM seminar at Cardiff University was co-sponsored by the Public Management Research Association (PMRA) which is a non-profit academic membership association that has grown out of the bi-annual series of Public Management Research Conferences based in the US. PMRA seeks to further research on public organizations and their management and serve as a voice for the public management research community. The event marked PMRA’s first foray overseas as it seeks to internationalize its networks and membership, and to nurture theory building and systematic testing of theory consistent with the canons of social science, using the full range of quantitative and qualitative methodologies in public management.

   Fifteen papers were delivered over the two-day event that was held from 6 to 8 May 2004 in the Glamorgan Building at Cardiff University. The papers, listed alphabetically, were:

   Rhys Andrews (Cardiff University), George A. Boyne (Cardiff University), Kenneth J. Meier (Texas A&M University and Cardiff University), Laurence J. O’Toole Jr. (University of Georgia) and Richard M. Walker (University of Hong Kong and Cardiff University). Diversity and organizational performance: An empirical analysis

   Gene A. Brewer (University of Georgia). In the eye of the storm: Frontline supervisors and federal agency performance

   Yousek Choi and Carolyn J. Heinrich (University of Wisconsin-Madison). Privatization and performance-based contracting in public welfare programs: The challenge of promoting accountable administration

   Young Han Chun (Yonsei University) and Hal G. Rainey (University of Georgia). Consequences of goal ambiguity in public organizations

   Rachael Addicott and Ewan Ferlie (University of London – Royal Holloway). Determinants of performance in cancer networks – A process evaluation

   Melissa Forbes and Laurence E. Lynn Jr. (Texas A&M University). Studying governance: Are the United States and the rest of the world in step?

   Katharina Hauck, Nigel Rice, Peter C. Smith and Andrew Street (University of York). Explaining variations in health authority performance: A multivariate hierarchical modelling approach

   Carolyn J. Heinrich (University of Wisconsin-Madison) and Carolyn J. Hill (Georgetown University). How does governance influence substance abuse treatment strategies? State policies and naltrexone adoption

   Gregory Hill (Texas A&M University). Long-term effects of managerial succession: An application of the Boyne/Dahya model

   Graeme A Hodge (Monash University) and Anne Rouse (University of Melbourne). Outsourcing government information technology services: An Australian case study

   Patrick Kenis (Tilburg University). Control as a determinant of performance in public organizations

   Mary O’Mahony (National Institute of Economic and Social Research). Outcome based measures in international comparisons of public service provision

   Sanjay K. Pandey (Rutgers University), David H. Coursey (Florida State University), and Donald P. Moynihan (Texas A&M University). Management capacity and organizational performance: Can organizational culture trump bureaucratic red tape?

   David W. Pitts (University of Georgia). Diversity, representation and performance: Evidence about race and ethnicity in public organizations

   Keith G. Provan (University of Arizona), Kimberley Roussin Isett (Texas A&M University) and H. Brinton Milward (University of Arizona). Cooperation and compromise: A network response to conflicting institutional pressures in community mental health

   The seminar was also attended by other academics and practitioners, including H. George Frederickson (University of Kansas) and M. Jae Moon (who also contributes to this volume with his co-authors) and Derek Egan (Office of the Deputy Prime Minister, England). The formal and informal discussions at the seminar provided an important opportunity for this mix of established and more junior academics to develop new directions in public management research. Much of that thinking is reflected in the chapters in this book, and further avenues yet to be explored are identified in the concluding chapter.

Measuring and assessing performance in public agencies

This book deals with qualitative and quantitative approaches to the measurement of public service performance. Longstanding interest has been expressed in the measurement of performance in public agencies (see Ostrom 1973; Park 1984; Kelly and Swindell 2002) and the most appropriate way to analyse such data (Heinrich and Lynn 2001; Martin and Smith 2005; Gill and Meier 2000).

   Empirical studies of the performance of public organizations have been limited. This may reflect traditional concerns amongst public administration scholars with organizational processes rather than outputs and outcomes. In addition, many studies of ‘policy outputs’ have taken costs or resource utilization as the dependent variable (Boyne 1996; see also Smith, Chapter 5). Though questions about expenditure are important for public management scholars, the focus of practitioners (and, more recently, researchers) has shifted along the public service production line from financial inputs to service delivery and performance.

   This change of emphasis has opened up important new agendas for public management scholars. The performance of public organizations is contested and multidimensional; for example, Boyne’s (2002) review of dimensions of performance included measures of output quantity, output quality, efficiency, effectiveness, accountability, equity, probity, democracy and impact. Although many such classifications exist, little clarity has been offered on the most appropriate ways to measure organizational performance, or how to analyse performance data. These questions are of immediate academic and practical concern. As governments around the world rate the performance of their public agencies, be it cities in China, federal agencies in the US, local governments and health services in the UK or the Putting Service First Scheme in Australia, the types of data used and the way that data is manipulated become important issues. The practical consequences of these regimes are not inconsiderable. In the UK, for example the regimes that rate hospitals or local authorities have major consequences for future resources flows and the autonomy of these agencies. Work by Andrews et al. (2005) in England has shown how such regimes do not take account of the context within which organizations operate and thereby penalize those working in difficult conditions.

   These performance management regimes often rely upon scorecards as the means to communicate oversight agencies’ assessments to citizens. Although such summaries of performance (often in league tables) may be readily interpreted by the layperson, the construction of scorecards is complex involving different types of assessment of performance, and uses a mixture of qualitative and quantitative techniques (see Weimer and Gormley, 1999 and Andrews et al., 2006 for a discussion). The use of performance data from different sources has been an issue of longstanding debate in public management (see Park 1984). The central question posed here is: what is the best source of performance data? Two types of data are typically presented, perceptual or ‘subjective’ data and archival or ‘objective’ data. In public management the majority of the debate has been conceptual, with little empirical evidence presented on the relative validity and reliability of these types of performance data (though see for example, Kelly and Swindell 2002).

   Important questions about the hotly-debated merits of subjective and objective evaluations of performance are tackled in Chapters 2 and 3. In Chapter 2, entitled ‘Subjective and Objective Measures of Organizational Performance: An Empirical Exploration’, Rhys Andrews, George A. Boyne and Richard M. Walker assess the relative merits of different methods of measuring performance. The questions they pose include: What types of measures should be used? Are the different types of measure valid? Who is best placed to make assessments – stakeholders in the organization or those outside it? The authors analyse a new dataset from Welsh local government which includes measures of internal and external perceptions of performance. Their analysis reaffirms that organizational performance in the public sector is complex and multidimensional. Its complexity relates to the number of dimensions of performance and the number of stakeholders. The performance of public organizations cannot be reduced to a single dimension, and is inescapably contestable.

   Chapter 3 by Gene Brewer entitled ‘All Measures of Performance are Subjective: More Evidence from US Federal Agencies’ develops this theme. Using data from the 2000 Merit Principles Survey, US Merit Systems Protection Board, the chapter examines the role of frontline supervisors in the twenty-two largest Federal agencies, and explores their contributions to organizational performance and effectiveness (see Brewer 2005 for a full discussion of these results). Attention in this chapter is focused on the validity, reliability, and sensitivity of perceptual measures of organizational performance, as reported by civil servants. To explore issues of common-source bias, Brewer splits the sample: one part of the organization assesses management, the other performance. The results suggest that common-source bias is perhaps more limited than is sometimes claimed.

   Performance measures are derived from a variety of sources including inspections, user and citizen satisfaction surveys and archival data. The performance assessments of oversight agencies are often qualitative in form, based on observations and interviews. Little consideration has been given to the use of qualitative data in the literature on performance in public agencies. Chapter 4 by Rachael Addicott and Ewan Ferlie, entitled ‘A Qualitative Evaluation of Public Sector Organizations: Assessing Organizational Performance in Healthcare’ demonstrates how qualitative data can be used in studies of organizational performance. Qualitative methods are argued to capture the experiences of users, contribute towards positive social and organizational change though action research and offer insights into why things work. Their approach is to track organizational processes through longitudinal and comparative case studies. This methodology is applied to a complex managed cancer network in the UK’s National Health Service. Methods utilized include a Delphi panel, interviews, documentary analysis and observation. The example shows how these techniques can be used to construct assessments of organizational performance.

   In Chapter 5, entitled ‘Quantitative Approaches towards Assessing Organizational Performance’, Peter Smith provides a comprehensive critique of the application of quantitative techniques to the study of the performance of public organizations. He reviews techniques for the analysis of one performance indicator and studies that use multiple performance measures in a single model. While providing a comprehensive overview of the application of these techniques, Smith is keen to point out that for most public organizations service delivery or programme evaluation is reliant on multiple measures of performance. The case is made for the use of seemingly unrelated regressions in future studies of the determinants of performance in public organizations for two reasons. First, the technique does not suffer from some of the technical problems associated with Stochastic Frontier Analysis or Data Envelopment Analysis. Second, it moves beyond the typical piecemeal modeling of individual performance indicators and does not place excessive demands on data or modelling methodology. Smith concludes by noting the need to use other research methods alongside quantitative techniques.

The performance consequences of management

Empirical studies of the impact of management on the performance of public organizations are scarce (Boyne 2003, 2004). Prior wisdom suggests that the actions of public servants are constrained by the rule of law and the external environment within which they operate. Over recent years a small number of researchers (many of whom make contributions to this book) have demonstrated that a range of management variables influence the performance of public organizations, including innovation, leadership, managerial quality, and strategy. In this section of the book a number of major public management trends are explored. The question of the nature of goals in public organizations has long been debated (Rainey 2003). A central theme of this debate is about the clarity of goals in public organizations, and the consequences that arise from opaque goals. Linked to this are questions about control mechanisms within and without public agencies. A key argument in the public management literature is that vague goals are compounded by poor control arrangements, both internal and external. In recent decades there has been a growth of network forms of governance and a spread of contracts with external suppliers of public services. Networks are likely to enhance problems of goal ambiguity and control while contracting, theoretically, should reduce these problems. The chapters in this section of the book explore the performance consequences of these management arrangements.

   Chapter 6 addresses an issue of central importance to the management of public organizations: goal ambiguity. Young Han Chun and Hal G. Rainey build on the important work (2005a; 2005b) that has been exploring the determinants and performance impacts of goal ambiguity, taking the case of thirty-two US federal agencies and using their published strategic plans and performance reports as statements of their goals. In this chapter they move on to examine the ‘Consequences of Goal Ambiguity in Public Organizations’. They study the relationship between three aspects of goal ambiguity (directive, evaluative and priority) and measures of red tape, decentralization, reward expectancy, and job satisfaction. The findings support the value of clear goals and objectives for public organizations. In particular Chun and Rainey argue that clear goals can reduce procedural regulations, support more decentralization, and increase levels of reward expectancy and job satisfaction. Nonetheless, tensions between the need for managerial clarity and political opaqueness are recognized as a key issue in public management.

   In Chapter 7, Patrick Kenis discusses ‘Performance Control and Public Organizations’. The chapter reviews the concept of performance control, which is a cybernetic process of setting objectives, measuring performance, and feeding back information. Two extreme examples of performance control are examined: management performance control and agency control. In relation to each part of the cybernetic process, Kenis offers propositions on the likely relationship between performance management control and agency control. The review concludes that there is no one best form of performance control and that prescriptions in this area are likely to be difficult to identify.

   Red tape, one of the concerns of Chun and Rainey, is salient to public organizations around the world. Frequently, public organizations are criticized for procedural regulations that adversely affect their performance, and governments increasingly seek to reduce levels of red tape. New public management and reinvention sought to enhance organizational flexibility, but little academic work has examined the performance consequences of red tape. In Chapter 8, Sanjay K. Pandey and Donald P. Moynihan examine ‘Bureaucratic Red Tape and Organizational Performance: Testing the Moderating Role of Culture and Political Support’. The authors use data collected from managers in USstate health and human service agencies as part of the National Administrative Studies Project. Pandy and Moynihan conclude that red tape does have a negative impact on organizational performance. However, their results show that if levels of developmental culture and political support are high then an increase in red tape is associated with improved performance, suggesting that organizations adapt in different ways to similar constraints. Thus red tape per se is not bad; rather, its impact depends on other characteristics of public organizations.

   In Chapter 9, Kenneth J. Meier, Laurence J. O’Toole Jr. and Yi Lu focus on networks and organizational performance. In their chapter entitled ‘All That Glitters Is Not Gold: Disaggregating Networks and the Impact on Performance’, they note that much theoretical discussion has been presented about the ways that many public programmes are implemented in complex interorganizational networks of public, private and non-profit sector units, and how managers create, sustain, and nourish such networks. In recent years a growing body of empirical research has analysed the impact of network arrangements on public programme performance. Meier, O’Toole and Li contend that largely omitted in the discussion of networks to date has been attention to the possibility that network contact and network interaction might not be beneficial to the performance of the organization at the network’s center. This chapter addresses both the positive and negative forces that network nodes bring to bear on the policy performance of a core organization. The analysis focuses on 500 US school districts over a three-year period. Although earlier research has demonstrated that managerial networking has a consistent positive impact on organizational performance, this study disaggregates the network into its component parts to demonstrate that the interaction with some nodes reduces performance. Meier, O’Toole and Lu provide further evidence to validate the management matters argument, however they also suggest that what matters is perhaps more complex that hitherto demonstrated.

   Keith G. Provan, Hendry Brinton Milward and Kimberley Roussin Isett examine the evolution and performance of a network of community-based, non-profit, health and human service agencies providing publicly funded services to adults with serious mental illness in Chapter 10. In ‘Network Evolution and Performance Under Public Contracting’, data from Tucson/Pima County in Arizona is examined from two points in time; first, shortly after the introduction of a new system of funding by the state, based on financial risk under managed care; and then, four years later, after the system had matured. Despite concerns that the new system would increase competition and force agencies to emphasize cost control at the expense of services, their findings indicate that performance did not suffer and that collaboration among key provider agencies increased substantially. The authors draw conclusions concerning the impact of risk-based public contracting on network performance and evolution.

   Chapter 11, ‘The Design and Management of Performance-based Contracts for Public Welfare Services’ sees Youseok Choi and Carolyn J. Heinrich explore the pioneering Wisconsin Work (W-2) programme of public welfare reform. This changed the administrative structure for public welfare services delivery from county government administration to one that allows for private sector management of programmes and performance-based contracting. The authors ask some fundamental questions about contract design and administration. They provide a detailed longitudinal case study of the Wisconsin Works programme. This chapter demonstrates the complexities that public agencies are likely to face in service contracting, one of which is the incomplete nature of contracting arrangements. The primary difficulties experienced were renegotiation and problems of contract administration and management. Choi and Heinrich argue that if public sector contracting is dominated by economic assumptions of completeness and efficiency then the results are likely to be inadequate.

   The complexities and uncertainties of contracting are the theme of Chapter 12 where Graeme A. Hodge and Anne C. Rouse examine ‘Outsourcing Government Information Technology Services: An Australian Case Study’. They examine the policy promises made when outsourcing IT services, and review the range of global evidence to date on the effectiveness of this technique. The chapter provides a detailed analysis of the outsourcing experience of several major federal government operations in Australia. The authors contrast the empirical experience of a US$1billion contracting deal with the promises made. Their central finding is that the modest 15 per cent saving promised initially was largely not delivered. The reasons for this are analysed and contrasted against the common assumptions made in determining successful outsourcing performance. Finally, the chapter draws together a series of general lessons on the outsourcing of IT at a time when governments are increasingly intent on adopting private means for providing public sector services and infrastructure. The main lesson is that IT outsourcing is more complex and risky than has hitherto been acknowledged.

Global questions in measurement, management and performance

This book takes some of the detailed questions examined earlier and locates them on the international stage. Chapters 13 and 14 by O’Mahony and Stevens and Welch, Moon and Wong consider issues of measurement, while Forbes, Hill and Lynn examine the international evidence on management and the governance in Chapter 15.

   Chapter 13: ‘International Comparisons of Output and Productivity in Public Service Provision: A Review’ by Mary O’Mahony and Philip Stevens considers the feasibility of international comparisons of public service provision at the aggregate sector level, taking the examples of health and education. The authors make a persuasive case for measuring public service output and productivity in ways that mirror the methods used in the market sector. The body of the chapter deals with how output and productivity measurement can be achieved across international borders, with an application to the education sector. The core of their argument is that private and public services are comparable because they both seek to meet consumer demand, and public preferences should, therefore, dictate how services are evaluated. As we noted in the earlier discussion, such studies are now possible because of data availability, which in turn means that, as O’Mahony and Stevens conclude, it will become increasingly possible to extend the types of models presented in this chapter to examine more complex public management relationships in other contexts.

   Melissa Forbes, Carolyn J. Hill and Laurence E. Lynn Jr. turn the readers’ attention to the international research literature on management and government performance. In Chapter 14, ‘Public Management and Government Performance: An International Review’ Forbes, Hill and Lynn apply the ‘logic of governance’ (Lynn et al. 2001) analytical framework to a database of nearly 1,000 articles from over fifty countries. The chapter offers a synthesis of the prior studies by Forbes and Lynn (2005) and Hill and Lynn (2005) on ‘how researchers understand what works and how it works’. On methodology, the authors note clear variations between US and non-USresearch, with the latter more likely to favor ‘linear’ managerialist hypotheses. They speculate that this may reflect the federal, diffuse and polycentric nature of the US system, in contrast to unitary or statist regimes. The most substantive finding of their study is that hierarchy continues as the backbone of governance, raising questions about the extent of the recent emphasis upon networked or horizontal governance in the public management literature and in the rhetoric of public and political debates on the nature of public service delivery.

---

© Cambridge University Press