Read an Excerpt
Measuring the Statutory and Regulatory Constraints on DoD AcquisitionResearch Design for an Empirical Study
By Jeffrey A. Drezner Raj Raman Irv Blickstein John Ablard Melissa A. Bradley Brent Eastwood Maria Falvo Dikla Gavrieli Monica Hertzman Darryl Lenhardt Megan McKernan
Rand CorporationCopyright © 2006 RAND Corporation
All right reserved.
Background and Motivation
Commissions and organizations have tried to improve the defense acquisition process for more than 50 years, and studies addressing the topic have fairly consistently identified the same set of problems and proposed the same solutions. In 1986, the Packard Commission recognized that legislative and regulatory constraints on the Department of Defense (DoD) acquisition process and personnel affect the efficiency and effectiveness of the acquisition process. In 1992, DoD created the Office of Acquisition Reform (AR) to be the focal point for identifying changes that could improve both process efficiency and outcomes while maintaining a necessary level of accountability and oversight. Many of the changes proposed either through past studies or AR recommendations have been implemented. For example:
The modifications to DoD acquisition policy and implementation guidance (DoD Directive 5000.1 and Instruction 5000.2) and the associated requirements generation process (CJCSI 3170.01) that occurred in 2003 resulted in a significant restructuring of the acquisition process. Programmatic changes suchas the imposition of the now-defunct program stability wedge, which enhanced the Services' ability to maintain stable funding profiles for programs, and use of the Cost Analysis Improvement Group (CAIG) cost estimate for Acquisition Category (ACAT) I programs were implemented to ensure full funding. Changes in law, such as the Clinger-Cohen Act (changes in the management of information technology) and the Federal Acquisition Streamlining Act (FASA) (changes to the Federal Acquisition Regulations), were recommended and approved by Congress.
Nevertheless, the perceived problems surrounding defense acquisition have changed little over the years. Part of the problem may stem from a "regulatory pendulum" that responds to fraud, waste, and abuse with increased regulation and then swings back in response to complaints of regulatory burden. Laws and regulations became tighter in the late 1980s in response to several highly visible examples of fraud earlier in that decade. The 1990s saw increased acquisition reform activity, mostly targeted at removing those regulatory constraints. Recent examples of abuse will likely result in tighter rules in the near future. For now, however, it is widely believed by the DoD acquisition community (including decisionmakers, program managers, and analysts) that DoD acquisition programs continue to operate under a series of statutory and regulatory constraints that stifle innovation, impair productivity, and result in increased costs and schedule delays.
Costs and Benefits of Statutes and Regulations
Most acquisition officials agree that statutory and regulatory constraints adversely affect program outcomes, but they are nevertheless unable to provide credible estimates of the magnitude of those effects. Past studies have produced widely varying and generally unsatisfying results: It is very difficult to demonstrate the effects of these constraints because compliance is deeply embedded in acquisition processes and institutions, and, generally, the effects cannot be identified separately. Acquisition managers recognize both the statutory and regulatory constraints (though they cannot always distinguish between them) and adjust the resulting inefficient process. Relevant data on the cost of compliance are not collected during the course of routine program execution, and acquisition officials have little basis for making credible estimates of consequences.
The lack of any empirical analysis of the costs (and benefits) of statutory and regulatory compliance makes targeted change difficult, resulting in extreme swings of the regulatory pendulum. In April 2003, DoD submitted to Congress the Defense Transformation for the 21st Century Act, which contained Office of the Secretary of Defense (OSD) proposals for changing personnel management, the acquisition process, and selected administrative and budgetary processes within OSD. The last 88 pages of the 207-page document contain a list of 183 congressionally mandated reporting requirements affecting DoD; following each requirement is a proposal to either repeal or to change it, as well as a short justification for the proposal. Table 1.1 categorizes the main justifications. While Congress eventually enacted some of the legislative proposals, it passed few of the proposals to repeal or change specific reporting requirements, in part because DoD could not demonstrate that the costs of these reports were higher than the perceived benefits. With few exceptions, there were no quantitative estimates of the costs of the reporting requirement, and for the few estimates provided, there was no substantiation.
Despite the lack of hard evidence, compliance with statutes and regulations clearly entails "costs." Such costs manifest as reporting requirements, coordination and approval processes, schedule delays during the wait for approval, the need for additional personnel dedicated exclusively to statutory and regulatory compliance, and the need for senior-level program officials to focus on such issues rather than on the management of their programs.
At the same time, statutes and regulations also entail "benefits," otherwise they would not have been enacted. Benefits include oversight and accountability; standardization and formalization of decision processes; prevention of fraud, waste, and abuse; ensuring fairness; and providing guidance for inexperienced personnel. One might debate the merits of such benefits and whether they are worth the costs, but the proponents of statutes and regulations clearly thought they were addressing a problem.
Reporting requirements provide a good example of the costs and benefits of regulation. The need to provide reports is generally assessed as a cost by the organization responsible for reporting if it has to expend resources without receiving any obvious benefits. In general, the costs of program status reporting accrue at the program level, while the benefits accrue at higher organizational levels.
At the program office level, monthly status reports are provided to the Program Executive Officer or Service Acquisition Executive (SAE); the quarterly Defense Acquisition Executive Summary (DAES) to the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD[AT&L]); and the Selected Acquisition Report (SAR) to Congress. The program office bears the costs (program management resources, time spent by program personnel) of generating these reports, but does not generally see any direct benefit to its program. As a recipient of the DAES, the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (OUSD[AT&L]) uses it to monitor the health of Major Defense Acquisition Programs (MDAPs) across DoD. Additionally, OSD Program Analysis and Evaluation (PA&E) benefits from the DAES and SAR as analytical tools to track current and historical performance while using the information for estimating and projecting future impacts. Congress uses the SAR to monitor program status and to force the identification and resolution of problems. The program office pays for these reports through staff time and contractor support, but the beneficiaries are OSD and Congress. It is therefore understandable that the program office may view these reports as burdensome while the other organizations do not.
The program office does benefit from some reports, however. It needs monthly status reports, such as the Contractor Funds Status Report (CFSR), Cost Schedule Status Report (CSSR), and Cost Performance Report (CPR), as part of its Earned Value Management (EVM) system of monitoring program status. Semiannual reports, such as the Contractor Cost Data Report (CCDR), also allow the program to make estimates and projections for any program modifications based on historical expenditure patterns. Although the prime contractors prepare these reports, a portion of the program office budget funds their preparation. This is a situation in which the program office pays for, as well as benefits from, the reporting requirements.
A crucial gap clearly exists between the perceived impacts and documented effects of statutes and regulations on the defense acquisition process. Our research attempts to fill the gap and provide an empirical analysis of the effects of statutory and regulatory constraints on outcomes at the program office level.
To a large extent, our approach is experimental. RAND and others have analyzed specific issues, including Other Transaction Authority, special-access programs, and pilot programs in the context of a case study approach, but there has been no truly empirical, systematic study.
The problem is that statutory and regulatory constraints are deeply embedded in existing procedures, making it difficult to separate the consequences of legislative or regulatory actions from the many other controls and events that affect program cost, schedule, and performance outcomes. Additionally, in a government context, there will always be some regulation and oversight because other measures, such as profit, are not sufficient or are inappropriate. The optimal research design would be to define a program's "path not taken" (i.e., the program as it would be without legislative or regulatory constraints) and compare a program executed with and without statutory and regulatory constraints. Unfortunately, the complexity of the regulatory environment does not allow us to credibly define and assess the path not taken. This conundrum presents a difficult research challenge. Thus, our research involves an important methodological component: demonstrating an approach to identify and quantify the effects of statutes and regulations on acquisition programs.
Our goal is to identify specific instances in which compliance with acquisition-related legislation or regulations has led to a specific, identifiable penalty. That penalty might be time lost, additional cost incurred, loss of system capability, additional demands on critical staff, or some other imposition on the program office. If no effects can be proven through the documentation process, we will identify that as well. If significant effects are found, we will develop alternative concepts for mitigating those constraints. This includes working with OSD staff to either change existing policy or develop legislative alternatives to existing law. Over the long run, the existence of even limited empirical data may help policymakers design policies and processes that achieve expected benefits at minimal perceived and actual costs.
Our research was conducted over a period of two years and was divided into three interrelated phases:
Phase 1: Research Design
To be confident that the research will succeed, we spent significant time designing the overall research approach and the data collection protocol in particular. This phase involved discussions with officials throughout the DoD acquisition community to identify statutes and regulations perceived as burdensome and to develop metrics to capture the effects of those statutes and regulations. This phase also included the selection of candidate programs for participation and the development and test (through a pilot test conducted at the beginning of Phase 2) of a Web-based data collection protocol.
An important research task in this phase consisted of a comprehensive review of existing studies and databases. We assessed studies of the legislative and regulatory constraints on acquisition processes both to assemble the substantive results from prior research on this topic and to draw methodological lessons from that prior analysis experience. This task provided the foundation for the development of our own research design and will provide the appropriate background and context for interpreting the results from the primary data collection.
Phase 2: Data Collection
This phase involved implementing the data collection protocol developed in Phase 1, beginning with a pilot test period in which the data collection protocol was fielded at two program offices. We assessed the results of the pilot test and incorporated lessons into a revised instrument prior to fielding it more widely. Full fielding of the protocol subsequent to the pilot test period was envisioned to take approximately 14 months (to capture a full annual cycle of program activities). During this stage, we worked closely with the participating program offices to ensure that the data collection was accurate and only minimally disruptive to the primary mission of the program. We periodically reviewed and summarized the data being collected in order to understand emerging patterns and results and to inform any reform proposals emerging from OSD or Service Acquisition Officials.
Phase 2 also included several related research activities that enabled better insight and more confident interpretation of the data collected through the protocol. These activities included compiling abbreviated case studies of the participating programs, collecting and analyzing additional program data relating to changes in program schedule or scope, and conducting periodic discussions with program officials to identify the consequences of the activities we were tracking. Additionally, we held discussions with Service and OSD officials responsible for the statutory and regulatory areas of interest.
Because of the need for primary data collection, this research required the support of OSD and Service acquisition leadership, as well as the full participation of the programs using the data protocol.
Phase 3: Analysis and Implementation
After the 14-month data collection activity, we performed a comprehensive analysis of the data collected in Phase 2. This task was intended to yield solid, persuasive evidence linking specific statutes or regulations to specific effects on specific programs, or to demonstrate that such evidence does not exist or is inconsequential. Should specific constraints be identified, we would assist OSD in drafting proposals to change the statutory or regulatory basis of acquisition-related processes.
Organization of This Report
This is an interim report documenting the initial research design phase of the study, as well as the pilot test conducted at the beginning of Phase 2. We felt that publishing a separate document on methodology would allow us to provide more detail on our approach, which would allow readers to better assess the validity of our approach and assist in interpreting results. The full fielding of the data collection protocol, the analysis, and the results are to be published separately.
Chapter Two presents a brief history of acquisition reform as well as a review of relevant acquisition reform literature focused on attempts to identify, measure, or mitigate the impact of statutory and regulatory constraints on acquisition programs. This provides important context and places our study firmly within the decades of acquisition reform and policy analysis. Chapter Three summarizes the results of our initial round of discussions with officials in the acquisition community at the OSD, Service, program executive office (PEO), and program office levels. These interviews helped us to identify a set of statutes and regulations for further examination. Chapter Four describes in detail our research approach, including the statutory and regulatory areas chosen for further analysis, the design of the Web-based data collection tool, and the results of the pilot test of that tool. The final chapter summarizes our observations to date.
The appendixes contain additional details about our approach and focus. Appendix A contains the user manual developed for our Web-based data collection tool. Appendix B contains screenshots of that tool. Appendix C provides additional information on the five statutory and regulatory areas selected for analysis.
Excerpted from Measuring the Statutory and Regulatory Constraints on DoD Acquisition by Jeffrey A. Drezner Raj Raman Irv Blickstein John Ablard Melissa A. Bradley Brent Eastwood Maria Falvo Dikla Gavrieli Monica Hertzman Darryl Lenhardt Megan McKernan Copyright © 2006 by RAND Corporation. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.