Scanner Data and Price Indexes

Hardcover (Print)
Buy New
Buy New from
Used and New from Other Sellers
Used and New from Other Sellers
from $9.91
Usually ships in 1-2 business days
(Save 87%)
Other sellers (Hardcover)
  • All (8) from $9.91   
  • New (3) from $70.45   
  • Used (5) from $9.91   


Every time you buy a can of tuna or a new television, its bar code is scanned to record its price and other information. These "scanner data" offer a number of attractive features for economists and statisticians, because they are collected continuously, are available quickly, and record prices for all items sold, not just a statistical sample. But scanner data also present a number of difficulties for current statistical systems.

Scanner Data and Price Indexes assesses both the promise and the challenges of using scanner data to produce economic statistics. Three papers present the results of work in progress at statistical agencies in the U.S., United Kingdom, and Canada, including a project at the U.S. Bureau of Labor Statistics to investigate the feasibility of incorporating scanner data into the monthly Consumer Price Index. Other papers demonstrate the enormous potential of using scanner data to test economic theories and estimate the parameters of economic models, and provide solutions for some of the problems that arise when using scanner data, such as dealing with missing data.

Read More Show Less

Product Details

Meet the Author

Robert C. Feenstra is associate professor of economics at the University of California at Davis and is coeditor of the Journal for International Economics.

Read More Show Less

Read an Excerpt

Scanner Data and Price Indexes

The University of Chicago Press
Copyright © 2003 National Bureau of Economic Research
All right reserved.

ISBN: 978-0-226-23965-1

Chapter One
Using Scanner Data to Improve the Quality of Measurement in the Consumer Price Index

William J. Hawkes and Frank W. Piotrowski

"The only emperor is the emperor of ice cream." -Wallace Stevens, 1922

1.1 Introduction

Twenty-five years ago, a major technological change swept through U.S. retailing and left the field of marketing research profoundly altered in its wake. Since then, the same tidal wave has moved across most of Europe and the developed countries of Asia and Latin America as well. This technological change involved the source-coding of most fast-moving packaged consumer goods by their manufacturers, using the newly developed Universal Product Code. It also involved the installation, by retailers, of electronic scanning equipment at the checkout counter to "read" and record each item purchased in the store. Subsequently, other kinds of electronic point-of-sale (EPOS) systems were introduced to record transactions of durable goods (e.g., toasters and refrigerators) that had not been source-coded at the manufacturer level.

As a result of this new method of data collection, market research firms were able to obtain and summarize information on consumer sales and retail prices in a much faster, more detailed, and more cost-efficient manner than before. Scanner-based data quickly became the "common language" used by manufacturers, retailers, and marketing research companies to describe and interpret developments in the retail marketplace.

As shown in table 1.1 below, currently in the United States around 10 percent of total consumer expenditures, and around one-quarter of consumer expenditures on goods (as opposed to services, such as housing services and haircuts), are made in categories that can, in large measure, be represented through scanning data obtained from supermarkets, mass merchandisers, and drugstores.

Price index theorists and practitioners have long been observing and measuring the same consumer behavior as have marketing researchers, even though they sometimes use a slightly different vocabulary to describe the transactions that they are studying, as shown in table 1.2 below. Both price index and marketing research theorists are concerned with the response, or elasticity, of consumer purchases to changes in retail price. Despite this commonality of interest, the public sector involved in producing consumer price indexes has been slower than the private sector of marketing research to utilize and benefit from this new technology. Recently, however, government statistical agencies in many countries have to begun to investigate and utilize this new source of data in their consumer price indexes.

This paper, written from the perspective of market researchers who have spent the past twenty-five years working with scanner data, discusses specific ways in which the quality of consumer price information can be improved using this new data source. It also shows how the measurement of product quality can be enhanced through the use of these data. The paper will make use of actual scanner data for a particular product category and will, we believe for the first time, present and discuss a "total U.S." simulated price index for a specific Consumer Price Index (CPI) "food-at-home" commodity (item stratum), comparing the results with the corresponding "urban U.S." figures produced by the U.S. Bureau of Labor Statistics. The paper concludes with a discussion of data aggregation issues in CPI construction.

1.2 How Scanner Data Can Improve the Quality of Consumer Price Indexes' Measurements

The potential benefits from using scanner or other EPOS data in CPIs can be grouped into three categories: (a) more data and, consequently, less variance; (b) better data and, consequently, less bias; and (c) better methods. We shall consider each in turn.

1.2.1 More Data

In most developed countries, scanner data for supermarket items are based on a number of data points (outlets, items, and weeks) that exceed those currently used in these countries' CPIs by several orders of magnitude -generally in the range of 1,000 to 1. To cite one example, CPI data for the U.S. breakfast cereals "item stratum" are based on around 675 individual price observations, one observation per month for two or three items per store in a sample of around 300 individual outlets. In contrast, ACNielsen scanner data for supermarkets are based on four or five price observations per month for over 200 cereal items per store in a sample of around 3,000 supermarkets. If the scanner data reporting period, for CPI purposes, is constrained to the first three weeks of each month, then scanner data consist of 1,800,000 price observations per month (3 x 200 x 3000). This is 2,700 times as many price observations as are currently being obtained for breakfast cereals (1,800,000 divided by 675) in the CPI program. These price records are, in every case, accompanied by actual quantities sold, each week, in each supermarket for each item, in contrast to the implicit quantity weights for each price observation in the CPI, which usually remain unchanged for four or five years.

1.2.2 Better Data

Even within the framework of current CPI designs in most countries, scanner data provide the opportunity for significant quality improvement in terms of bias reduction along several dimensions:

1. Sample outlet selection. Scanner retail outlet samples are generally selected from a well-defined frame that lists all universe supermarkets, mass merchandisers, and large drugstores. For the U.S. CPI, sample retail outlets are drawn from a list of "point-of-purchase" outlets obtained from a sample of around 3,500 households nationally. In many other countries, individual CPI retail outlets are selected from incomplete or geographically restricted frames.

2. Outlet sample updating. Scanner samples are usually designed to incorporate new outlets with minimum delay. Consumer price index outlet samples, in most countries, are refreshed or replaced only at periodic intervals, generally once every several years.

3. Item selection. Designation of individual items to be priced is carried out, in most countries, in one of two ways:

Selection of items within a store with probability proportionate to measures of size, in theory based on actual expenditures but often based on shop owners' memory or estimates, or shelf space, or some other "proxy" means. This is the procedure currently used in the United States.

Purposive or judgmental selection of items by product characteristics, with specific products, or varieties then chosen in the field or, in certain instances, centrally designated. This is the procedure currently used in the United Kingdom and in Canada.

In contrast, scanner data are provided for every item in every category handled and scanned by the store. At the very least, scanner data could be used to check out the validity of the "purposive" item selection methods used in areas such as the United Kingdom and Canada.

4. Item updating. Scanner data automatically include all new items appearing in each sampled retail outlet, generally in "real time," or with a delay of a few weeks at most to allow for a full product description to be de- fined for each new item code. In contrast, CPI new items are brought in only when existing items are discontinued at the individual outlet level or when a complete item reselection is carried out, generally once every several years. Both new items and new outlets are generally linked to the previous price index generated by the old items and old outlets, with no allowance for differences in price levels between new and old, except for item strata where explicit hedonic adjustments can be made. In an important paper, Reinsdorf (1993) showed that the combined effect of new items and new outlets in the U.S. CPI for food-at-home items was to reduce average price levels for food-at-home commodities by 0.25 percent per year.

5. Better lower-level (within item stratum) expenditure weights. Lower-level item expenditure or quantity weights used in CPI construction are generally based on estimates made at infrequent intervals from a variety of sources: consumers' recall, consumer purchase diaries, and shopkeepers' estimates. Use of out-of-date expenditure weights is likely to result in an overstatement of inflation. In contrast, scanner data provide current, up-to-date expenditure weights each week.

1.2.3 Improved CPI Scope, Definitions, or Methods

For many years, price theorists have written of "superlative" or "ideal" price indexes more as a concept than as a reality, since current period quantities have generally not been available. In a scanning environment, this restriction no longer exists. Accordingly, a number of methodological questions come immediately to mind. Scanning data can help provide answers to these important questions:

1. Should expenditure weights be computed for each specific month, or is it better to use more stable weights (e.g., for the most recent year), on the assumption that trading off some temporal "characteristicity" in weights will be more than offset by reducing the greater intransitivity associated with chaining true Fisher or Törnqvist indexes? Triplett (1998) has pointed out the instability and intransitivity that can result from the chaining together of even superlative indexes when quantities and prices change abruptly from month to month. Recent studies at the Bureau of Labor Statistics (BLS) seem to support the desirability of using the most recent annual quantity weights with scanning data, updated each year, rather than using the monthly quantity weights that accompany the monthly prices. Silver (1995) and Diewert (2000b) advocate constructing an annual moving weight to avoid seasonal bounce.

2. To what extent should weekly sales and quantities for individual items in individual outlets be aggregated to construct "unit values" across items, outlets, and time? Although a unit value index fails the identity and proportionality tests and thus, according to Balk (1998), "cannot be called a price index," it is also true that the unit value index passes the circularity or transitivity test (cited by Balk as a further axiomatic test for a price index), which the Fisher and the Törnqvist, both considered superlative, are guaranteed to fail! The same is true of the "consistency in aggregation" test: the unit value passes this test, whereas the above superlative indexes fail it. Moreover, the unit value is (or can be) automatically adjusted for new items and new outlets. The question of aggregation is discussed at greater length in section 1.5 of this paper.

3. How can scanner data be used to improve the stratum weights used to aggregate the city-by-category lower-level indexes into higher-level indexes? In a recent paper, Diewert (2000b, 26) has cited the "large measurement errors" in these weights as a serious problem in producing an accurate national price index. Certainly the current need to produce expenditure weights for the 1,292 "scannable food-at-home" strata (thirty-eight geographic areas times thirty-four item strata) places a high degree of stress on the sample of roughly 10,000 households each providing two weeks of expenditure data in the U.S. Consumer Expenditure Survey.

An obvious alternative is to make use of aggregate outlet-based scanner data for this purpose. As shown in table 1.3 below, Consumer Expenditure Survey data for 1998 and 1997 agree fairly closely at the national level with ACNielsen ScanTrack data for one product category that will be examined extensively in this paper, but this apparent agreement obscures the fact that the Consumer Expenditure data are probably around 16 percent too low when nonscanning outlet types, sales taxes, and Alaska and Hawaii are taken into account. It would seem an easier matter to estimate this missing 16 percent from the BLS point-of-purchase survey or from household-based scanner data than to estimate the entire 100 percent from the Consumer Expenditure Survey.

In addition to using scanner-based data, ACNielsen also measures consumer sales directly through a 55,000 household sample. The sampled households are supplied a hand-held scanner with a downloaded list of retail establishments in their neighborhood. Households are instructed to scan all UPC coded items they purchase and identify the retail establishment where the purchase occurred. The ACNielsen HomeScan service indicates that 89 percent of ice cream-type products are purchased in traditional supermarkets. This would support the claim that the Consumer Expenditure data may underestimate true sales.

In any event, it would be a useful exercise to carry out this comparison for the thirty-three other scannable edible item strata as well, first at the national level and eventually at the geographic area level as well.

4. How should product categories be defined and subdivided in such a way as to maximize the temporal and geographic transitivity of the resulting indexes and to avoid item "churn" (i.e., excessive item turnover) while enhancing the comparability of items across time and across geography?

5. The foregoing leads to the next question: how many item strata should there be in the CPI, and how should they be defined? The current fifty-three food-at-home item strata have changed very little in the past twenty years. Essentially, the number of item strata has been determined by the limitations of the 10,000-household Consumer Expenditure Survey sample, the need for continuity and for seasonal adjustment, and the data collection budget. If we were freed from these constraints, how might we want to proceed?

An orthogonal view, or at least an alternative view, of the food-at-home universe is shown in table 1.4. Note that the ACNielsen item structure tends to reflect the "department" or physical layout of the typical supermarket, whereas the BLS structure is driven by classification of item complements and substitutes. Thus, ACNielsen classifies Ice Cream as a frozen food, rather than as a dairy product; ACNielsen classifies vegetables in three different places: frozen, canned, and perishable, just as they are found in the supermarket, whereas BLS groups them together under the fruits and vegetables heading and then divides them into subgroups by form.

Obviously, there is no right or wrong in these taxonomies, but there are differences. Of greater interest is that ACNielsen has further subdivided its 64 categories into 603 separate modules. To some extent BLS also carries out further subdivisions of its 34 item strata, both explicitly into its 44 "entry-level items" (ELIs), and implicitly (for sampling purposes of individual items to be priced in specified outlets) through a "disaggregation" procedure that partitions "entry-level items" down into successively smaller subgroupings based on criteria such as form and package size. Through this process, the 34 "scannable edible" item strata are thus further subdivided into 79 "clusters" or mutually exclusive and exhaustive building blocks. These 79 clusters might be a natural next level for category detail to be used as input to higher-level index aggregates in the CPI if resources allowed.

However, the conceptual distinction between what should be a cluster and what should merely be a disaggregation criterion is not always clear. In the CPI example above, fresh fruits and vegetables are divided into eight separate item strata, whereas canned vegetables share an item stratum with canned fruits, although the item stratum is subdivided two separate clusters, one for fruits and one for vegetables. ACNielsen has sixty-nine separate modules for canned fruits and vegetables. Is sixty-nine too many? Is one too few? Is two?

With scanner data, alternative partitionings of the entire product space can be carried out on an experimental basis to determine what the optimal clustering rules should be, in order to produce the most reliable and efficient overall index at various geographical levels.

6. There has been a blizzard of papers written over the past decade, both within BLS and from interested observers, on the subject of within-stratum price elasticity. These papers have been written in the context of two very real and important issues:

Did the Laspeyres assumption of a Leontief preference function lead to a serious upward bias in the CPI prior to 1999?

Has the geometric mean formula used in lower-level index construction starting in 1999, which assumes Cobb-Douglas preferences and unit elasticities, fully corrected the problem?


Excerpted from Scanner Data and Price Indexes Copyright © 2003 by National Bureau of Economic Research. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Read More Show Less

Table of Contents

Prefatory Note
Introduction - Robert C. Feenstra and Matthew D. Shapiro
I. Scanner Data in Official Statistics: Advancing the State of the Art
1. Using Scanner Data to Improve the Quality of Measurement in the Consumer Price Index
William J. Hawkes and Frank W. Piotrowski
2. Scanner Indexes for the Consumer Price Index
David H. Richardson
3. Price Collection and Quality Assurance of Item Sampling in the Retail Prices Index: How Can Scanner Data Help?
David Fenwick, Adrian Ball, Peter Morgan, and Mick Silver
4. Estimating Price Movements for Consumer Durables Using Electronic Retail Transactions Data
Robin Lowe and Candace Ruscher
Roundtable Discussion - Dennis Fixler, John S. Greenlees, David Fenwick, Robin Lowe, and Mick Silver
II. Aggregation across Time
5. High-Frequency Substitution and the Measurement of Price Indexes
Robert C. Feenstra and Matthew D. Shapiro
Comment: Marshall B. Reinsdorf
6. Using Scanner Data in Consumer Price Indexes: Some Neglected Conceptual Considerations
Jack E. Triplett
III. Using Price Data to Study Market Structure
7. What Can the Price Gap between Branded and Private-Label Products Tell Us about Markups?
Robert Barsky, Mark Bergen, Shantanu Dutta, and Daniel Levy
Comment: Julio Rotemberg
8. The Long Shadow of Patent Expiration: Generic Entry and Rx-to-OTC Switches
Ernst R. Berndt, Margaret K. Kyle, and Davina C. Ling
Comment: Steve Morgan
IV. Measuring Change in Quality and Imputing Missing Observations
9. The Measurement of Quality-Adjusted Price Changes
Mick Silver and Saeed Heravi
10. Hedonic Regressions: A Consumer Theory Approach
Erwin Diewert
11. Price Index Estimation Using Price Imputation for Unsold Items
Ralph Bradley
Comment: Eduardo Ley
Author Index
Subject Index


Read More Show Less

Customer Reviews

Be the first to write a review
( 0 )
Rating Distribution

5 Star


4 Star


3 Star


2 Star


1 Star


Your Rating:

Your Name: Create a Pen Name or

Barnes & Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation


  • - By submitting a review, you grant to Barnes & and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Terms of Use.
  • - Barnes & reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously

    If you find inappropriate content, please report it to Barnes & Noble
    Why is this product inappropriate?
    Comments (optional)