The index is usually computed monthly, or quarterly in some countries, as a weighted average of sub-indices for different components of consumer expenditure, such as food, housing, clothing, each of which is in turn a weighted average of sub-sub-indices. At the most detailed level, the elementary aggregate level, (for example, men's trousers sold in department stores in the Northwest), detailed weighting information is unavailable, so elementary aggregate indices are computed using an unweighted arithmetic or geometric mean of the prices of the sampled product offers. (However, the growing use of scanner data is gradually making weighting information available even at the most detailed level.) These indices compare prices each month with prices in the price-reference month. The weights used to combine them into the higher-level aggregates, and then into the overall index, relate to the estimated expenditures during a preceding whole year of the consumers covered by the index on the products within its scope in the area covered. Thus the index is a fixed-weight index, but rarely a true Laspeyres index, since the weight-reference period of a year and the price-reference period, usually a more recent single month, do not coincide. It takes time to assemble and process the information used for weighting which, in addition to household expenditure surveys, may include trade and tax data.
Ideally, the weights would relate to the composition of expenditure during the time between the price-reference month and the current month. There is a large technical economics literature on index formulae which would approximate this and which can be shown to approximate what economic theorists call a true cost of living index. Such an index would show how consumer expenditure would have to move to compensate for price changes so as to allow consumers to maintain a constant standard of living. Approximations can only be computed retrospectively, whereas the index has to appear monthly and, preferably, quite soon. Nevertheless, in some countries, notably in North America and Sweden, the philosophy of the index is that it is inspired by and approximates the notion of a true cost of living (constant utility) index, whereas in most of Europe it is regarded more pragmatically. The coverage of the index may be limited. Consumers' expenditure abroad is usually excluded; visitors' expenditure within the country may be excluded in principle if not in practice; the rural population may or may not be included; certain groups such as the very rich or the very poor may be excluded. Black market expenditure and expenditure on illegal drugs and prostitution are often excluded for practical reasons, although the professional ethics of the statistician require objective description free of moral judgments. Saving and investment are always excluded, though the prices paid for financial services provided by financial intermediaries may be included along with insurance.
The index reference period, usually called the base year, often differs both from the weight-reference period and the price reference period. This is just a matter of rescaling the whole time-series to make the value for the index reference-period equal to 100. Annually revised weights are a desirable but expensive feature of an index, for the older the weights the greater is the divergence between the current expenditure pattern and that of the weight reference-period.
In the European Union's Harmonised Index of Consumer Prices, for example, each country computes some 80 prescribed sub-indices, their weighted average constituting the national Harmonised Index. The weights for these sub-indices will consist of the sum of the weights of a number of component lower level indexes. The classification is according to use, developed in a national accounting context. This is not necessarily the kind of classification that is most appropriate for a Consumer Price Index. Grouping together of substitutes or of products whose prices tend to move in parallel might be more suitable.
For some of these lower level indexes detailed reweighting to make them be available, allowing computations where the individual price observations can all be weighted. This may be the case, for example, where all selling is in the hands of a single national organisation which makes its data available to the index compilers. For most lower level indexes, however, the weight will consist of the sum of the weights of a number of elementary aggregate indexes, each weight corresponding to its fraction of the total annual expenditure covered by the index. An 'elementary aggregate' is a lowest-level component of expenditure, one which has a weight but within which, weights of its sub-components are usually lacking Thus, for example: Weighted averages of elementary aggregate indexes (e.g. for men’s shirts, raincoats, women’s dresses etc.) make up low level indexes (e.g. Outer garments),
Weighted averages of these in turn provide sub-indices at a higher, more aggregated level,(e.g. Clothing) and Weighted averages of the latter provide yet more aggregated sub-indices (e.g. Clothing and Footwear).
Some of the elementary aggregate indexes, and some of the sub-indexes can be defined simply in terms of the types of goods and/or services they cover, as in the case of such products as newspapers in some countries and postal services, which have nationally uniform prices. But where price movements do differ or might differ between regions or between outlet types, separate regional and/or outlet-type elementary aggregates are ideally required for each detailed category of goods and services, each with its own weight. An example might be an elementary aggregate for sliced bread sold in supermarkets in the Northern region.
Most elementary aggregate indexes are necessarily 'unweighted' averages for the sample of products within the sampled outlets. However in cases where it is possible to select the sample of outlets from which prices are collected so as to reflect the shares of sales to consumers of the different outlet types covered, self-weighted elementary aggregate indexes may be computed. Similarly, if the market shares of the different types of product represented by product types are known, even only approximately, the number of observed products to be priced for each of them can be made proportional to those shares.
How the weights are calculated, and in how much detail, depends upon the availability of information and upon the scope of the index. In the UK the RPI does not relate to the whole of consumption, for the reference population is all private households with the exception of a) pensioner households that derive at least three-quarters of their total income from state pensions and benefits and b) “high income households” whose total household income lies within the top four per cent of all households. The result is that it is difficult to use data sources relating to total consumption by all population groups.
For products whose price movements can differ between regions and between different types of outlet:
The situation in most countries comes somewhere between these two extremes. The point is to make the best use of whatever data are available.
No firm rules can be suggested on this issue for the simple reason that the available statistical sources differ between countries. However, all countries conduct periodical Household Expenditure surveys and all produce breakdowns of Consumption Expenditure in their National Accounts. The expenditure classifications used there may however be different. In particular:
Even with the necessary adjustments, the National Account estimates and Household Expenditure Surveys usually diverge.
The statistical sources required for regional and outlet-type breakdowns are usually weaker. Only a large-sample Household Expenditure survey can provide a regional breakdown. Regional population data are sometimes used for this purpose, but need adjustment to allow for regional differences in living standards and consumption patterns. Statistics of retail sales and market research reports can provide information for estimating outlet-type breakdowns, but the classifications they use rarely correspond to COICOP categories.
The increasingly widespread use of bar codes and scanners in shops has meant that detailed cash register printed receipts are provided by shops for an increasing share of retail purchases. This development makes possible improved Household Expenditure surveys, as Statistics Iceland has demonstrated. Survey respondents keeping a diary of their purchases need to record only the total of purchases when itemised receipts were given to them and keep these receipts in a special pocket in the diary. These receipts provide not only a detailed breakdown of purchases but also the name of the outlet. Thus response burden is markedly reduced, accuracy is increased, product description is more specific and point of purchase data are obtained, facilitating the estimation of outlet-type weights.
There are only two general principles for the estimation of weights: use all the available information and accept that rough estimates are better than no estimates.
Infrequent reweighing saves costs for the national statistical office but delays the introduction into the index of new types of expenditure. For example, subscriptions for Internet Service entered index compilation with a considerable time lag in some countries, and account could be taken of digital camera prices between re eightings only by including some digital cameras in the same elementary aggregate as film cameras.