Proceedings of Q2008 European Conference on Quality in Official Statistics
Revision Analysis to Detect Possible Weakness in the Estimation Procedures An application to the Italian IIP Anna Ciammola, Teresa Gambuti, Anna Rita Mancini, ISTAT1
Abstract Economic indicators are very often revised and the size of revisions, computed comparing subsequent estimates and previous ones, allow to assess their reliability. The importance of this quality dimension is confirmed by the growing interest in revision analysis from international organizations (Eurostat, OECD) and National Statistics Institutes (NSIs) as well as their efforts towards transparency providing information about past revisions, scheduling future revisions due to changes in methods and definitions, creating real-time data bases. Most of these efforts are users-oriented because users see with a certain criticism the fact of revising economic statistics. However, in some cases, revision analysis could help detecting possible “weakness” in the estimation procedures and to suggest suitable measures to counteract them. This document is aimed at proving some examples of this kind of tools, using an application to the Italian Index of Industrial Production (IIP) monthly released by ISTAT. Keywords: Measuring accuracy, revisions, industrial production index.
1. Introduction Economic statistics are typically revised and the size of revisions reflects the trade-off between accuracy/reliability and timeliness. In particular accuracy refers to the closeness between the estimated value and the true value measured by the statistic (usually unknown); reliability refers to closeness of the initial estimate to subsequent (revised) estimates. The latter is measurable and the size of revisions, computed comparing subsequent estimates with previous ones, allows to assess the reliability, though non revised estimates are not to be considered automatically as reliable estimates (see Di Fonzo, 2005). Users often see with a certain criticism the revision of economic statistics. To improve the communication about the revision process many NSIs have made efforts towards transparency, providing information about past revisions, scheduling future revisions both statistical and definitional, creating real-time databases where all the vintages are gathered, analysing size, bias and efficiency of revisions. Most of these efforts are users-oriented, since revisions spark off controversies and seem to threaten NSIs credibility. This contribute to explains why revision analysis is mainly restricted to key economic indicators (in seasonally adjusted form). However, revision 1
Address for correspondence: Anna Ciammola, ISTAT - Istituto Nazionale di Statistica, via Tuscolana 1778 - Rome, Italy, 00173 (e-mail:
[email protected]).
1
analysis could be utilized in different contexts and specifically can eventually help detecting problems in the statistical estimation/compilation process. In fact, as the Statistics Commission stated in its report (Statistics Commission, 2004, pag. 4) some revisions are not the consequence of additional information, but are potentially “avoidable”, as they are “due to errors or to weakness in the estimation procedures, or to tractable weakness in the underlying data systems”. In particular the report highlights four categories of avoidable circumstances that affect the revision process (pag. 24): • substantial mistakes in early processing; • the models used to compute early estimates are not “best practice”; • timetables could be more rapid than they actually are; • the methods used are “best practice”, but they are implemented without sufficient resources. Revision analysis could be a useful tool both to detect such circumstances and to suggest suitable measures to counteract them. This document is aimed at proving some example of these kind of tools, using an application to the monthly Italian IIP released by ISTAT. It is organized as follows: section 1 states the matter; section 2 sketches the main features of the IIP with the sources of revisions and their timing; section 3 describes the measures used to analyse revisions on raw data and how such analysis carried out according to a top-down approach (i.e. analysing first the highest level and then proceeding to the more detailed levels) has allowed to identify the specific sector “responsible” for the slight positive bias in revisions of IIP; section 4 compares the main results derived from the revision analysis with those obtained analysing the response rates and indicates what does not probably represent the “best practice” in the computation of early estimates; section 5 examines the stability of results over time; section 6 concludes.
2. Sources and timing of revisions for Italian IIP In Italy the revision process of the IIP changed in October 2004 (with the release concerning the August 2004 data), introducing a revision policy more articulated than before2 . In the current practice, three sources of revision can be identified: 1. additional data arriving from late respondents (increase in response rate); 2. effects due to corrections of errors in data already embodied in the estimates, either caused by the incorrect internal treatment of source data or resulting from wrong information previously provided by respondents and replaced later on (very often after direct contacts with the respondents); 3. the revision of statistics (external to the survey) utilized in compiling the IIP: productivity coefficients (calculated as value added per hour worked drawn from national accounts) are utilized in selected sectors where the output is evaluated measuring the labour input (hours of work); those sectors account for about 7% of the total industrial production in the base year. 2
The introduction of this more complex revision policy can significantly influence comparisons over different periods of the size and characteristics of IIP revisions (see McKenzie and Park, 2006).
2
The current revision policy is clearly described in the IIP metadata and can be summarized as follows: • the first revision of indexes (based mainly on information provided by late respondents) is released after one month, when the new current month statistics are published; • a second revised version of the indexes is released at fixed points in time: in April (releasing February data) and in October (releasing August data). The April release takes into account all the sources of information listed above and in particular the revised version of yearly National Account estimates (released in March) concerning the previous three years. In October only the cumulative effects of the first source of error are considered, revising IIP indexes referring to the first semester of the current year.3 As far as the seasonally adjusted data are concerned, the revision process reflects both the previously discussed sources of revision (concerning the estimation of the raw index) and the use of symmetric two-sided filters. When the observation for the current time period is adjusted, future observations are not available, thus they are forecast. The TRAMO-SEATS procedure (Gomez and Maravall, 1997) is used to seasonally adjust the index according to the concurrent approach: reg-ARIMA models are revised yearly in April to take the above revisions into account; the parameters of the ARIMA models and the resulting seasonal factors are re-estimated each month and the entire revised time series of seasonally adjusted values is released. In the ARIMA model-based approach of TRAMO-SEATS, revisions in seasonally adjusted data reflect both the use of two-sided symmetric filters and the ARIMA model estimated on raw data and decomposed into ARIMA models for the latent components in accordance with the canonical decomposition (see Gomez and Maravall, 1997). This implies that the properties of the revision process depend on the ARIMA model and generally there is a trade-off between the size of revisions and the smoothness of the seasonally adjusted series. Moreover, if the current index value released at time t is very different with respect to the forecast estimated at time t − 1 (i.e. there is a large one-step-ahead forecast error), preliminary estimates at t − s, t − 2s,. . . , where s = 4, 12 for quarterly and monthly time series respectively are significantly revised. Non-seasonal and seasonal revisions may be very different with regards to their size, their properties and their convergence. As a consequence, the revision analysis carried out only on seasonally adjusted IIP is not able to describe completely the properties of the overall revision process and it is of little use for the producer of the statistics. Sophisticate approaches (recently Mehrhoff, 2008) have been proposed to decompose the two sources of revision: the revision of raw unadjusted data and the (generally symmetric) filters used for the seasonal adjustment. Such approaches are certainly very interesting, but they could be difficult to apply, especially when seasonally adjusted data are computed disaggregating annual data through quarterly seasonally adjusted indicators. A simpler approach is based on the analysis of three sets of revisions (on unadjusted data, on working-day adjusted (WDA) data and on seasonally adjusted (SA) data) and on the 3
Occasionally, with the April 2007 release, the previous six years were revised, from January 2001 to December 2006.
3
comparison of the related statistical measures. In fact, stability of seasonal factors are always checked in the seasonal adjustment procedures, but the check is implemented on the last vintage. Consequently, useful information can be derived through the comparison of revisions at different stages of the production process. This will be discussed in the next section.
3. Revision analysis Following the introduction of a new and more complex revision policy for the Italian IIP, ISTAT has released in October 2006 a first “real-time” database and some statistical measures on revisions in order to fulfil the transparency principle and to provide users with a useful tool. The starting point of that analysis is the work that ONS and OECD have already carried out. Raw, WDA and SA data are considered and the results of revision analysis compared. Firstly, revisions on unadjusted data are compared with revisions on WDA data. Stability of the factors used to remove calendar effects is checked but it is usually analysed on the last release of the indicator to be adjusted, without considering the impact of the revisions in raw data. When the latter occur, revision in WDA data depends on two factors: revisions in raw data and changes in the parameter estimates in reg-ARIMA models. Changes in these parameters are expected to be small, because they reflect, in some way, habits of employees and employers that don’t change from a month to another. This implies that unadjusted and WDA data should have very close revision processes and possible divergences between them should be explained to users. Then revisions of SA data are considered. The revision process due to the SA is more complicated than the revision process due to the update of the working-day factors. In fact, since two-sided filters are used to remove the seasonal component, revisions reflect the forecast errors, that in turn depend on the volatility of the unadjusted series and on the estimated ARIMA model (as an ARIMA model based method is used in ISTAT). This means that unlike the revision on WDA data, the revision process is not expected to be very close to the revision process on raw data. In spite of that, size and other properties of revisions on SA data should be assessed comparing them with the revisions on raw or WDA data. In fact, if revisions in SA data show a systematic component, this could depend either on the revision process of unadjusted data or on the seasonal adjustment procedure. Revision analysis of IIP is carried out from January 2001. The year-on-year and the month-on-month growth rates (the latter only for seasonally adjusted data), rather than the levels, are considered. The measures suggested in Di Fonzo (2005) are computed over two different periods, starting from January 2001 and January 2003, respectively, but only the latter are considered because of several changes occurred at the beginning of 2003: the change of the base year, the passage to a new classification system, the choice of a different ARIMA model to seasonally adjust IIP and the use of the regressor approach to remove the calendar effects. In the subsequent years further changes have been introduced: a new revision policy and finally a different treatment of the National holidays. The indicators used to describe the revision processes are: 1. mean of revisions (MR) and its standard error; 4
Figure 1: Revisions on year-on-year growth rates of Italian IIP 0.5 0.4 0.3 0.2 0.1 0 −0.1 −0.2 −0.3 −0.4 −0.5 Jun
Dec−03
Jun
Dec−04
Jun
Dec−05
Jun
Dec−06
Jun
Dec−07
2. mean of absolute revisions (MAR); 3. relative mean of absolute revisions (RMAR); 4. mean squared revisions (MSR) and its decomposition (UM, UR and UD); 5. minimum, maximum, range and other statistics. The measure 1 may reveal whether revisions are systematic or not (i.e. affected by an apparent bias); the measures 2 and 3 show the size of the revisions; the decomposition of 4 displays possible systematic components in the revision process; the measures 5 are descriptive statistics that complete the characterisation of the revision process. The results of the analysis are displayed in table 1 and they refer to raw, WDA and SA indices. As far as the year-on-year growth rates on raw index are concerned, two aspects have to be stressed for revisions after 1 month (h = 1): firstly, the mean of absolute revisions (MAR), in the second line, shows that the average size of revisions is rather small but not negligible from the user point of view; secondly, the mean of revisions (MR), in the eighth line, is positive and statistically significant. The average size of revisions after 12 months (h = 12) is larger but not systematic. Figure 1 shows the revisions between the preliminary growth rates and the revised growth rates after one month. With regards to the growth rates on WDA index, the revision process is, as expected, rather close to the one concerning the raw index. Finally, growth rates on SA indexes are larger. They do not show any systematic bias at the level α = 5%. In order to verify if the source of systematic revisions can be traced back to particular components (i.e. specific sectors) of the index, which may hint at the effect of sub-optimal practices in the computation of early estimates, the analysis can proceed according to a top-down approach (i.e. analysing first the highest level and then proceeding to the more detailed levels). The first step is to consider an analysis of the Main Industrial Groupings (MIGS) indices. The year-on-year growth rates on raw indices are 5
Table 1: Measures of revision for Italian IIP (Jan. 2003 - Dec. 2007) Year-on-year growth rates Statistics Raw data WDA data SA data h = 1 h = 12 h = 1 h = 12 h = 1 h = 12 n 60 48 60 48 60 48 MAR 0.14 0.25 0.15 0.33 0.2 0.33 RMAR 0.05 0.09 0.08 0.16 0.32 0.59 MR 0.08 0.08 0.06 0.05 0.02 0.07 SDHAC 0.021 0.056 0.026 0.069 0.026 0.047 t−value 3.56 1.49 2.14 0.72 0.72 1.51 t(1−0.05/2,n−1) 2.00 2.01 2.00 2.01 2.00 2.01 MR Significance YES NO YES NO NO NO MSR 0.040 0.105 0.045 0.186 0.073 0.179 UM 0.141 0.066 0.072 0.013 0.005 0.028 UR 0.041 0.093 0.01 0.03 0.05 0.151 UD 0.818 0.84 0.918 0.956 0.946 0.821 MIN -0.4 -1 -0.5 -1.1 -0.6 -1 MAX 0.5 0.7 0.6 1.1 0.8 0.9 RANGE 0.9 1.7 1.1 2.2 1.4 1.9 %L>P 51.7 54.2 50 52.1 35 50 % SL = SP 100 97.9 95 89.6 81.7 72.9 Legend: n - number of revisions; MAR - Mean Absolute Revision; RMAR - Relative MAR; MR - Mean Revision; SDHAC - Heteroskedasticity and Autocorrelation Consistent Standard Deviation; t-value - t-statistics for significance of mean revision; t(1−0.05/2,n−1) critical value of t statistics (α = 5%); MR Significance - statistical significance of the mean revision; MSR - Mean Squared Revision; UM, UR and UD - Theil’s decomposition of MSR; MIN(MAX) - value of the lowest (highest) revision; RANGE - (MAX - MIN); %L>P percentage of positive revisions; %SL=SP - percentage of observations where the sign of later estimate and the sign of earlier estimate are the same. For a better interpretation of these measures see http://www.oecd.org/dataoecd/47/18/40315546.pdf.
considered, as the adjustment for both calendar effects and seasonality could mask the properties of the revisions caused by the collection process and compilation procedures. Moreover, only revisions after one month are considered. This for three reasons: firstly the yearly update of the productivity coefficients coming from National Account data generates steps in the revision time series that, as a consequence, shows significant autocorrelations at lags 1, 2, 3, . . . ; secondly, excluding the indices referring to the January of each year, the revisions after one month mainly reflect the treatment of non-responses and the correction of errors in the collection process; finally, the revision analysis of the aggregate index indicates that revisions after one month are small, but characterized by a systematic component. MIGS weights in the compilation of the overall index are displayed in table 2. Table 3 shows the revision measures concerning the MIGS indices. All these series show larger revisions in comparison to the total index, especially capital goods (CAP) and consumer durables (CDU) whose MAR are 0.38 and 0.42, respectively (however their 6
Table 2: MIGS weights Short name Name Weights (%) CND Consumer non-durables 22.9 CDU Consumer durables 6.1 CAP Capital goods 23.8 INT Intermediate goods 35.5 ENE Energy 11.7 RMAR is 0.09 and 0.08). On the other hand, the analysis of revisions shows a systematic bias for the series INT, characterized by a small but statistically significant positive average error (0.14): this is highlighted by both the t−value and the decomposition of the MSR (see measure UM).4 This result could point out where the systematic component of the IIP revisions is generated. Table 3: Measures of revision for MIGS (Jan. 2003 - Dec. 2007) Year-on-year growth rates on raw indices (h = 1) Statistics CND CDU CAP INT ENE n 60 60 60 60 60 MAR 0.27 0.42 0.39 0.22 0.15 RMAR 0.08 0.08 0.09 0.07 0.04 MR 0.092 0.072 0.042 0.143 -0.003 SDHAC 0.047 0.103 0.071 0.030 0.042 t−value 1.962 0.694 0.589 4.724 -0.079 t(1−0.05/2,n−1) 2.001 2.001 2.001 2.001 2.001 MR significance NO NO NO YES NO MSR 0.123 0.609 0.286 0.084 0.080 UM 0.068 0.008 0.006 0.244 0.000 UR 0.004 0.024 0.040 0.002 0.046 UD 0.928 0.967 0.954 0.754 0.954 MIN -0.7 -3.4 -1.4 -0.3 -0.6 MAX 1.0 3.4 1.4 1.0 1.0 RANGE 1.7 6.8 2.8 1.3 1.6 %L>P 56.7 50.0 48.3 66.7 20.0 % SL = SP 96.7 96.7 96.7 98.3 98.3 At this point the computation of the average contribution of each MIGS to the MR of the overall index could be particularly informative. Since for months July 2004, January 2005, January 2006 and January 2007 the revised estimates are also affected by the revision of the productivity coefficients, ad hoc estimates are computed for such months. This 4
The result concerning the t−value of the INT revision process derives from the application of the Newey-West statistic suggested by Di Fonzo (2005), although both the standard t−statistic and the adjusted t−statistics (Jenkinson and Stuttard, 2004) also highlight a positive bias. With reference to the decomposition of the MSR, computational details could be found in Di Fonzo (2005). It is worth noting that such decomposition could highlight both the presence of a systematic component and anomalous revisions (i.e. one revision much larger that the other ones).
7
simulation exercise is aimed at: • isolating the effect of the imputation of late respondents removing the effects due to the revision of the productivity coefficients; • fulfilling the condition necessary to compute the average contribution of each MIGS. In fact, revisions refer to the year-on-year growth rates and the average contribution of each MIGS to the IIP MR can be computed only if, for each month, both the preliminary and the revised growth rates have the same denominator. Let ytp and ytr be the preliminary and the revised year-on-year growth rates at time t: ytp =
Itp It−12
ytr =
−1
Itr It−12
−1
where Itp and Itr are, respectively, the preliminary and the revised indices. The revision rt is: rt = ytr − ytp . Since It¦ is computed according to Laspeyres’ formula, it could be expressed as a weighted average of m component indices: It¦ =
m X
¦ wj Ij,t
j=1
where the weights wj fulfil the condition rt =
Pm j=1
wj = 1. After simple calculations:
m p r X wj (Ij,t − Ij,t ) j=1
It−12
=
m X
cj,t
j=1
where cj,t is the contribution of the j − th component to the revision rt . This last formula stresses that, when revisions concern growth rates, the computation of contribution cj,t relies on the condition that both ytp and ytr have the same denominator It−12 . In our exercise, this condition is always fulfilled, apart from those months, already listed, where ytp and ytr are affected by different productivity coefficients. Table 4, shows the average contribution of each MIGS to the MR of the overall index (from January 2004 to December 2007) which confirms the results of table 3. In figure 2 revisions for both INT and CAP are displayed. Series INT has a smaller revision process than series CAP (see the panels on the left side), but the former is biased towards positive values, especially for the January months (see the panel on the top right corner). Similarly for series CAP, January is a critical month as revisions vary approximately from -1.5 to 1 (see the panel on the bottom right corner).5 In the next section possible sources for such systematic patterns are investigated. 5
This is confirmed by the coverage rates of the January months. They are usually smaller because of three factors: firstly, indices are compiled in March when firms have to draw up their balance sheets; secondly, new questionnaires are sent, sometimes by mail with possible delivery delays; finally, corporate changes (e.g. mergers) are generally concentrated in the first months of the year.
8
Table 4: Average MIGS contribution to the MR of the overall index (*) IIP MR CND CDU CAP INT ENE 0.079 0.019 0.006 0.010 0.047 -0.003 (*) These results refer to a simulation exercise over the period January 2004-December 2007 excluding from the revisions the component due to the productivity coefficients.
Figure 2: Revisions on year-on-year growth rates of Intermediate and Capital Goods Capital goods
Capital goods by month
1.5
1.5
1
1
0.5
0.5
0
0
−0.5
−0.5
−1
−1
−1.5
Dec−03
Dec−04 Dec−05 Intermediate goods
Dec−06
Dec−07
−1.5
1.2
Jan Feb Mar
Apr May Jun Jul Aug Sep Oct Nov Dec Intermediate goods by month
Jan Feb Mar
Apr May Jun
1.2
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
−0.2
−0.2
−0.4
Dec−03
Dec−04
Dec−05
Dec−06
Dec−07
−0.4
Jul
Aug Sep Oct Nov Dec
4. Sources of the bias in the INT revision process In the previous section the use of a top-down approach has pointed out elements suggesting to consider the role of the intermediate goods component as the main source of systematic revisions in the total index, this component index having both the largest weight in the compilation of IIP and the most important average contribution to the MR of IIP. As discussed in section 2 the main source of revisions between preliminary estimates (released 40 days after the end of the reference month) and the revised estimates (30 days later) is the inclusion of late respondent units. Table 5 describes the weighted response rates6 of IIP and MIGS for the years 2004-2007. Two aspects can be deduced analysing the table: firstly, there is a decreasing trend in the degree of coverage as the base year gets more distant; secondly, the two most important MIGS, INT and CAP, display the lowest response rates. However, as evidenced in table 3, their over6
The weighted response rates are obtained as the weighted sum of elementary rates, given by the ratio between production collected from respondent units and the overall production (coming from the imputation of nonrespondent). Compared to the simple ratio between the number of respondent units and the total units, this indicator takes into account the firm size.
9
all revision processes are different: apart from the measures RMAR7 nearly identical, INT has smaller revisions than CAP but with a systematic component. Specifically, the preliminary estimates are on average revised upwards. These results suggest to further develop the analysis of the statistical method utilized in the imputation of nonresponses, trying to disentangle any source of systematic underestimation of the production levels of late respondents. Table 5: Weighted response rates Name Year Estimate Measure IIP CND CDU CAP Mean 91.5 93.9 94.3 90.3 First Min 87.6 89.1 89.2 85.6 Max 93.6 96.0 96.6 92.5 2004 Mean 95.0 95.7 96.1 93.4 Second Min 93.2 93.4 92.3 92.0 Max 96.2 96.6 97.8 95.8 Mean 90.2 90.6 93.5 88.1 First Min 82.8 81.5 84.5 79.9 Max 92.5 93.8 97.8 91.0 2005 Mean 93.3 93.1 95.4 91.3 Second Min 90.7 88.0 92.2 88.3 Max 95.1 95.9 97.8 94.1 Mean 88.7 89.0 90.4 87.4 First Min 85.2 83.3 83.7 85.2 Max 91.2 91.6 95.0 89.9 2006 Mean 91.7 91.4 92.5 90.1 Second Min 88.0 89.1 88.7 87.8 Max 93.3 94.4 96.2 92.6 Mean 83.7 84.7 82.4 80.8 First Min 78.1 77.5 72.9 76.2 Max 86.4 88.1 88.7 86.0 2007 Mean 87.6 88.4 86.0 85.6 Second Min 84.6 85.0 80.0 82.2 Max 89.7 91.4 89.6 89.4
INT 88.3 83.3 92.2 93.8 90.0 95.0 87.9 80.3 91.9 92.4 88.6 93.7 86.4 80.8 90.8 90.1 82.4 92.7 80.6 73.6 84.9 84.9 80.5 88.2
ENE 97.4 87.9 99.7 99.6 98.4 100.0 98.7 96.9 100.0 100.0 99.7 100.0 97.3 77.7 100.0 99.9 98.8 100.0 97.6 88.1 99.7 99.0 90.1 100.0
Legend: Mean - mean of the monthly response rates; Min - lowest response rate; Max - highest response rate.
To proceed further in the analysis, the top-down approach is again applied to the INT components (7 divisions and 20 groups of the NACE Rev. 1.1 nomenclature). For each component, several elements are analysed: • the revision processes by computing the measures listed above; • the average contributions to the MR of INT growth rates; 7
The measure RMAR is useful for comparing the size of revisions (of growth rates) across components whose average growth rates differ largely.
10
• the increase in the weighted response rates between the first and the second release; • the correlation between the increase in the weighted response rates (from the first to the second release) and the absolute revisions. This analysis allows to identify the specific sectors (divisions and groups) belonging to the INT aggregate marked out by systematic revisions. In accordance with the diagram of figure 3, for each of such sectors the four digits components (NACE classes) are considered and their revision processes are analysed. Figure 3: Diagram describing the top-down approach in the revision analysis of INT (D, G and C stand, respectively, for NACE divisions, groups and classes). ¨ ¥ §IIP ¦ A ¡¢@ ¡¢ A@ A @ ¡ ¢ A @ ¡ ¢ ¢ A @ ¡ @ ¢ A ¡ @ ¢ A ¡ @ ¢ A ¡ R ¢® ª ¡ ? ¨ AU ¥ @ CND CDU CAP §INT ¦ ENE ! © ´ !¡ @ ¢CAS © !´ © ! !©´¡¢ CAS@ ! © !© ´¡ ¢ C AS@ !© ´ !© C A S@ ! ´ ¡ ¢ ! © ´ ¡ ¢ ! © C A S@ ! ´ ! ©© ´ C A S @ ¢ ! ¡ ´ !! ©© C A S @ ¢ ¡ ! ´ © ! A S @ C ¢ ! ¡ ´ © ! ¢® ¡ ª ! + ¥ ¼ ¨ ´ © S @ A C INT T IN T IN T IN T D S @ A C DIN D D D 1 ... 7 § j ¦ ... @ S A C ¢¤CA @ S A C ¢¤ CA @ S A C ¢¤ CA @ S A C ¢ ¤ C A @ S A C ¢ ¤ C A @ S A C ¢ ¤ C A @ S AU CW R w ? ¥ ¨ ¢ ¤ C A INT IN T T IN T IN T ¢ ¤ C A G G GIN G1 G... 20 § k ¦ ... ¢ C ¤ A ¡¢A@ ¢ C ¤ A ¡¢ A@ ¢ C ¤ A ¡ ¢ A @ ¢ C ¤ A ¡ ¢ A @ ¢ C ¤ A ¡ ¢ A @ ¢ C ¤ A ¡ A ¢ @ ¢ C ¤ A ¡ A ¢ @ ¢ C ¤ A ¡ A ¢ @ CW AU ¢® ¤² ? ? ¥ AU ª ¢® ¨ R ¥ ¨ ¨ @ ¨ ¥ ¡
¥
G D DINT,j GINT,k IN T,j IN T,k INT,j INT,k CDIN T,j CD CGIN T,k CG C1 IN T,j CD C IN T,k CG ... ... § ... ¦1 § ... ¦ §Cm ¦ ... §Cn ¦ ...
To show how much this approach is effective, table 6 displays some revision measures computed on three subsets: the subset S based on 19 selected sectors, the subset S C,IN T that represents the complement of S in INT and the subset S C,IIP that represents the complement of S in IIP. The period analysed is January 2004-December 2007. 11
Table 6: Some measures of revision for the subsets S, S C,IN T and S C,IIP (year-on-year growth rates on raw data; Jan. 2004 - Dec. 2007) Statistics S S C,IN T S C,IIP n 48 48 48 Weights % 32.3 11.5 67.7 88.5 MAR 0.362 0.263 0.159 RMAR 0.100 0.082 0.055 MR 0.263 0.071 0.056 Contribution to MR 0.088 0.030 0.047 0.049 t−value 3.985 1.407 1.766 P-value 0.000 0.166 0.084 A more accurate study of these sectors has revealed interesting elements. Firstly, the subset S is made up of only 19 classes (out of 92 INT classes and 189 total classes) whose weight is 32.3% and 11.5% of INT and IIP weight, respectively. Secondly, the results show that the selected classes have a larger revision size (measured by both MAR and RMAR) with a systematic component (not present in the other subsets at the 5% significance level). With reference to the features of the subset S, its sectors may be very different in terms of either business concentration or production process (on order or not). Moreover, the reasons for revisions can be traced back either to partial information previously provided by respondents (especially for small firms) or to the estimation of the production levels of non respondents (when the first release is disseminated). In the latter case two possible countermeasures could be taken: intensive follow up of specific groups of units (especially for large firms that work on orders) and different methods for the imputation of non responses. Recently some methodological proposals have been considered (Gismondi et al., 2006 and Gismondi and Carone, 2007) and some of them have been already implemented in the production process of IIP. A final consideration concerns the relationship between the revision process and the weighted response rates for the subset S. The following section deals with this issue.
5. Revisions and weighted response rates for S Since the top-down approach implemented to identify S is based on the analysis of both the revision processes and the weighted response rates, the latter are expected to be lower for S than for S C,IN T . Table 7, describing the weighted response rates for such complementary sets, confirms our expectations. Two aspects can be deduced analysing the table: firstly, there is a decreasing trend in the degree of coverage as the base year gets more distant; secondly, the difference between the means of the preliminary and the revised weighted response rates a nearly steady for both the subsets. In order to understand whether this stable gap produces a stable revision process as well, a rolling analysis is implemented on the revisions of the growth rates for the subset S., i.e. the revision measures are computed over different time spans, beginning in January 2004 and lasting 25, 26, . . . , 48 months, till January 2006, February 2006, . . . , December 2007. For the sake of completeness, the same exercise is carried out on the weighted response rates. The results are displayed in the top panels of figure 12
Table 7: Weighted response rates for subsets S and S C,IN T S S C,IN T Estimate Measure 2004 2005 2006 2007 2004 2005 2006 2007 Mean 87.0 86.5 83.5 75.8 88.9 88.6 87.7 82.8 First Min 77.6 78.2 79.3 69.6 86.0 80.0 81.5 74.5 Max 93.0 90.6 91.3 80.5 94.6 93.2 92.3 87.2 Mean 91.0 90.9 87.9 80.5 93.9 93.1 91.1 87.0 Second Min 88.3 88.4 80.6 76.7 89.0 87.8 83.3 81.3 Max 94.5 93.3 92.5 85.9 96.9 94.9 93.3 89.8 4 (the horizontal axes represent the final month of each span). In particular, the plot in the upper right panel confirms both the decreasing trend in the weighted response rates, especially in the last year, and the constant gap between preliminary and revised response rates. Similarly, the plot in the upper left panel shows rather steady (but statistically significant) MR measures, with a slightly decreasing trend up to December 2006. Moreover, the confidence intervals get narrower over longer spans because of either the increasing degrees of freedom or other peculiar features of the revision process. To this aim, the analysis is repeated using moving windows, i.e. over different time spans lasting two years and beginning in February 2004, March 2004, . . . , January 2006. The results are displayed in the bottom panels of figure 4. From the left plot we can deduce that the revision process is larger and much more volatile in the first year (2004), while in the last two years the variability is steady and the revision size grows reflecting the drop in the weighted response rates displayed in the right plot. It is worth noting that also in this exercise the gap between the preliminary rates and the rates after one month keeps nearly constant. To conclude, this exercise shows a satisfying degree of stability in the subset S revisions and, consequently, confirms the usefulness and the reliability of the top-down approach.
6. Conclusions This work shows how revision analysis could be a useful tool from the producer side in order to detect “avoidable” circumstances of revisions. An application to the Italian monthly IIP released by ISTAT is developed. In such context, revision analysis highlights some critical aspects not shown by analysis based only on coverage rates and allows to hint at possible sources of systematic revisions. The analysis is carried out according to a top-down approach based on several steps. Firstly, the overall IIP index is considered and the revisions of raw, working-day adjusted and seasonally adjusted data are investigated. In fact, over a span of few years, raw and working-day adjusted data are expected to have similar features of the revision processes8 . As a consequence, if revisions in working-day adjusted data differ in a significant way from revision in raw data (they are much larger or show a systematic component), this 8
This hypothesis is plausible for industrial production and it cannot be generalized to other economic indicators. An important example is the retail trade index whose calendar effects may reflect changes both in regulations in force and in the features of the distributive system (the growth of large-scale retail trade).
13
Figure 4: Means of revisions and means of weighted coverage rates over rolling and moving spans Means of revisions − Rolling windows
Weighted response rates − Rolling windows
0.6 0.5
92 MR MR − CI
After 1 month Preliminary 90
0.4 88 0.3 86 0.2 84 0.1 82
0 −0.1
80 Mar−06 Jun−06 Sep−06Dec−06Mar−07 Jun−07 Sep−07Dec−07
Mar−06 Jun−06 Sep−06Dec−06Mar−07 Jun−07 Sep−07Dec−07
Means of revisions − Moving windows
Weighted response rates − Moving windows
0.6 0.5
92 MR MR − CI
After 1 month Preliminary 90
0.4 88 0.3 86 0.2 84 0.1 82
0 −0.1
80 Mar−06 Jun−06 Sep−06Dec−06Mar−07 Jun−07 Sep−07Dec−07
Mar−06 Jun−06 Sep−06Dec−06Mar−07 Jun−07 Sep−07Dec−07
could reveal an inappropriate calendar adjustment. As far as revisions of seasonally adjusted data are concerned, on the one hand they reflect revisions both in raw and in working-day adjusted data, on the other hand they embody the effects of two-sided filters, forecast errors and specification of ARIMA models (when a model-based decomposition is utilized). The comparison between raw, working-day adjusted and seasonally adjusted data allows to confine possible critical aspects of the revision process to the raw data estimation process: in particular it emerges that the second release of total IIP corrects on average the first one slightly but systematically upwards. Secondly, the MIGS components are considered. In this step, only revision on raw data after one month are analysed. This analysis points out where the systematic component of the IIP revisions is generated: the intermediate goods. Such a result is confirmed both by the computation of the contribution of each sector to the MR of the overall index and by the calculation of the weighted response rates. Thirdly, given the large number of the INT components, divisions and groups defining this MIGS are considered and their revision processes studied together with: i) the weighted response rates; ii) the average contribution to the MR of INT; iii) the increase in the weighted response rates between the first and the second release; the correlation between 14
the increase in the weighted response rates (from the first to the second release) and the absolute revisions. The sectors having a statistically significant measure MR are, in turn, inspected driving the analysis up to the NACE class level. The final result is the identification of two complementary subsets of the INT sector. One of them, representing the 32.3% of the weight of the total INT aggregate, shows positively biased revisions. These elements are the starting point of further investigations aimed at identifying in a more clear way if the apparently systematic behaviour of the revision process of specific sectors can be traced back to specific features of their treatment (for instance concerning the imputation techniques of missing response data in the early estimates).
References Di Fonzo, T. (2005), “The OECD project on revisions analysis: First elements for discussion”, paper presented at the OECD STESEG Meeting, Paris, France. Gismondi, R. et al. (2006), “La stima della mancate risposte nell’indagine mensile sulla produzione industriale”, paper presented at the ISTAT seminar “La rilevazione mensile della produzione industriale: aggiornamento metodologico e disegno del nuovo sistema informativo”, Rome, Italy. Gismondi, R. and Carone, A. (2007), “Statistical Criteria to Manage Non-respondents’ Intensive Follow Up in Surveys Repeated along Time”, paper presented at the Istat seminar “Stima anticipata di indicatori congiunturali: teoria e applicazioni”, Rome, Italy. Gomez, V. and Maravall, A. (1997), “Programs TRAMO and SEATS: Instruction for the User. Beta version: November 1997”, Banco de Espana. Jenkinson, G. and Stuttard, N. (2004), “Revisions information in ONS first releases”, Economic Trends, 604, pp.70-72. McKenzie, R. and Park S.Y. (2006), “Revisions analysis of the index of industrial production for OECD countries and major non-member economies”, paper presented at the OECD STESWP Meeting, Paris, France. Mehrhoff, J. (2008), “Sources of Revisions of Seasonally Adjusted Real Time Data”, paper presented at the 5th EUROSTAT Colloquium on Modern Tools for Business Cycle Analysis, Luxembourg. Statistics Commission (2004), “Revisions to economic statistics”, Technical Report 17, Statistics Commission.
15