Treatment of Outliers in Business Surveys

: The Case of Short-term Economic Survey of Enterprises in Japan (Tankan)

September 2010
Atsushi Ishikawa ^*1
Shunsuke Endo ^*2
Tetsuya Shiratori ^*3

Abstract

This paper discusses the appropriate treatment of outliers in Tankan . Under the current population estimate method, if one of the data which greatly diverges from the rest of the data (the so-called"outlier") does not represent the population, the estimates could also greatly diverge from the real value for the population. Since the real picture of the population when the outlier occurred is unknown, it is difficult to choose the optimum method through empirical analyses. Considering 1) the accuracy of Tankan , 2) the maintenance of its"easy to understand" quality, and 3) the reduction of its work load, we assume that outliers in Tankan are those which have extremely large impact on the rate of change from the previous year or the rate of revision from the previous survey of the estimated values for the population. The method of detecting and treating outliers is suitable for the characteristics of Tankan through empirical analyses of actual Tankan data. Specifically, detection method is based on the adjusted"range edit" method, which uses 1 percentile and 99 percentile, so that it can be applied to non-parametric data, for which the incidence of"0" (zero) is very high. The outliers thus detected are regarded as missing values and should be augmented by applying the missing value imputation, which is already being used for Tankan .

Keywords
TANKAN; outlier; stratified sampling; weight; stratum jumpers

We are grateful for helpful discussion and comments from Hiroshi Saigo, Hiroe Tsubaki, Hibiki Ichiue, Seisaku Kameda, Takatoshi Sekine, Kazuo Monma, as well as the research staff at the Bank of Japan. The views expressed in this paper are solely those of the authors and should not be interpreted as reflecting the views of the Bank of Japan.

*1 Research and Statistics Department, Bank of Japan
E-mail : atsushi.ishikawa@boj.or.jp
*2 Bank of Japan
E-mail : shunsuke.endou@boj.or.jp
*3 Research and Statistics Department, Bank of Japan
E-mail : tetsuya.shiratori@boj.or.jp

Notice

Papers in the Bank of Japan Working Paper Series are circulated in order to stimulate discussion and comments. Views expressed are those of authors and do not necessarily reflect those of the Bank.If you have any comment or question on the working paper series, please contact each author.
When making a copy or reproduction of the content for commercial purposes, please contact the Public Relations Department (webmaster@info.boj.or.jp) at the Bank in advance to request permission. When making a copy or reproduction, the source, Bank of Japan Working Paper Series, should explicitly be credited.