Skip to main content

Compilation of Experimental Price Indices Using Big Data and Machine Learning: A Comparative Analysis and Validity Verification of Quality Adjustments

August 20, 2018
Nobuhiro Abe*1
Kimiaki Shinozaki*2


This paper compiles experimental price indices for 20 home electrical appliances and digital consumer electronic products using big data obtained from, the largest price comparison website in Japan, and a machine-learning algorithm which pairs legacy and successor products with high precision. In so doing, authors examine the validity of quality adjustment methods by performing comparative analyses on the difference these methods have on price indices. Findings from the analyses are as follows: Indices applied with the Webscraped Prices Comparison Method-- the quality adjustment method newly developed and introduced by the Bank of Japan-- are more cost-effective than those applied with the Hedonic Regression Method which is known to possess high accuracy in index creation. Indices applied with the Matched-Model Method, which is frequently applied to price indices using big data is unable to precisely reflect price increases intended to ensure the profitability often seen in home electronics at time of product turnover. This indicates the significant downward bias in price indices. These findings once again highlight the importance of selecting the appropriate quality adjustment method when compiling price indices.

price index, quality adjustment method, hedonic approach, support vector machine

JEL Classification
C43, C45, E31

This paper was presented at the Meeting of the Group of Experts on Consumer Price Indices held in Geneva, Switzerland on 7-9 May 2018. The authors would like to thank staff members of the Bank of Japan for their useful comments; however, the opinions expressed here, as well as any remaining errors, are those of the authors and should not be ascribed to the Bank.

  1. *1Research and Statistics Department, Bank of Japan
    E-mail :
  2. *2Research and Statistics Department, Bank of Japan
    E-mail :


Papers in the Bank of Japan Working Paper Series are circulated in order to stimulate discussion and comments. Views expressed are those of authors and do not necessarily reflect those of the Bank.
If you have any comment or question on the working paper series, please contact each author. When making a copy or reproduction of the content for commercial purposes, please contact the Public Relations Department ( at the Bank in advance to request permission. When making a copy or reproduction, the source, Bank of Japan Working Paper Series, should explicitly be credited.