Data for replicating the MRW (QJE, 1992) estimation

I show here how to build a database for replicating the estimations of Mankiw, Romer and Weil (QJE, 1992), by extracting the data reproduced in the PDF of the article.

In the article "A Contribution to the Empirics of Economic Growth", Mankiw, Romer and Weil (1992) propose several estimations of the Solow model and of a version augmented with human capital. In this note, I show how to retrieve the data used by these three authors, reproduced in a table at the end of the article (pages 434 to 436), with a small Python script.

We use two libraries, which may have to be installed with the pip command: tabula-py, which extracts tables from PDF documents, and Pandas. Note that tabula-py is only a wrapper around a Java library: a Java runtime environment (version 8 or later) must therefore also be available on the PATH, otherwise the pip installation will succeed but the call to tabula.read_pdf() will fail.

import tabula
import pandas as pd

To download the PDF version of the article we use the requests module:

import requests

response = requests.get('https://eml.berkeley.edu/~dromer/papers/MRW_QJE1992.pdf')

with open('./mrw-1992.pdf', 'wb') as f:
  f.write(response.content)

Without even reading the documentation of the tabula-py library, the table on the first two pages is easily retrieved. The only problem is that the columns (of the DataFrame object returned by tabula.read_pdf()) have to be renamed.

t1 = tabula.read_pdf('./mrw-1992.pdf', pages=28, silent=True)
t1 = t1[0]
t1.rename(columns={'0':'O', '1960':'GDP_1960', '1985':'GDP_1985', 'GDP':'g(GDP)', 'age pop':'g(POP)', 'Unnamed: 0':'I/Y', 'Unnamed: 1':'SCHOOL'}, inplace=True)

t2 = tabula.read_pdf('./mrw-1992.pdf', pages=29, silent=True)
t2 = t2[0]
t2.rename(columns={'0':'O', '1960':'GDP_1960', '1985':'GDP_1985', 'GDP':'g(GDP)', 'age pop':'g(POP)', 'Unnamed: 0':'I/Y', 'Unnamed: 1':'SCHOOL'}, inplace=True)

The last page is a bit more complicated. The tabula.read_pdf() function needs some help to identify the columns (it fails to separate the O and 1960 columns). One can give it the coordinates of the column separators (identified by opening the PDF file in Gimp and setting the rulers, at the top left, to points as the unit of measurement). Even with this extra information, the function returns inconsistent data on the first and last rows (because of the peculiar layout of the table header and of the notes at the bottom of the table). These rows are therefore dropped.

t3 = tabula.read_pdf('./mrw-1992.pdf',pages=30,silent=True, columns=[68, 150, 160, 170, 178, 205, 233, 251, 285, 306], guess=False)
t3 = t3[0][6:43]
t3.rename(columns={'436':'Number', 'QUARTERL':'Country', 'Y':'N', 'JO':'I', 'U':'O', 'RNAL':'GDP_1960', 'OF ECO':'GDP_1985', 'NOM':'g(GDP)', 'ICS':'g(POP)','Unnamed: 0':'I/Y', 'Unnamed: 1':'SCHOOL'}, inplace=True)

Finally, it only remains to concatenate the three DataFrame objects t1, t2 and t3 with the concat function of Pandas, and to save the result in a csv file:

mrwdata = pd.concat([t1,t2,t3], ignore_index=True)
mrwdata.to_csv('./mrw-1992.csv', index=False)