#+OPTIONS:   H:3 num:nil toc:nil \n:nil @:t ::t |:t ^:nil -:t f:t *:t TeX:t LaTeX:t skip:t d:t tags:not-in-toc creator:t timestamp:nil author:nil title:nil
#+AUTO_TANGLE: t
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="../../../css/stylesheet-blog.css">
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="../../../css/theorems-en.css">
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="../../../fontawesome/css/all.css" />
#+HTML_HEAD: <link rel="stylesheet" href="../../../highlight/styles/base16/solarized-light.min.css">
#+HTML_HEAD: <script src="../../../highlight/highlight.min.js"></script>
#+HTML_HEAD: <script>hljs.highlightAll();</script>
#+LANGUAGE: en-GB
#+STARTUP: latexpreview
#+TITLE: Data for replicating the MRW (QJE, 1992) estimation
#+DATE: January 2023
#+AUTHOR: Stéphane Adjemian
#+EMAIL: stephane.adjemian@univ-lemans.fr
#+PROPERTY: header-args:python :python /tmp/blog-mrw-data/bin/python

#+BEGIN_QUOTE
I show here how to build a database for replicating the estimations of
Mankiw, Romer and Weil (QJE, 1992), by extracting the data reproduced in the
PDF of the article.
#+END_QUOTE

\\
\\
\\

In the article "/A Contribution to the Empirics of Economic Growth/", Mankiw,
Romer and Weil (1992) propose several estimations of the Solow model and of
a version augmented with human capital. In this note, I show how to retrieve
the data used by these three authors, reproduced in a table at the end of the
article (pages 434 to 436), with a small Python script.

We use two libraries, which may have to be installed with the =pip= command:
[[https://pypi.org/project/tabula-py/][tabula-py]], which extracts tables from PDF documents, and [[https://pypi.org/project/pandas/][Pandas]]. Note that
=tabula-py= is only a wrapper around a Java library: a Java runtime
environment (version 8 or later) must therefore also be available on the
=PATH=, otherwise the =pip= installation will succeed but the call to
=tabula.read_pdf()= will fail.

#+BEGIN_SRC bash :results silent :exports none :async t
  python3 -m venv /tmp/blog-mrw-data
  source /tmp/blog-mrw-data/bin/activate
  pip install numpy matplotlib tabula-py pandas requests tabulate
#+END_SRC


#+begin_src python :session mrw-data-en :exports code :results none :mkdirp yes :tangle codes/mrw-data.py
  import tabula
  import pandas as pd
#+end_src

To download the PDF version of the article we use the =requests= module:

#+begin_src python :session mrw-data-en :exports code :results none :mkdirp yes :tangle codes/mrw-data.py
  import requests

  response = requests.get('https://eml.berkeley.edu/~dromer/papers/MRW_QJE1992.pdf')

  with open('./mrw-1992.pdf', 'wb') as f:
    f.write(response.content)
#+end_src

Without even reading the documentation of the =tabula-py= library, the table
on the first two pages is easily retrieved. The only problem is that the
columns (of the =DataFrame= object returned by =tabula.read_pdf()=) have to be
renamed.

#+begin_src python :session mrw-data-en :exports code :results none :mkdirp yes :tangle codes/mrw-data.py
  t1 = tabula.read_pdf('./mrw-1992.pdf', pages=28, silent=True)
  t1 = t1[0]
  t1.rename(columns={'0':'O', '1960':'GDP_1960', '1985':'GDP_1985', 'GDP':'g(GDP)', 'age pop':'g(POP)', 'Unnamed: 0':'I/Y', 'Unnamed: 1':'SCHOOL'}, inplace=True)

  t2 = tabula.read_pdf('./mrw-1992.pdf', pages=29, silent=True)
  t2 = t2[0]
  t2.rename(columns={'0':'O', '1960':'GDP_1960', '1985':'GDP_1985', 'GDP':'g(GDP)', 'age pop':'g(POP)', 'Unnamed: 0':'I/Y', 'Unnamed: 1':'SCHOOL'}, inplace=True)
#+end_src

The last page is a bit more complicated. The =tabula.read_pdf()= function
needs some help to identify the columns (it fails to separate the =O= and
=1960= columns). One can give it the coordinates of the column separators
(identified by opening the PDF file in [[https://www.gimp.org/][Gimp]] and setting the rulers, at the top
left, to =points= as the unit of measurement). Even with this extra
information, the function returns inconsistent data on the first and last
rows (because of the peculiar layout of the table header and of the notes at
the bottom of the table). These rows are therefore dropped.

#+begin_src python :session mrw-data-en :exports code :results none :mkdirp yes :tangle codes/mrw-data.py
  t3 = tabula.read_pdf('./mrw-1992.pdf',pages=30,silent=True, columns=[68, 150, 160, 170, 178, 205, 233, 251, 285, 306], guess=False)
  t3 = t3[0][6:43]
  t3.rename(columns={'436':'Number', 'QUARTERL':'Country', 'Y':'N', 'JO':'I', 'U':'O', 'RNAL':'GDP_1960', 'OF ECO':'GDP_1985', 'NOM':'g(GDP)', 'ICS':'g(POP)','Unnamed: 0':'I/Y', 'Unnamed: 1':'SCHOOL'}, inplace=True)
#+end_src

Finally, it only remains to concatenate the three =DataFrame= objects =t1=,
=t2= and =t3= with the =concat= function of Pandas, and to save the result in
a =csv= file:

#+begin_src python :session mrw-data-en :exports code :results none :mkdirp yes :tangle codes/mrw-data.py
  mrwdata = pd.concat([t1,t2,t3], ignore_index=True)
  mrwdata.to_csv('./mrw-1992.csv', index=False)
#+end_src

#+begin_src python :session mrw-data-en :exports results :results value raw
  from tabulate import tabulate
  tabulate(mrwdata, headers="keys", tablefmt="orgtbl")
#+end_src

#+RESULTS:
|     | Number | Country            | N | I | O | GDP_1960 | GDP_1985 | g(GDP) | g(POP) |  I/Y | SCHOOL |
|-----+--------+--------------------+---+---+---+----------+----------+--------+--------+------+--------|
|   0 |      1 | Algeria            | 1 | 1 | 0 |     2485 |     4371 |    4.8 |    2.6 | 24.1 |    4.5 |
|   1 |      2 | Angola             | 1 | 0 | 0 |     1588 |     1171 |    0.8 |    2.1 |  5.8 |    1.8 |
|   2 |      3 | Benin              | 1 | 0 | 0 |     1116 |     1071 |    2.2 |    2.4 | 10.8 |    1.8 |
|   3 |      4 | Botswana           | 1 | 1 | 0 |      959 |     3671 |    8.6 |    3.2 | 28.3 |    2.9 |
|   4 |      5 | Burkina Faso       | 1 | 0 | 0 |      529 |      857 |    2.9 |    0.9 | 12.7 |    0.4 |
|   5 |      6 | Burundi            | 1 | 0 | 0 |      755 |      663 |    1.2 |    1.7 |  5.1 |    0.4 |
|   6 |      7 | Cameroon           | 1 | 1 | 0 |      889 |     2190 |    5.7 |    2.1 | 12.8 |    3.4 |
|   7 |      8 | CentralAfr. Rep.   | 1 | 0 | 0 |      838 |      789 |    1.5 |    1.7 | 10.5 |    1.4 |
|   8 |      9 | Chad               | 1 | 0 | 0 |      908 |      462 |   -0.9 |    1.9 |  6.9 |    0.4 |
|   9 |     10 | Congo, Peop. Rep.  | 1 | 0 | 0 |     1009 |     2624 |    6.2 |    2.4 | 28.8 |    3.8 |
|  10 |     11 | Egypt              | 1 | 0 | 0 |      907 |     2160 |      6 |    2.5 | 16.3 |      7 |
|  11 |     12 | Ethiopia           | 1 | 1 | 0 |      533 |      608 |    2.8 |    2.3 |  5.4 |    1.1 |
|  12 |     13 | Gabon              | 0 | 0 | 0 |     1307 |     5350 |      7 |    1.4 | 22.1 |    2.6 |
|  13 |     14 | Gambia, The        | 0 | 0 | 0 |      799 |      nan |    3.6 |    nan | 18.1 |    1.5 |
|  14 |     15 | Ghana              | 1 | 0 | 0 |     1009 |      727 |      1 |    2.3 |  9.1 |    4.7 |
|  15 |     16 | Guinea             | 0 | 0 | 0 |      746 |      869 |    2.2 |    1.6 | 10.9 |    nan |
|  16 |     17 | Ivory Coast        | 1 | 1 | 0 |     1386 |     1704 |    5.1 |    4.3 | 12.4 |    2.3 |
|  17 |     18 | Kenya              | 1 | 1 | 0 |      944 |     1329 |    4.8 |    3.4 | 17.4 |    2.4 |
|  18 |     19 | Lesotho            | 0 | 0 | 0 |      431 |     1483 |    6.8 |    1.9 | 12.6 |      2 |
|  19 |     20 | Liberia            | 1 | 0 | 0 |      863 |      944 |    3.3 |      3 | 21.5 |    2.5 |
|  20 |     21 | Madagascar         | 1 | 1 | 0 |     1194 |      975 |    1.4 |    2.2 |  7.1 |    2.6 |
|  21 |     22 | Malawi             | 1 | 1 | 0 |      455 |      823 |    4.8 |    2.4 | 13.2 |    0.6 |
|  22 |     23 | Mali               | 1 | 1 | 0 |      737 |      710 |    2.1 |    2.2 |  7.3 |      1 |
|  23 |     24 | Mauritania         | 1 | 0 | 0 |      777 |     1038 |    3.3 |    2.2 | 25.6 |      1 |
|  24 |     25 | Mauritius          | 1 | 0 | 0 |     1973 |     2967 |    4.2 |    2.6 | 17.1 |    7.3 |
|  25 |     26 | Morocco            | 1 | 1 | 0 |     1030 |     2348 |    5.8 |    2.5 |  8.3 |    3.6 |
|  26 |     27 | Mozambique         | 1 | 0 | 0 |     1420 |     1035 |    1.4 |    2.7 |  6.1 |    0.7 |
|  27 |     28 | Niger              | 1 | 0 | 0 |      539 |      841 |    4.4 |    2.6 | 10.3 |    0.5 |
|  28 |     29 | Nigeria            | 1 | 1 | 0 |     1055 |     1186 |    2.8 |    2.4 |   12 |    2.3 |
|  29 |     30 | Rwanda             | 1 | 0 | 0 |      460 |      696 |    4.5 |    2.8 |  7.9 |    0.4 |
|  30 |     31 | Senegal            | 1 | 1 | 0 |     1392 |     1450 |    2.5 |    2.3 |  9.6 |    1.7 |
|  31 |     32 | Sierra Leone       | 1 | 0 | 0 |      511 |      805 |    3.4 |    1.6 | 10.9 |    1.7 |
|  32 |     33 | Somalia            | 1 | 0 | 0 |      901 |      657 |    1.8 |    3.1 | 13.8 |    1.1 |
|  33 |     34 | S. Africa          | 1 | 1 | 0 |     4768 |     7064 |    3.9 |    2.3 | 21.6 |      3 |
|  34 |     35 | Sudan              | 1 | 0 | 0 |     1254 |     1038 |    1.8 |    2.6 | 13.2 |      2 |
|  35 |     36 | Swaziland          | 0 | 0 | 0 |      817 |      nan |    7.2 |    nan | 17.7 |    3.7 |
|  36 |     37 | Tanzania           | 1 | 1 | 0 |      383 |      710 |    5.3 |    2.9 |   18 |    0.5 |
|  37 |     38 | Togo               | 1 | 0 | 0 |      777 |      978 |    3.4 |    2.5 | 15.5 |    2.9 |
|  38 |     39 | Tunisia            | 1 | 1 | 0 |     1623 |     3661 |    5.6 |    2.4 | 13.8 |    4.3 |
|  39 |     40 | Uganda             | 1 | 0 | 0 |      601 |      667 |    3.5 |    3.1 |  4.1 |    1.1 |
|  40 |     41 | Zaire              | 1 | 0 | 0 |      594 |      412 |    0.9 |    2.4 |  6.5 |    3.6 |
|  41 |     42 | Zambia             | 1 | 1 | 0 |     1410 |     1217 |    2.1 |    2.7 | 31.7 |    2.4 |
|  42 |     43 | Zimbabwe           | 1 | 1 | 0 |     1187 |     2107 |    5.1 |    2.8 | 21.1 |    4.4 |
|  43 |     44 | Afghanistan        | 0 | 0 | 0 |     1224 |      nan |    1.6 |    nan |  6.9 |    0.9 |
|  44 |     45 | Bahrain            | 0 | 0 | 0 |      nan |      nan |    nan |    nan |   30 |   12.1 |
|  45 |     46 | Bangladesh         | 1 | 1 | 0 |      846 |     1221 |      4 |    2.6 |  6.8 |    3.2 |
|  46 |     47 | Burma              | 1 | 1 | 0 |      517 |     1031 |    4.5 |    1.7 | 11.4 |    3.5 |
|  47 |     48 | Hong Kong          | 1 | 1 | 0 |     3085 |    13372 |    8.9 |      3 | 19.9 |    7.2 |
|  48 |     49 | India              | 1 | 1 | 0 |      978 |     1339 |    3.6 |    2.4 | 16.8 |    5.1 |
|  49 |     50 | Iran               | 0 | 0 | 0 |     3606 |     7400 |    6.3 |    3.4 | 18.4 |    6.5 |
|  50 |     51 | Iraq               | 0 | 0 | 0 |     4916 |     5626 |    3.8 |    3.2 | 16.2 |    7.4 |
|  51 |     52 | Israel             | 1 | 1 | 0 |     4802 |    10450 |    5.9 |    2.8 | 28.5 |    9.5 |
|  52 |     53 | Japan              | 1 | 1 | 1 |     3493 |    13893 |    6.8 |    1.2 |   36 |   10.9 |
|  53 |     54 | Jordan             | 1 | 1 | 0 |     2183 |     4312 |    5.4 |    2.7 | 17.6 |   10.8 |
|  54 |     55 | Korea, Rep. of     | 1 | 1 | 0 |     1285 |     4775 |    7.9 |    2.7 | 22.3 |   10.2 |
|  55 |     56 | Kuwait             | 0 | 0 | 0 |    77881 |    25635 |    2.4 |    6.8 |  9.5 |    9.6 |
|  56 |     57 | Malaysia           | 1 | 1 | 0 |     2154 |     5788 |    7.1 |    3.2 | 23.2 |    7.3 |
|  57 |     58 | Nepal              | 1 | 0 | 0 |      833 |      974 |    2.6 |      2 |  5.9 |    2.3 |
|  58 |     59 | Oman               | 0 | 0 | 0 |      nan |    15584 |    nan |    3.3 | 15.6 |    2.7 |
|  59 |     60 | Pakistan           | 1 | 1 | 0 |     1077 |     2175 |    5.8 |      3 | 12.2 |      3 |
|  60 |     61 | Philippines        | 1 | 1 | 0 |     1668 |     2430 |    4.5 |      3 | 14.9 |   10.6 |
|  61 |     62 | Saudi Arabia       | 0 | 0 | 0 |     6731 |    11057 |    6.1 |    4.1 | 12.8 |    3.1 |
|  62 |     63 | Singapore          | 1 | 1 | 0 |     2793 |    14678 |    9.2 |    2.6 | 32.2 |      9 |
|  63 |     64 | Sri Lanka          | 1 | 1 | 0 |     1794 |     2482 |    3.7 |    2.4 | 14.8 |    8.3 |
|  64 |     65 | Syrian Arab Rep.   | 1 | 1 | 0 |     2382 |     6042 |    6.7 |      3 | 15.9 |    8.8 |
|  65 |     66 | Taiwan             | 0 | 0 | 0 |      nan |      nan |      8 |    nan | 20.7 |    nan |
|  66 |     67 | Thailand           | 1 | 1 | 0 |     1308 |     3220 |    6.7 |    3.1 |   18 |    4.4 |
|  67 |     68 | U. Arab Emirates   | 0 | 0 | 0 |      nan |    18513 |    nan |    nan | 26.5 |    nan |
|  68 |     69 | Yemen              | 0 | 0 | 0 |      nan |     1918 |    nan |    2.5 | 17.2 |    0.6 |
|  69 |     70 | Austria            | 1 | 1 | 1 |     5939 |    13327 |    3.6 |    0.4 | 23.4 |      8 |
|  70 |     71 | Belgium            | 1 | 1 | 1 |     6789 |    14290 |    3.5 |    0.5 | 23.4 |    9.3 |
|  71 |     72 | Cyprus             | 0 | 0 | 0 |     2948 |      nan |    5.2 |    nan | 31.2 |    8.2 |
|  72 |     73 | Denmark            | 1 | 1 | 1 |     8551 |    16491 |    3.2 |    0.6 | 26.6 |   10.7 |
|  73 |     74 | Finland            | 1 | 1 | 1 |     6527 |    13779 |    3.7 |    0.7 | 36.9 |   11.5 |
|  74 |     75 | France             | 1 | 1 | 1 |     7215 |    15027 |    3.9 |      1 | 26.2 |    8.9 |
|  75 |     76 | Germany, Fed. Rep. | 1 | 1 | 1 |     7695 |    15297 |    3.3 |    0.5 | 28.5 |    8.4 |
|  76 |     77 | Greece             | 1 | 1 | 1 |     2257 |     6868 |    5.1 |    0.7 | 29.3 |    7.9 |
|  77 |     78 | Iceland            | 0 | 0 | 0 |     8091 |      nan |    3.9 |    nan |   29 |   10.2 |
|  78 |     79 | Ireland            | 1 | 1 | 1 |     4411 |     8675 |    3.8 |    1.1 | 25.9 |   11.4 |
|  79 |     80 | Italy              | 1 | 1 | 1 |     4913 |    11082 |    3.8 |    0.6 | 24.9 |    7.1 |
|  80 |     81 | Luxembourg         | 0 | 0 | 0 |     9015 |      nan |    2.8 |    nan | 26.9 |      5 |
|  81 |     82 | Malta              | 0 | 0 | 0 |     2293 |      nan |      6 |    nan | 30.9 |    7.1 |
|  82 |     83 | Netherlands        | 1 | 1 | 1 |     7689 |    13177 |    3.6 |    1.4 | 25.8 |   10.7 |
|  83 |     84 | Norway             | 1 | 1 | 1 |     7938 |    19723 |    4.3 |    0.7 | 29.1 |     10 |
|  84 |     85 | Portugal           | 1 | 1 | 1 |     2272 |     5827 |    4.4 |    0.6 | 22.5 |    5.8 |
|  85 |     86 | Spain              | 1 | 1 | 1 |     3766 |     9903 |    4.9 |      1 | 17.7 |      8 |
|  86 |     87 | Sweden             | 1 | 1 | 1 |     7802 |    15237 |    3.1 |    0.4 | 24.5 |    7.9 |
|  87 |     88 | Switzerland        | 1 | 1 | 1 |    10308 |    15881 |    2.5 |    0.8 | 29.7 |    4.8 |
|  88 |     89 | Turkey             | 1 | 1 | 1 |     2274 |     4444 |    5.2 |    2.5 | 20.2 |    5.5 |
|  89 |     90 | United Kingdom     | 1 | 1 | 1 |     7634 |    13331 |    2.5 |    0.3 | 18.4 |    8.9 |
|  90 |     91 | Barbados           | 0 | 0 | 0 |     3165 |      nan |    4.8 |    nan | 19.5 |   12.1 |
|  91 |     92 | Canada             | 1 | 1 | 1 |    10286 |    17935 |    4.2 |      2 | 23.3 |   10.6 |
|  92 |     93 | Costa Rica         | 1 | 1 | 0 |     3360 |     4492 |    4.7 |    3.5 | 14.7 |      7 |
|  93 |     94 | Dominican Rep.     | 1 | 1 | 0 |     1939 |     3308 |    5.1 |    2.9 | 17.1 |    5.8 |
|  94 |     95 | El Salvador        | 1 | 1 | 0 |     2042 |     1997 |    3.3 |    3.3 |    8 |    3.9 |
|  95 |     96 | Guatemala          | 1 | 1 | 0 |     2481 |     3034 |    3.9 |    3.1 |  8.8 |    2.4 |
|  96 |     97 | Haiti              | 1 | 1 | 0 |     1096 |     1237 |    1.8 |    1.3 |  7.1 |    1.9 |
|  97 |     98 | Honduras           | 1 | 1 | 0 |     1430 |     1822 |      4 |    3.1 | 13.8 |    3.7 |
|  98 |     99 | Jamaica            | 1 | 1 | 0 |     2726 |     3080 |    2.1 |    1.6 | 20.6 |   11.2 |
|  99 |    100 | Mexico             | 1 | 1 | 0 |     4229 |     7380 |    5.5 |    3.3 | 19.5 |    6.6 |
| 100 |    101 | Nicaragua          | 1 | 1 | 0 |     3195 |     3978 |    4.1 |    3.3 | 14.5 |    5.8 |
| 101 |    102 | Panama             | 1 | 1 | 0 |     2423 |     5021 |    5.9 |      3 | 26.1 |   11.6 |
| 102 |    103 | Trinidad & Tobago  | 1 | 1 | 0 |     9253 |    11285 |    2.7 |    1.9 | 20.4 |    8.8 |
| 103 |    104 | United States      | 1 | 1 | 1 |    12362 |    18988 |    3.2 |    1.5 | 21.1 |   11.9 |
| 104 |    105 | Argentina          | 1 | 1 | 0 |     4852 |     5533 |    2.1 |    1.5 | 25.3 |      5 |
| 105 |    106 | Bolivia            | 1 | 1 | 0 |     1618 |     2055 |    3.3 |    2.4 | 13.3 |    4.9 |
| 106 |    107 | Brazil             | 1 | 1 | 0 |     1842 |     5563 |    7.3 |    2.9 | 23.2 |    4.7 |
| 107 |    108 | Chile              | 1 | 1 | 0 |     5189 |     5533 |    2.6 |    2.3 | 29.7 |    7.7 |
| 108 |    109 | Colombia           | 1 | 1 | 0 |     2672 |     4405 |      5 |      3 |   18 |    6.1 |
| 109 |    110 | Ecuador            | 1 | 1 | 0 |     2198 |     4504 |    5.7 |    2.8 | 24.4 |    7.2 |
| 110 |    111 | Guyana             | 0 | 0 | 0 |     2761 |      nan |    1.1 |    nan | 32.4 |   11.7 |
| 111 |    112 | Paraguay           | 1 | 1 | 0 |     1951 |     3914 |    5.5 |    2.7 | 11.7 |    4.4 |
| 112 |    113 | Peru               | 1 | 1 | 0 |     3310 |     3775 |    3.5 |    2.9 |   12 |      8 |
| 113 |    114 | Surinam            | 0 | 0 | 0 |     3226 |      nan |    4.5 |    nan | 19.4 |    8.1 |
| 114 |    115 | Uruguay            | 1 | 1 | 0 |     5119 |     5495 |    0.9 |    0.6 | 11.8 |      7 |
| 115 |    116 | Venezuela          | 1 | 1 | 0 |    10367 |     6336 |    1.9 |    3.8 | 11.4 |      7 |
| 116 |    117 | Australia          | 1 | 1 | 1 |     8440 |    13409 |    3.8 |      2 | 31.5 |    9.8 |
| 117 |    118 | Fiji               | 0 | 0 | 0 |     3634 |      nan |    4.2 |    nan | 20.6 |    8.1 |
| 118 |    119 | Indonesia          | 1 | 1 | 0 |      879 |     2159 |    5.5 |    1.9 | 13.9 |    4.1 |
| 119 |    120 | New Zealand        | 1 | 1 | 1 |     9523 |    12308 |    2.7 |    1.7 | 22.5 |   11.9 |
| 120 |    121 | Papua New Guinea   | 1 | 0 | 0 |     1781 |     2544 |    3.5 |    2.1 | 16.2 |    1.5 |