Download the Jupyter Notebook for this section: blp.ipynb

Random Coefficients Logit Tutorial with the Automobile Data¶

:
import pyblp
import numpy as np
import pandas as pd

pyblp.options.digits = 2
pyblp.options.verbose = False
pyblp.__version__
:
'0.8.0'

In this tutorial, we’ll use data from Berry, Levinsohn, and Pakes (1995) to solve the paper’s automobile problem.

Application of Random Coefficients Logit with the Automobile Data¶

This tutorial is similar to the fake cereal tutorial, but exhibits some other features of pyblp:

• Incorporating a supply side into demand estimation.

• Allowing for simple price-income demographic effects.

• Calculating clustered standard errors.

We’ll use pandas to load two sets of data:

1. product_data, which contains prices, shares, and other product characteristics.

2. agent_data, which contains draws from the distribution of heterogeneity.

:
:
market_ids clustering_ids car_ids firm_ids region shares prices hpwt air mpd ... supply_instruments2 supply_instruments3 supply_instruments4 supply_instruments5 supply_instruments6 supply_instruments7 supply_instruments8 supply_instruments9 supply_instruments10 supply_instruments11
0 1971 AMGREM71 129 15 US 0.001051 4.935802 0.528997 0 1.888146 ... 1.055037 0.150499 -2.234214 1.021490 0.955167 0.584095 0.122192 -0.728782 0.447728 2.024221
1 1971 AMHORN71 130 15 US 0.000670 5.516049 0.494324 0 1.935989 ... 1.042628 0.128299 -2.286422 1.046958 0.877064 0.570652 0.131951 -0.692700 0.416704 2.026706
2 1971 AMJAVL71 132 15 US 0.000341 7.108642 0.467613 0 1.716799 ... 1.098263 0.235564 -2.129088 0.974158 1.302888 0.601261 0.087406 -0.725265 0.570155 1.882594
3 1971 AMMATA71 134 15 US 0.000522 6.839506 0.426540 0 1.687871 ... 1.105849 0.250612 -2.119819 0.971471 1.369823 0.602352 0.078418 -0.710436 0.592103 1.846079
4 1971 AMAMBS71 136 15 US 0.000442 8.928395 0.452489 0 1.504286 ... 1.151893 0.343587 -1.969699 0.897435 1.733810 0.631745 0.050603 -0.748933 0.730975 1.760927

5 rows × 32 columns

The product_data contains market IDs, product IDs, firm IDs, shares, prices, a number of product characteristics, and some pre-computed excluded instruments constructed with principal component analysis. The product IDs are called clustering_ids because they will be used to compute clustered standard errors. For more information about the instruments and the example data as a whole, refer to the data module.

The agent_data contains market IDs, integration weights $$w_{it}$$, integration nodes $$\nu_{it}$$, and demographics $$d_{it}$$. Here we use $$I_t = 200$$ equally weighted Monte Carlo draws per market.

In non-example problems, it is usually a better idea to use many more draws, or a more sophisticated Integration configuration such as sparse grid quadrature.

:
:
market_ids weights nodes0 nodes1 nodes2 nodes3 nodes4 income
0 1971 0.005 0.548814 0.457760 0.564690 0.395537 0.392173 9.728478
1 1971 0.005 0.715189 0.376918 0.839746 0.844017 0.041157 7.908957
2 1971 0.005 0.602763 0.702335 0.376884 0.150442 0.923301 11.079404
3 1971 0.005 0.544883 0.207324 0.499676 0.306309 0.406235 17.641671
4 1971 0.005 0.423655 0.074280 0.081302 0.094570 0.944282 12.423995

Setting up the Problem¶

Unlike the fake cereal problem, we won’t absorb any fixed effects in the automobile problem, so the linear part of demand $$X_1$$ has more components. We also need to specify a formula for the random coefficients $$X_2$$, including a random coefficient on the constant, which captures correlation among all inside goods.

The primary new addition to the model relative to the fake cereal problem is that we add a supply side formula for product characteristics that contribute to marginal costs, $$X_3$$. The patsy-style formulas support functions of regressors such as the log function used below.

We stack the three product formulations in order: $$X_1$$, $$X_2$$, and $$X_3$$.

:
product_formulations = (
pyblp.Formulation('1 + hpwt + air + mpd + space'),
pyblp.Formulation('1 + prices + hpwt + air + mpd + space'),
pyblp.Formulation('1 + log(hpwt) + air + log(mpg) + log(space) + trend')
)
product_formulations
:
(1 + hpwt + air + mpd + space,
1 + prices + hpwt + air + mpd + space,
1 + log(hpwt) + air + log(mpg) + log(space) + trend)

The original specification for the automobile problem includes the term $$\log(y_i - p_j)$$, in which $$y$$ is income and $$p$$ are prices. Instead of including this term, which gives rise to a host of numerical problems, we’ll follow Berry, Levinsohn, and Pakes (1999) and use its first-order linear approximation, $$p_j / y_i$$.

The agent formulation for demographics, $$d$$, includes a column of $$1 / y_i$$ values, which we’ll interact with $$p_j$$. To do this, we will treat draws of $$y_i$$ as demographic variables.

:
agent_formulation = pyblp.Formulation('0 + I(1 / income)')
agent_formulation
:
I(1 / income)

As in the cereal example, the Problem can be constructed by combining the product_formulations, product_data, agent_formulation, and agent_data. We’ll also choose the functional form of marginal costs $$c_{jt}$$. A linear marginal cost specification is the default setting, so we’ll need to use the costs_type argument of Problem to employ the log-linear specification used by Berry, Levinsohn, and Pakes (1995).

:
problem = pyblp.Problem(product_formulations, product_data, agent_formulation, agent_data, costs_type='log')
problem
:
Dimensions:
=======================================================
T    N     F    I     K1    K2    K3    D    MD    MS
---  ----  ---  ----  ----  ----  ----  ---  ----  ----
20   2217  26   4000   5     6     6     1    12    18
=======================================================

Formulations:
=====================================================================================
Column Indices:            0          1       2       3          4         5
-----------------------------  --------  ---------  ----  --------  ----------  -----
X1: Linear Characteristics       1        hpwt     air     mpd       space
X2: Nonlinear Characteristics     1       prices    hpwt    air        mpd      space
X3: Log Cost Characteristics      1      log(hpwt)  air   log(mpg)  log(space)  trend
d: Demographics         1/income
=====================================================================================

The problem outputs a table of dimensions:

• $$T$$ denotes the number of markets.

• $$N$$ is the length of the dataset (the number of products across all markets).

• $$F$$ denotes the number of firms.

• $$I = \sum_t I_t$$ is the total number of agents across all markets (200 draws per market times 20 markets).

• $$K_1$$ is the number of linear demand characteristics.

• $$K_2$$ is the number of nonlinear demand characteristics.

• $$K_3$$ is the number of linear supply characteristics.

• $$D$$ is the number of demographic variables.

• $$M_D$$ is the number of demand instruments, including exogenous regressors.

• $$M_S$$ is the number of supply instruments, including exogenous regressors.

The formulations table describes all four formulas for linear characteristics, nonlinear characteristics, cost characteristics, and demographics.

Solving the Problem¶

The only remaining decisions are:

• Choosing $$\Sigma$$ and $$\Pi$$ starting values, $$\Sigma_0$$ and $$\Pi_0$$.

• Potentially choosing bounds for $$\Sigma$$ and $$\Pi$$.

The decisions we will use are:

• Use published estimates as our starting values in $$\Sigma_0$$.

• Interact the inverse of income, $$1 / y_i$$, only with prices, and use the published estimate on $$\log(y_i - p_j)$$ as our starting value for $$\alpha$$ in $$\Pi_0$$.

• Bound $$\Sigma_0$$ to be positive since it is a diagonal matrix where the diagonal consists of standard deviations.

• Constrain the $$p_j / y_i$$ coefficient to be negative. Specifically, we’ll use a bound that’s slightly smaller than zero because when the parameter is exactly zero, there are matrix inversion problems with estimating marginal costs.

When using a routine that supports bounds, Problem chooses some default bounds. These bounds are often very wide, so it’s usually a good idea to set your own more restrictive bounds.

:
initial_sigma = np.diag([3.612, 0, 4.628, 1.818, 1.050, 2.056])
initial_pi = np.c_[[0, -43.501, 0, 0, 0, 0]]
sigma_bounds = (
np.zeros_like(initial_sigma),
np.diag([100, 0, 100, 100, 100, 100])
)
pi_bounds = (
np.c_[[0, -100, 0, 0, 0, 0]],
np.c_[[0, -0.1, 0, 0, 0, 0]]
)

Note that there are only 5 nonzeros on the diagonal of $$\Sigma$$, which means that we only need 5 columns of integration nodes to integrate over these 5 dimensions of unobserved heterogeneity. Indeed, agent_data contains exactly 5 columns of nodes. If we were to ignore the $$\log(y_i - p_j)$$ term (by not configuring $$\Pi$$) and include a term on prices in $$\Sigma$$ instead, we would have needed 6 columns of integration nodes in our agent_data.

A downside of the log-linear marginal costs specification is that nonpositive estimated marginal costs can create problems for the optimization routine when computing $$\log c(\hat{\theta})$$. We’ll use the costs_bounds argument to bound marginal costs from below by a small number.

Finally, as in the original paper, we’ll use W_type and se_type to cluster by product IDs, which were specified as clustering_ids in product_data.

:
results = problem.solve(
initial_sigma,
initial_pi,
sigma_bounds=sigma_bounds,
pi_bounds=pi_bounds,
costs_bounds=(0.001, None),
W_type='clustered',
se_type='clustered'
)
results
:
Problem Results Summary:
==============================================================================================================
GMM   Objective    Projected    Reduced Hessian  Reduced Hessian  Clipped  Weighting Matrix  Covariance Matrix
Step    Value    Gradient Norm  Min Eigenvalue   Max Eigenvalue    Costs   Condition Number  Condition Number
----  ---------  -------------  ---------------  ---------------  -------  ----------------  -----------------
2    +9.9E-02     +1.5E-10        -4.7E-20         +1.1E-03         0         +7.3E+04          +1.5E+09
==============================================================================================================

Cumulative Statistics:
================================================================
Computation  Optimization   Objective   Fixed Point  Contraction
Time       Iterations   Evaluations  Iterations   Evaluations
-----------  ------------  -----------  -----------  -----------
00:06:34         33           37          3131         9917
================================================================

Nonlinear Coefficient Estimates (Robust SEs Adjusted for 999 Clusters in Parentheses):
===================================================================================================
Sigma:      1        prices      hpwt        air         mpd        space     |   Pi:     1/income
------  ----------  --------  ----------  ----------  ----------  ----------  |  ------  ----------
1      +0.0E+00                                                             |    1      +0.0E+00
(+8.4E+00)                                                            |
|
prices   +0.0E+00   +0.0E+00                                                  |  prices   -1.8E+01
|          (+1.6E+00)
|
hpwt    +0.0E+00   +0.0E+00   +3.4E-01                                       |   hpwt    +0.0E+00
(+2.0E+01)                                      |
|
air     +0.0E+00   +0.0E+00   +0.0E+00    +0.0E+00                           |   air     +0.0E+00
(+1.3E+01)                          |
|
mpd     +0.0E+00   +0.0E+00   +0.0E+00    +0.0E+00    +0.0E+00               |   mpd     +0.0E+00
(+4.1E+00)              |
|
space    +0.0E+00   +0.0E+00   +0.0E+00    +0.0E+00    +0.0E+00    +1.3E-01   |  space    +0.0E+00
(+5.6E+00)  |
===================================================================================================

Beta Estimates (Robust SEs Adjusted for 999 Clusters in Parentheses):
==========================================================
1          hpwt        air         mpd        space
----------  ----------  ----------  ----------  ----------
-7.2E+00    +8.1E+00    +3.4E+00    -4.4E-01    +2.4E+00
(+3.9E+00)  (+1.0E+01)  (+6.3E+00)  (+2.2E+00)  (+2.9E+00)
==========================================================

Gamma Estimates (Robust SEs Adjusted for 999 Clusters in Parentheses):
======================================================================
1       log(hpwt)      air       log(mpg)   log(space)    trend
----------  ----------  ----------  ----------  ----------  ----------
+2.6E+00    +6.9E-01    +7.4E-01    -5.3E-01    +1.0E-01    +1.6E-02
(+1.2E-01)  (+9.4E-02)  (+4.5E-02)  (+8.7E-02)  (+1.1E-01)  (+2.5E-03)
======================================================================

There are some discrepancies between our results and the original paper. Our instruments are different, we use different agent data, and the first-order linear approximation to the $$\ln(y_i - p_j)$$ term is different as well.