pyblp.Problem¶

class
pyblp.
Problem
(product_formulations, product_data, agent_formulation=None, agent_data=None, integration=None, distributions=None, epsilon_scale=1.0, costs_type='linear', add_exogenous=True)¶ A BLPtype problem.
This class is initialized with relevant data and solved with
Problem.solve()
. Parameters
product_formulations (Formulation or sequence of Formulation) –
Formulation
configuration or a sequence of up to threeFormulation
configurations for the matrix of demandside linear product characteristics, \(X_1\), for the matrix of demandside nonlinear product characteristics, \(X_2\), and for the matrix of supplyside characteristics, \(X_3\), respectively. If the formulation for \(X_3\) is not specified or isNone
, a supply side will not be estimated. Similarly, if the formulation for \(X_2\) is not specified or isNone
, the logit (or nested logit) model will be estimated.Variable names should correspond to fields in
product_data
. Theshares
variable should not be included in the formulations for \(X_1\) or \(X_2\). Ifshares
is included in the formulation for \(X_3\), care should be taken when solving for equilibrium prices in, for example,ProblemResults.compute_prices()
, since this routine assumes that marginal costs remain constant.The
prices
variable should not be included in the formulation for \(X_3\), but it should be included in the formulation for \(X_1\) or \(X_2\) (or both). Theabsorb
argument ofFormulation
can be used to absorb fixed effects into \(X_1\) and \(X_3\), but not \(X_2\). Characteristics in \(X_2\) should generally be included in \(X_1\). The typical exception is characteristics that are collinear with fixed effects that have been absorbed into \(X_1\).By default, characteristics in \(X_1\) that do not involve
prices
, \(X_1^\text{ex}\), will be combined with excluded demandside instruments (specified below) to create the full set of demandside instruments, \(Z_D\). Any fixed effects absorbed into \(X_1\) will also be absorbed into \(Z_D\). Similarly, characteristics in \(X_3\) that do not involveshares
, \(X_3^\text{ex}\), will be combined with the excluded supplyside instruments to create \(Z_S\), and any fixed effects absorbed into \(X_3\) will also be absorbed into \(Z_S\). Theadd_exogenous
flag can be used to disable this behavior.Warning
Characteristics that involve prices, \(p\), or shares, \(s\), should always be formulated with the
prices
andshares
variables, respectively. If another name is used,Problem
will not understand that the characteristic is endogenous, so it will be erroneously included in \(Z_D\) or \(Z_S\), and derivatives computed with respect to prices or shares will likely be wrong. For example, to include a \(p^2\) characteristic, includeI(prices**2)
in a formula instead of manually constructing and including aprices_squared
variable.product_data (structured arraylike) –
Each row corresponds to a product. Markets can have differing numbers of products. The following fields are required:
market_ids : (object)  IDs that associate products with markets.
shares : (numeric)  Marketshares, \(s\), which should be between zero and one, exclusive. Outside shares should also be between zero and one. Shares in each market should sum to less than one.
prices : (numeric)  Product prices, \(p\).
If a formulation for \(X_3\) is specified in
product_formulations
, firm IDs are also required, since they will be used to estimate the supply side of the problem:firm_ids : (object, optional)  IDs that associate products with firms.
Excluded instruments are typically specified with the following fields:
demand_instruments : (numeric)  Excluded demandside instruments, which, together with the formulated exogenous demandside linear product characteristics, \(X_1^\text{ex}\), constitute the full set of demandside instruments, \(Z_D\). To instead specify the full matrix \(Z_D\), set
add_exogenous
toFalse
.supply_instruments : (numeric, optional)  Excluded supplyside instruments, which, together with the formulated exogenous supplyside characteristics, \(X_3^\text{ex}\), constitute the full set of supplyside instruments, \(Z_S\). To instead specify the full matrix \(Z_S\), set
add_exogenous
toFalse
.
The recommendation in Conlon and Gortmaker (2020) is to start with differentiation instruments of Gandhi and Houde (2017), which can be built with
build_differentiation_instruments()
, and then compute feasible optimal instruments withProblemResults.compute_optimal_instruments()
in the second stage.For guidance on how to construct instruments and add them to product data, refer to the examples in the documentation for the
build_blp_instruments()
andbuild_differentiation_instruments()
functions.If
firm_ids
are specified, custom ownership matrices can be specified as well:ownership : (numeric, optional)  Custom stacked \(J_t \times J_t\) ownership or product holding matrices, \(\mathscr{H}\), for each market \(t\), which can be built with
build_ownership()
. By default, standard ownership matrices are built only when they are needed to reduce memory usage. If specified, there should be as many columns as there are products in the market with the most products. Rightmost columns in markets with fewer products will be ignored.
Note
Fields that can have multiple columns (
demand_instruments
,supply_instruments
, andownership
) can either be matrices or can be broken up into multiple onedimensional fields with column index suffixes that start at zero. For example, if there are three columns of excluded demandside instruments, ademand_instruments
field with three columns can be replaced by three onedimensional fields:demand_instruments0
,demand_instruments1
, anddemand_instruments2
.To estimate a nested logit or random coefficients nested logit (RCNL) model, nesting groups must be specified:
nesting_ids (object, optional)  IDs that associate products with nesting groups. When these IDs are specified,
rho
must be specified inProblem.solve()
as well.
To use certain types of micro moments, product IDs must be specified:
product_ids (object, optional)  IDs that identify individual products within markets. The IDs referenced by
DemographicExpectationMoment
orDiversionProbabilityMoment
must be unique within the relevant markets.
Finally, clustering groups can be specified to account for withingroup correlation while updating the weighting matrix and estimating standard errors:
clustering_ids (object, optional)  Cluster group IDs, which will be used if
W_type
orse_type
inProblem.solve()
is'clustered'
.
Along with
market_ids
,firm_ids
,nesting_ids
,product_ids
,clustering_ids
, andprices
, the names of any additional fields can typically be used as variables inproduct_formulations
. However, there are a few variable names such as'X1'
, which are reserved for use byProducts
.agent_formulation (Formulation, optional) –
Formulation
configuration for the matrix of observed agent characteristics called demographics, \(d\), which will only be included in the model if this formulation is specified. Since demographics are only used if there are demandside nonlinear product characteristics, this formulation should only be specified if \(X_2\) is formulated inproduct_formulations
. Variable names should correspond to fields inagent_data
.agent_data (structured arraylike, optional) –
Each row corresponds to an agent. Markets can have differing numbers of agents. Since simulated agents are only used if there are demandside nonlinear product characteristics, agent data should only be specified if \(X_2\) is formulated in
product_formulations
. If agent data are specified, market IDs are required:market_ids : (object)  IDs that associate agents with markets. The set of distinct IDs should be the same as the set in
product_data
. Ifintegration
is specified, there must be at least as many rows in each market as the number of nodes and weights that are built for the market.
If
integration
is not specified, the following fields are required:weights : (numeric, optional)  Integration weights, \(w\), for integration over agent choice probabilities.
nodes : (numeric, optional)  Unobserved agent characteristics called integration nodes, \(\nu\). If there are more than \(K_2\) columns (the number of demandside nonlinear product characteristics), only the first \(K_2\) will be retained. If any columns of
sigma
inProblem.solve()
are fixed at zero, only the first few columns of these nodes will be used.
The convenience function
build_integration()
can be useful when constructing custom nodes and weights.Note
If
nodes
has multiple columns, it can be specified as a matrix or broken up into multiple onedimensional fields with column index suffixes that start at zero. For example, if there are three columns of nodes, anodes
field with three columns can be replaced by three onedimensional fields:nodes0
,nodes1
, andnodes2
.Along with
market_ids
, the names of any additional fields can be typically be used as variables inagent_formulation
. The exception is the name'demographics'
, which is reserved for use byAgents
.integration (Integration, optional) –
Integration
configuration for how to build nodes and weights for integration over agent choice probabilities, which will replace anynodes
andweights
fields inagent_data
. This configuration is required ifnodes
andweights
inagent_data
are not specified. It should not be specified if \(X_2\) is not formulated inproduct_formulations
.If this configuration is specified, \(K_2\) columns of nodes (the number of demandside nonlinear product characteristics) will be built. However, if
sigma
inProblem.solve()
is left unspecified or specified with columns fixed at zero, fewer columns will be used.distributions (sequence of str, optional) –
Random coefficient distributions. By default, random coefficients in (3) are assumed to be normally distributed. Nondefault distributions can be specified with a list of the following supported strings:
'normal'
(default)  The random coefficient is assumed to be normal.'lognormal'
 The random coefficient is assumed to be lognormal. The coefficient’s column in (3) is exponentiated before being premultiplied by \(X_2\).
The list should have as many strings as there are columns in \(X_2\). Each string determines the distribution of the random coefficient on the corresponding product characteristic in \(X_2\).
A typical example of a lognormal coefficient is one on prices. Implementing this typically involves having a
I(prices)
in the formulation for \(X_2\), and instead of includingprices
in \(X_1\), including a1
in theagent_formulation
. Then the corresponding coefficient in \(\Pi\) will serve as the mean parameter for the lognormal random coefficient on negative prices, \(p_{jt}\).epsilon_scale (float, optional) –
Factor by which the Type I Extreme Value idiosyncratic preference term, \(\epsilon_{ijt}\), is scaled. By default, \(\epsilon_{ijt}\) is not scaled. The typical use of this parameter is to approximate the pure characteristics model of Berry and Pakes (2007) by choosing a value smaller than
1.0
. As this scaling factor approaches zero, the model approaches the pure characteristics model in which there is no idiosyncratic preference term.In practice, this is implemented by dividing \(V_{ijt} = \delta_{jt} + \mu_{ijt}\) by the scaling factor when solving for the mean utility \(\delta_{jt}\). For small scaling factors, this leads to large values of \(V_{ijt}\), which when exponentiated in the logit expression can lead to overflow issues discussed in Berry and Pakes (2007). The safe versions of the contraction mapping discussed in the documentation for
fp_type
inProblem.solve()
(which is used by default) eliminate overflow issues at the cost of introducing fewer (but still common for a small scaling factor) underflow issues. Throughout the contraction mapping, some values of the simulated shares \(s_{jt}(\delta, \theta)\) can underflow to zero, causing the contraction to fail when taking logs. By default,shares_bounds
inProblem.solve()
bounds these simulated shares from below by1e300
, which eliminates these underflow issues at the cost of making it more difficult for iteration routines to converge.With this in mind, scaling epsilon is not supported for nonlinear contractions, and is also not supported when there are nesting groups, since these further complicate the problem. In practice, if the goal is to approximate the pure characteristics model, it is a good idea to slowly decrease the scale of epsilon (e.g., starting with
0.5
, trying0.1
, etc.) until the contraction begins to fail. To further decrease the scale, there are a few things that can help. One is passing a differentIteration
configuration toiteration
inProblem.solve()
, such as'lm'
, which can be robust in this situation. Another is to setpyblp.options.dtype = np.longdouble
when on a system that supports extended precision (seeoptions
for more information about this) and choose a smaller lower bound by configuringshares_bounds
inProblem.solve()
. Ultimately the model will stop being solvable at a certain point, and this point will vary by problem, so approximating the pure characteristics model requires some degree of experimentation.costs_type (str, optional) –
Functional form of the marginal cost function \(\tilde{c} = f(c)\) in (9). The following specifications are supported:
'linear'
(default)  Linear specification: \(\tilde{c} = c\).'log'
 Loglinear specification: \(\tilde{c} = \log c\).
This specification is only relevant if \(X_3\) is formulated.
add_exogenous (bool, optional) –
Whether to add characteristics in \(X_1\) that do not involve prices, \(X_1^\text{ex}\), to the
demand_instruments
field inproduct_data
(including absorbed fixed effects), and similarly, whether to add characteristics in \(X_3\) that do not involve shares, \(X_3^\text{ex}\), to thesupply_instruments
field. This is by defaultTrue
so that only excluded instruments need to be specified.If this is set to
False
,demand_instruments
andsupply_instruments
should specify the full sets of demand and supplyside instruments, \(Z_D\) and \(Z_S\), and fixed effects should be manually absorbed (for example, with thebuild_matrix()
function). This behavior can be useful, for example, when price is not the only endogenous product characteristic over which consumers have preferences. This model could be correctly estimated by manually adding the truly exogenous characteristics in \(X_1\) to \(Z_D\).Warning
If this flag is set to
False
because there are multiple endogenous product characteristics, care should be taken when including a supply side or computing optimal instruments. These routines assume that price is the only endogenous variable over which consumers have preferences.

product_formulations
¶ Formulation
configurations for \(X_1\), \(X_2\), and \(X_3\), respectively. Type
Formulation or sequence of Formulation

agent_formulation
¶ Formulation
configuration for \(d\). Type
Formulation

products
¶ Product data structured as
Products
, which consists of data taken fromproduct_data
along with matrices built according toProblem.product_formulations
. Thedata_to_dict()
function can be used to convert this into a more usable data type. Type
Products

agents
¶ Agent data structured as
Agents
, which consists of data taken fromagent_data
or built byintegration
along with any demographics built according toProblem.agent_formulation
. Thedata_to_dict()
function can be used to convert this into a more usable data type. Type
Agents

unique_market_ids
¶ Unique market IDs in product and agent data.
 Type
ndarray

unique_firm_ids
¶ Unique firm IDs in product data.
 Type
ndarray

unique_product_ids
¶ Unique product IDs in product data.
 Type
ndarray

unique_nesting_ids
¶ Unique nesting group IDs in product data.
 Type
ndarray

distributions
¶ Random coefficient distributions.
 Type
list of str

epsilon_scale
¶ Factor by which the Type I Extreme Value idiosyncratic preference term, \(\epsilon_{ijt}\), is scaled.
 Type
float

costs_type
¶ Functional form of the marginal cost function \(\tilde{c} = f(c)\).
 Type
str

T
¶ Number of markets, \(T\).
 Type
int

N
¶ Number of products across all markets, \(N\).
 Type
int

F
¶ Number of firms across all markets, \(F\).
 Type
int

I
¶ Number of agents across all markets, \(I\).
 Type
int

K1
¶ Number of demandside linear product characteristics, \(K_1\).
 Type
int

K2
¶ Number of demandside nonlinear product characteristics, \(K_2\).
 Type
int

K3
¶ Number of supplyside product characteristics, \(K_3\).
 Type
int

D
¶ Number of demographic variables, \(D\).
 Type
int

MD
¶ Number of demandside instruments, \(M_D\), which is typically the number of excluded demandside instruments plus the number of exogenous demandside linear product characteristics, \(K_1^\text{ex}\).
 Type
int

MS
¶ Number of supplyside instruments, \(M_S\), which is typically the number of excluded supplyside instruments plus the number of exogenous supplyside linear product characteristics, \(K_3^\text{ex}\).
 Type
int

ED
¶ Number of absorbed dimensions of demandside fixed effects, \(E_D\).
 Type
int

ES
¶ Number of absorbed dimensions of supplyside fixed effects, \(E_S\).
 Type
int

H
¶ Number of nesting groups, \(H\).
 Type
int
Examples
Methods
solve
([sigma, pi, rho, beta, gamma, …])Solve the problem.