7.3 Parameter Estimation

If $Y$ is a binary variable, a logistic model is:

\[ {P}(Y=1 \mid \mathbf{x})=\dfrac{\exp({\beta_{0}+\sum_{k=1}^{K} \beta_{k} x_{k}})}{1+\exp({\beta_{0}+\sum_{k=1}^{K} \beta_{k} x_{k}})} \]

$\beta_k$ is the partial regression coefficient of the predictor $x_k.$
Indicates the mean change in the logarithm of odds by increasing the variable $x_k,$ by one unit, keeping all other variables constant.
For each unit that $x_k,$ is increased, the odds are multiplied by $exp(\beta_k)$.

The parameters of coefficients $\boldsymbol{\beta}=\{\beta_{0}, \beta_{1},\ldots, \beta_{K}\}$ are estimated using the Maximum Likelihood method.

That is, to estimate the coefficients of a logistic regression, numerical algorithms are used to maximize the function:

\[ \begin{aligned} L(y ;(\mathbf{x}, \boldsymbol{\beta})) &=\prod_{i=1}^{n}\left({P}\left(Y_{i}=1 \mid \mathbf{x}_{i}, \boldsymbol{\beta}\right)\right)^{y_{i}}\left(1-{P}\left(Y_{i}=1 \mid \mathbf{x}_{i}, \boldsymbol{\beta}\right)\right)^{1-y_{i}} \\ & \prod_{i=1}^{n}\left(\frac{e^{\beta_{0}+\sum_{k=1}^{K} \beta_{k} x_{i, k}}}{1+e^{\beta_{0}+\sum_{k=1}^{K} \beta_{k} x_{k, i}}}\right)^{y_{i}}\left(\frac{1}{1+e^{\beta_{0}+\sum_{k=1}^{K} \beta_{k} x_{i, k}}}\right)^{1-y_{i}} \end{aligned} \]

$\beta_0$ is the expected value of the logarithm of odds when all predictors are zero. It can be transformed to probability with $(1+exp(\beta_0)/(1+exp(\beta_0))$. The result corresponds to the expected probability of belonging to class 1 when all predictors are zero.
The coefficients $\beta_{k}$ indicate the change in the $(\log \left( \dfrac{\pi}{1-\pi} \right)$ caused by the change by one unit in the value of $x_{k}$, while the $\exp(\beta_{k})$ defines the change in the odds ratio, $(\dfrac{\pi}{1-\pi}),$ caused by the change by one unit in the value of $x_{k}.
If $\beta_{k}$ is positive, $\exp(\beta_{k})$ will be greater than 1, that is, $\dfrac{\pi}{1-\pi}$ will increase.
If $\beta_{k}$ is negative, $\exp(\beta_{k})$ will be smaller than 1, and $\dfrac{\pi}{1-\pi}$ will decrease.
The change in the probability $\pi$ caused by a one-unit change in the value of $x_{k}$ is $\beta_{k}\left(\dfrac{\pi}{1-\pi}\right),$ i.e., it depends not only on the coefficient, but also on the probability level from which the change is measured.

Since the relationship between ${P}(Y=1)$ and $\mathbf{x}$ is not linear, the regression coefficients $\beta_k$ do not represent the change in the probability of $Y$ associated with increasing by one unit of $x_k$.

How much the probability of $Y$ increases per unit of $x_k$ depends on the value of $x_k$ , i.e., the position on the logistic curve in which it is located.

This is a very important difference with respect to the interpretation of the coefficients of a linear regression model.