Lecture 3: Lasso Theory — From Slow to Fast Rates & Beyond

This lecture deepens the theoretical analysis of the Lasso estimator for the high-dimensional linear model $Y = X\beta^0 + \varepsilon$ where $p \gg n$. Starting from the recap of the Lasso's behavior under orthonormal design (soft-thresholding) and the "slow rate" prediction bound from Corollary 6.1, the lecture develops the main theoretical machinery: the compatibility condition on the design matrix $X$, the resulting oracle inequality (Theorem 6.1), the much faster prediction and estimation rates that follow, and connections to the restricted eigenvalue condition. The lecture also covers practical aspects such as the regularization path, the Adaptive Lasso, and even a connection to deep neural networks.

1. Recap: Lasso Under Orthonormal Design

To build intuition, recall what happens in the simplest possible setting: when the columns of the design matrix $X$ are orthonormal, meaning $X^T X / n = I_{p \times p}$ (which requires $p \le n$). In this case, the Lasso has a beautiful closed-form solution — it simply applies soft-thresholding to the ordinary least squares (OLS) estimates.

What does this mean intuitively? Each OLS coefficient $Z_j$ gets "shrunk" toward zero. If $|Z_j|$ is smaller than the threshold $\lambda/2$, the Lasso sets that coefficient exactly to zero. If $|Z_j|$ exceeds the threshold, the estimate is pulled toward zero by the fixed amount $\lambda/2$. This is what creates sparsity: small, likely-noisy coefficients get killed off entirely.

1.1 The Bias Problem of Soft-Thresholding

A key observation from the threshold functions plot: the Lasso (soft-thresholding, dotted line) introduces bias even for large coefficients. When $|Z_j|$ is big (so there's strong evidence the coefficient is non-zero), the Lasso still shrinks the estimate by $\lambda/2$. In contrast, hard-thresholding (dashed line) keeps the OLS value for large $|Z_j|$ untouched — it either keeps or kills, with no shrinkage in between. The Adaptive Lasso (solid line) achieves a nice compromise, which we discuss later.

1.2 Hard-Thresholding and the $\ell_0$-Penalty

Hard-thresholding has less bias than soft-thresholding (still biased because $\mathbb{E}[\hat\beta_{\ell_0}] \neq \beta^0$ due to the truncation, but not shrinkage bias). However, the $\ell_0$ penalty makes the optimization problem non-convex and computationally hard (NP-hard in general). Classical criteria like AIC and BIC are actually $\ell_0$-penalized regression problems. Recent progress on mixed-integer programming can handle $p \le 500$, but for truly high-dimensional problems we need the convex Lasso.

2. The Slow Rate: No Conditions on $X$ Needed

The first major theoretical result for the Lasso requires essentially no assumptions on the design matrix $X$ at all.

2.1 What Does This Tell Us?

Taking $\lambda \asymp \sqrt{\log(p)/n}$ (e.g., by choosing $t^2 \asymp \log(p)$), the bound becomes:

This is a "slow rate" because the prediction error decreases like $n^{-1/2}$ rather than $n^{-1}$. Even in the very sparse case $\|\beta^0\|_1 = O(1)$, the rate is $O_P(\sqrt{\log(p)/n})$, which is much worse than the oracle benchmark.

2.2 Benchmark: The OLS Oracle

If an oracle told us exactly which variables are truly non-zero (the support set $S_0 = \{j : \beta_j^0 \ne 0\}$ with $s_0 = |S_0|$), we could run OLS on just those variables and achieve:

This is the "gold standard" — it depends on $s_0$ (number of relevant variables) rather than $\|\beta^0\|_1$, and it decays at rate $1/n$. To close this gap, we need the compatibility condition.

3. The Compatibility Condition

To go beyond the slow rate, we need some assumption on the design matrix $X$. The issue is that if columns of $X$ are too correlated, it becomes impossible to distinguish which variables are truly active. The compatibility condition is the key assumption that makes fast rates possible.

3.1 Why Do We Need a Condition on $X$?

Consider an extreme example: if $X^{(2)} = -X^{(1)}$ (two columns perfectly anti-correlated), then the models $\beta_1 = 1, \beta_2 = 0$ and $\beta_1 = 0, \beta_2 = -1$ produce identical predictions $X\beta$. We cannot tell them apart — the parameter $\beta^0$ is not identifiable. The compatibility condition rules out such pathological situations, but it does so in a subtle and minimal way.

3.2 Intuition Behind the Compatibility Condition

The condition says: for vectors $\beta$ that are "mostly supported on $S_0$" (the cone constraint $\|\beta_{S_0^c}\|_1 \le 3\|\beta_{S_0}\|_1$), the Gram matrix $\hat\Sigma$ doesn't annihilate them. In other words, the prediction $X\beta$ stays large enough relative to the coefficient on $S_0$. Think of it as a restricted version of saying "the smallest eigenvalue of $\hat\Sigma$ is positive" — but only tested on a special cone of vectors rather than all of $\mathbb{R}^p$.

3.3 Relation to the Restricted Eigenvalue Condition

A related but stronger condition is the restricted eigenvalue condition. Define:

The restricted eigenvalue looks at the ratio involving $\|\beta_{S_0}\|_2^2$, whereas compatibility uses $\|\beta_{S_0}\|_1^2 / s_0$. By Cauchy-Schwarz, $\|\beta_{S_0}\|_1^2 \le s_0 \|\beta_{S_0}\|_2^2$, so $\kappa^2(s_0, 3) > 0$ implies $\varphi_0^2 > 0$ but not vice versa. The compatibility condition is weaker and sufficient for the oracle inequality.

4. The Oracle Inequality — The Fast Rate

With the compatibility condition in hand, we arrive at the central result of this lecture.

4.1 Unpacking the Oracle Inequality

This single inequality gives us two results in one:

1. Prediction bound: $\|X(\hat\beta - \beta^0)\|_2^2 / n \le 4\lambda^2 s_0 / \varphi_0^2$
2. $\ell_1$-estimation bound: $\|\hat\beta - \beta^0\|_1 \le 4\lambda s_0 / \varphi_0^2$

4.2 The "Good Event" $\mathcal{T}$

The event $\mathcal{T}$ controls the maximum absolute correlation between the noise $\varepsilon$ and any column of $X$. Think of $\varepsilon^T X^{(j)} / n$ as how much column $j$ "aligns with the noise." We need all of these to be small. For Gaussian or sub-Gaussian errors:

The factor $\log(p)$ appears because we're taking a maximum over $p$ terms (union bound flavor). This is why $\log(p)$ shows up in the rate — it's the price for not knowing which of the $p$ variables are relevant.

4.3 Asymptotic Implications — The Fast Rate

Setting $\lambda = 2\lambda_0 \asymp \sqrt{\log(p)/n}$ and assuming $\varphi_0^2 \ge L > 0$:

5. Summary of Lasso Properties & Required Conditions

The following table (version of Table 2.2 from the book) gives a bird's-eye view of what the Lasso can achieve and what assumptions are needed:

The table reveals a hierarchy of difficulty: prediction is easiest (weakest assumptions), estimation requires more, variable screening is harder, and exact variable selection is the hardest task requiring the strongest conditions on both the design and the signal strength.

6. The Lasso Regularization Path

In practice, we don't pick a single $\lambda$ — we compute the Lasso solution for a whole range of $\lambda$ values. This gives the regularization path $\{\hat\beta(\lambda) : \lambda \ge 0\}$.

6.1 Key Properties

The Lasso regularization path has a remarkable property: it is piecewise linear in $\lambda$. This means we can compute the entire path efficiently. In practice, we compute it over a grid of $\lambda$-values and interpolate.

There is a maximal value $\lambda_{\max} = \max_j |2(X^T Y)_j / n|$ above which the Lasso solution is exactly zero: $\hat\beta(\lambda_{\max}^+) = 0$. This follows from the KKT conditions. As $\lambda$ decreases from $\lambda_{\max}$, variables enter the model one by one.

6.2 Choosing $\lambda$ in Practice

The most common approach is cross-validation (e.g., 10-fold CV). We evaluate the Lasso over a grid of $\lambda$ values (typically equi-spaced on a log-scale from $\lambda_{\max}$ down to a small value) and choose the $\lambda$ that minimizes the cross-validated prediction error. The theory tells us $\lambda$ should scale like $\sqrt{\log(p)/n}$, which is a useful sanity check for the cross-validated choice.

7. Addressing Lasso's Bias: Adaptive Lasso & Hard-Thresholding

The Lasso's soft-thresholding introduces systematic bias: even large coefficients are shrunk toward zero. Two approaches to address this:

7.1 The Adaptive Lasso

The Adaptive Lasso (Zou, 2006) is a two-stage procedure that re-weights the $\ell_1$-penalty using an initial Lasso fit:

The key idea: if the initial estimator $|\hat\beta_{\text{init},j}|$ is large (strong signal), the penalty weight $1/|\hat\beta_{\text{init},j}|$ is small, so that coefficient gets penalized less — meaning less bias. If the initial estimate is zero, then the Adaptive Lasso also sets it to zero (the penalty is $\infty$).

For orthonormal design with initial OLS estimates, the Adaptive Lasso becomes:

The threshold $\lambda/(2|Z_j|)$ decreases for large $|Z_j|$, so large signals are barely touched — much less bias than the constant shift of soft-thresholding.

7.2 Comparing the Three Threshold Functions

Under orthonormal design, the three estimators correspond to three thresholding functions applied to the OLS estimates $Z_j$:

• Soft-thresholding (Lasso): $g(z) = \text{sign}(z)(|z| - \lambda)_+$ — constant shrinkage for all non-zero estimates. Most biased for large signals.
• Hard-thresholding ($\ell_0$-penalty): $g(z) = z \cdot \mathbb{1}(|z| > \sqrt\lambda)$ — zero or unmodified OLS. Least biased but non-convex optimization.
• Adaptive Lasso: $g(z) = \text{sign}(z)(|z| - \lambda/|z|)_+$ — adaptive threshold that vanishes for large $|z|$. Good bias-computation trade-off.

8. Connection to Deep Neural Networks

An important modern connection: in deep neural networks, the prediction from the last layer to the response $Y$ is often based on a regularized linear model. Specifically, if $\varphi(X_i) \in \mathbb{R}^d$ are learned features from the last hidden layer, then the final prediction uses:

and the prediction is $\hat{f}(x) = \hat\beta(\lambda)^T \varphi(x)$. So even in the deep learning world, the Lasso theory developed in this lecture applies to the final linear layer. The features $\varphi(\cdot)$ are learned by the network, but the last step is still penalized linear regression.

9. Key Takeaways

• Without any conditions on $X$: the Lasso achieves a slow prediction rate $O_P(\|\beta^0\|_1 \sqrt{\log(p)/n})$ — useful but suboptimal.
• With the compatibility condition: we get the fast rate $O_P(s_0 \log(p)/n)$ for prediction and an $\ell_1$-estimation error bound — nearly matching the oracle OLS (up to a $\log(p)$ factor).
• The compatibility condition is weaker than requiring a positive minimum eigenvalue of $\hat\Sigma$; it only needs the restricted eigenvalue to be positive on a cone related to $S_0$.
• The Lasso regularization path is piecewise linear; cross-validation is the standard way to choose $\lambda$ in practice.
• The Adaptive Lasso (two-stage Lasso) reduces the bias of the plain Lasso by reweighting penalties, offering sparser models with better estimation properties.
• The hierarchy of Lasso guarantees — slow rate → fast rate → screening → exact selection — requires increasingly strong assumptions on both the design and signal strength.