On L1q Regularized Regression

Authors: Han Liu and Jian Zhang

Presented by Jun Liu

Problem (1)

Problem (2)

The number of groups is much larger than the number of samples

Outline

• Proposition 2.1, 2.2 (Subgradient, linearly dependent)

• Definition 2.4-2.7 (Properties to be established)

• Theorem 3.1 (Variable Selection Consistency)

• Lemma 4.1 (Technical lemma)

• Assumption 1, Theorem 4.3 (Consistency, linear model)

• Assumption 2. Theorem 4.5 (Inequality, misspecified model)

• Assumption 4, Theorem 5.1 (Risk consistency)

We want to find the such that the q’-norm of is either equal to a constant value (for nonzero groups) or bounded (for zero groups)

Scale invariant, Sign Preserving

Scale invariant,

Sign Preserving

Scale invariant, Sign Preserving

Hint

This result is similar to the Lasso. The key is that , so that any m>n columns of X are linearly dependent.

gj, j=2, …, s is not changed

Outline

• Proposition 2.1, 2.2 (Subgradient, linear dependent)

• Definition 2.4-2.7 (Properties to be established)

• Theorem 3.1 (Variable Selection Consistency)

• Lemma 4.1 (Technical lemma)

• Assumption 1, Theorem 4.3 (Consistency, linear model)

• Assumption 2. Theorem 4.5 (Inequality, misspecified model)

• Assumption 4, Theorem 5.1 (Risk consistency)

Outline

• Proposition 2.1, 2.2 (Subgradient, linear dependent)

• Definition 2.4-2.7 (Properties to be established)

• Theorem 3.1 (Variable Selection Consistency)

• Lemma 4.1 (Technical lemma)

• Assumption 1, Theorem 4.3 (Consistency, linear model)

• Assumption 2. Theorem 4.5 (Inequality, misspecified model)

• Assumption 4, Theorem 5.1 (Risk consistency)

Key Points in the Proof

• Objective

• Two parts

• Tools

Proof by construction Solution is not unique

Part 1

Part 1

Part 1

Part 2

Part 2

Part 2

Outline

• Proposition 2.1, 2.2 (Subgradient, linear dependent)

• Definition 2.4-2.7 (Properties to be established)

• Theorem 3.1 (Variable Selection Consistency)

• Lemma 4.1 (Technical lemma)

• Assumption 1, Theorem 4.3 (Consistency, linear model)

• Assumption 2. Theorem 4.5 (Inequality, misspecified model)

• Assumption 4, Theorem 5.1 (Risk consistency)

Outline

• Proposition 2.1, 2.2 (Subgradient, linear dependent)

• Definition 2.4-2.7 (Properties to be established)

• Theorem 3.1 (Variable Selection Consistency)

• Lemma 4.1 (Technical lemma)

• Assumption 1, Theorem 4.3 (Consistency, linear model)

• Assumption 2. Theorem 4.5 (Inequality, misspecified model)

• Assumption 4, Theorem 5.1 (Risk consistency)