PLS Classification section part 1 by kjell-stattenacity · Pull Request #126 · aml4td/website

kjell-stattenacity · 2026-08-17T02:03:16Z

Added background and mathematical description for PLS for classification

Added background and mathematical description

topepo · 2026-08-22T16:59:14Z


+## Partial Least Squares {#sec-cls-pls}
+
+As we just saw in @sec-logistic-multicollinearity, multicollinearity poses a fundamental problem for estimating the coefficients of a logistic regression model. The root of the problem is mathematical.  Specifically, solving for the coefficients requires inverting the covariance matrix of the predictors. When two predictors are perfectly correlated, this matrix is singular and cannot be inverted, so no unique set of coefficient estimates exists.  When predictors are highly collinear but not perfectly correlated, the inversion is possible but numerically unstable, which inflates the variance of the coefficient estimates and produces models that are unreliable for both interpretation and prediction. A closely related difficulty arises when the number of predictors exceeds the number of observations; in this case the covariance matrix is singular, and the usual estimation procedure fails. Regularization, as demonstrated above, is one effective solution to these problems.


I updated this paragraph and the next. There was a lot of redundancy with the previous sections and the PLS/PCA sections in the embedding chapter.

topepo · 2026-08-22T17:00:08Z

+
+To describe the method, we first must address a practical matter: the coding of the response. For a two-group problem, the classes are encoded into a set of 0/1 dummy variables. With $C$ classes, the result is a set of $C$ dummy variables, where each sample has a one in the column representing its corresponding class.^[Mathematically, if we know $C - 1$ of the dummy variables, then the value of the last dummy variable is directly implied. Hence, it is also possible to use only $C - 1$ dummy variables.] As a result, the response is represented by a *matrix* of dummy variables, $\boldsymbol{Y}$, rather than by a single vector. Because of this, the problem cannot be solved by the PLS regression approach for a univariate response and must move to the paradigm for a multivariate response.
+
+Briefly, the NIPALS algorithm iteratively seeks to find underlying, or *latent*, relationships among the predictors that are highly correlated with the response. Each iteration of the algorithm assesses the relationship between the predictors ($\boldsymbol{X}$) and the response ($\boldsymbol{Y}$), and numerically summarizes this relationship with a vector of weights ($\boldsymbol{w}$); this vector is also known as a *direction*. The predictor data are then orthogonally projected onto the direction to generate scores ($\boldsymbol{t}$). The scores are used to generate loadings ($\boldsymbol{p}$), which measure the correlation of the score vector to the original predictors. At the end of each iteration, the predictors and the response are "deflated" by subtracting the current estimate of the predictor and response structure, respectively. The deflated predictor and response information are then used to generate the next set of weights, scores, and loadings. These quantities are sequentially stored in matrices $\boldsymbol{W}$, $\boldsymbol{T}$, and $\boldsymbol{P}$, and are used for predicting new samples and computing predictor importance. A thorough explanation of the algorithm can be found in @Geladi1986.


Can we use similar notation to @sec-pca? It uses

$$ \underset{n\times m}{\boldsymbol{X}^*} = \underset{n\times p}{\boldsymbol{X}}\ \underset{p\times m}{\boldsymbol{Q}} $$

I'd like to avoid P for matrices and/or vectors.

It would also be good to add more math details here.

How much of this should we hold back for the regression section?

Can you also mention how to convert the prediction equation to a linear predictor? What is the math to get the corresponding model coefficients?

I updated the neural network section that used a matrix named $\boldsymbol{P}$ to use a different symbol too.

topepo · 2026-08-22T17:01:39Z

+\quad \text{subject to} \quad \|\boldsymbol{w}\| = 1,
+$$
+
+and the corresponding component scores are $\boldsymbol{t}_1 = \boldsymbol{X}\boldsymbol{w}_1$. Subsequent directions are found by applying the same maximization to the deflated predictor and response matrices, subject to the constraint that each new score vector is orthogonal to those already extracted. In the classification setting, where $\boldsymbol{Y}$ is the matrix of class dummy variables, this objective is to find the linear combination of the predictors whose scores have optimal covariance with the classification response. This is how PLS components are constructed using *both* predictor variability and response information.


Sooo is PLSDA the same as just PLS with an indicator outcome matrix?

How do we get valid probability estimates from the model?

topepo · 2026-08-22T17:04:32Z

+
+and the corresponding component scores are $\boldsymbol{t}_1 = \boldsymbol{X}\boldsymbol{w}_1$. Subsequent directions are found by applying the same maximization to the deflated predictor and response matrices, subject to the constraint that each new score vector is orthogonal to those already extracted. In the classification setting, where $\boldsymbol{Y}$ is the matrix of class dummy variables, this objective is to find the linear combination of the predictors whose scores have optimal covariance with the classification response. This is how PLS components are constructed using *both* predictor variability and response information.
+
+Applying PLS in the classification setting with a multivariate response has strong mathematical connections to both canonical correlation analysis and linear discriminant analysis (see @barker2003partial for technical details). Assuming the coding structure described above for the outcome, they showed that the PLS directions in this context are the eigenvectors of a slightly perturbed between-groups covariance matrix (i.e., $\boldsymbol{B}$ from LDA^[The perturbed covariance structure is due to the optimality constraints for the response matrix. @barker2003partial astutely recognized that the response optimality constraint in this setting did not make sense, removed the constraint, and re-solved the problem. Without the response-space constraint, the PLS solution is one that involves exactly the between-groups covariance matrix.]). PLS is therefore seeking to achieve optimal group separation while guided by between-group information. In contrast, PCA seeks to reduce dimensionality using the total variation as directed by the overall covariance matrix of the predictors, which mixes between-group and within-group variation without distinguishing between them.


I like this but I feel like it is out of place since we talk about LDA in the next chapter. Should we include this and the next paragraph?

kjell-stattenacity and others added 3 commits August 16, 2026 22:01

Added PLS background section

febb436

Added background and mathematical description

some updates

c30e445

remove some comments

342268a

topepo reviewed Aug 22, 2026

View reviewed changes

Navigation Menu

PLS Classification section part 1 - #126

PLS Classification section part 1#126
kjell-stattenacity wants to merge 3 commits into
mainfrom
cls-linear-pls-background

kjell-stattenacity commented Aug 17, 2026 •

edited by topepo

Loading

Uh oh!

topepo Aug 22, 2026

Uh oh!

topepo Aug 22, 2026

Uh oh!

topepo Aug 22, 2026

Uh oh!

topepo Aug 22, 2026

Uh oh!

topepo Aug 22, 2026

Uh oh!

topepo Sep 7, 2026

Uh oh!

topepo Aug 22, 2026

Uh oh!

topepo Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		## Partial Least Squares {#sec-cls-pls}

		As we just saw in @sec-logistic-multicollinearity, multicollinearity poses a fundamental problem for estimating the coefficients of a logistic regression model. The root of the problem is mathematical. Specifically, solving for the coefficients requires inverting the covariance matrix of the predictors. When two predictors are perfectly correlated, this matrix is singular and cannot be inverted, so no unique set of coefficient estimates exists. When predictors are highly collinear but not perfectly correlated, the inversion is possible but numerically unstable, which inflates the variance of the coefficient estimates and produces models that are unreliable for both interpretation and prediction. A closely related difficulty arises when the number of predictors exceeds the number of observations; in this case the covariance matrix is singular, and the usual estimation procedure fails. Regularization, as demonstrated above, is one effective solution to these problems.


		To describe the method, we first must address a practical matter: the coding of the response. For a two-group problem, the classes are encoded into a set of 0/1 dummy variables. With $C$ classes, the result is a set of $C$ dummy variables, where each sample has a one in the column representing its corresponding class.^[Mathematically, if we know $C - 1$ of the dummy variables, then the value of the last dummy variable is directly implied. Hence, it is also possible to use only $C - 1$ dummy variables.] As a result, the response is represented by a matrix of dummy variables, $\boldsymbol{Y}$, rather than by a single vector. Because of this, the problem cannot be solved by the PLS regression approach for a univariate response and must move to the paradigm for a multivariate response.

		Briefly, the NIPALS algorithm iteratively seeks to find underlying, or latent, relationships among the predictors that are highly correlated with the response. Each iteration of the algorithm assesses the relationship between the predictors ($\boldsymbol{X}$) and the response ($\boldsymbol{Y}$), and numerically summarizes this relationship with a vector of weights ($\boldsymbol{w}$); this vector is also known as a direction. The predictor data are then orthogonally projected onto the direction to generate scores ($\boldsymbol{t}$). The scores are used to generate loadings ($\boldsymbol{p}$), which measure the correlation of the score vector to the original predictors. At the end of each iteration, the predictors and the response are "deflated" by subtracting the current estimate of the predictor and response structure, respectively. The deflated predictor and response information are then used to generate the next set of weights, scores, and loadings. These quantities are sequentially stored in matrices $\boldsymbol{W}$, $\boldsymbol{T}$, and $\boldsymbol{P}$, and are used for predicting new samples and computing predictor importance. A thorough explanation of the algorithm can be found in @Geladi1986.


		and the corresponding component scores are $\boldsymbol{t}_1 = \boldsymbol{X}\boldsymbol{w}_1$. Subsequent directions are found by applying the same maximization to the deflated predictor and response matrices, subject to the constraint that each new score vector is orthogonal to those already extracted. In the classification setting, where $\boldsymbol{Y}$ is the matrix of class dummy variables, this objective is to find the linear combination of the predictors whose scores have optimal covariance with the classification response. This is how PLS components are constructed using both predictor variability and response information.

		Applying PLS in the classification setting with a multivariate response has strong mathematical connections to both canonical correlation analysis and linear discriminant analysis (see @barker2003partial for technical details). Assuming the coding structure described above for the outcome, they showed that the PLS directions in this context are the eigenvectors of a slightly perturbed between-groups covariance matrix (i.e., $\boldsymbol{B}$ from LDA^[The perturbed covariance structure is due to the optimality constraints for the response matrix. @barker2003partial astutely recognized that the response optimality constraint in this setting did not make sense, removed the constraint, and re-solved the problem. Without the response-space constraint, the PLS solution is one that involves exactly the between-groups covariance matrix.]). PLS is therefore seeking to achieve optimal group separation while guided by between-group information. In contrast, PCA seeks to reduce dimensionality using the total variation as directed by the overall covariance matrix of the predictors, which mixes between-group and within-group variation without distinguishing between them.

Conversation

kjell-stattenacity commented Aug 17, 2026 • edited by topepo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kjell-stattenacity commented Aug 17, 2026 •

edited by topepo

Loading