Regularization is a crucial concept in machine learning used to prevent overfitting, which happens when a model performs well on the training data but poorly on unseen or testing data due to excessive complexity or variance.
In this blog, we will learn about a regularization technique named Lasso regression in r, its pros-cons, implementation, and use case of this algorithm.
What is Lasso regression in R?
Lasso Regression stands for Least Absolute Shrinkage and Selection Operator. From the name itself, we can understand what it does, it helps shrink or reduce the importance of some features and even remove the ones that don’t matter much. This makes the model simpler and easier to understand.
It is a special type of linear regression that adds a penalty to the model. This penalty is based on the absolute values of the coefficients (the numbers in front of each feature). The idea is to keep only the most important features and reduce the rest — sometimes all the way down to zero. This process is called L1 regularization.
Lasso is useful when:
- You have too many features in your dataset.
- Some of those features are very similar to each other (high multicollinearity).
- You want to automatically select the best features and ignore the unimportant ones.
Although Lasso is similar to Ridge Regression (which also adds a penalty), the difference is:
- Ridge shrinks the coefficients but doesn’t make them exactly zero.
- Lasso can shrink coefficients all the way to zero, removing them from the model completely.
In short, Lasso Regression helps prevent overfitting, simplifies the model, and selects only the most useful features — making it a great choice when working with complex data.
The mathematical formulation of Lasso regression
In this algorithm, regularization helps to improve the model by adding a small extra term, called a “penalty,” to the equation we get from training the model. This helps the model perform better on new, unseen data (testing data) by keeping it from relying too much on certain features (predictor variables).
Regularization doesn’t remove any features from the model—it keeps the number of features the same—but it makes the importance (or weight) of each feature smaller. This is done by shrinking the size of the coefficients (the numbers that multiply the features).
We can use different types of regression methods with regularization, like Ridge or Lasso regression, to solve this kind of problem.
Let us now see how we are going to represent the Lasso regression in mathematical terms. The mathematical equation of the Lasso regression can be implied through this equation:
Cost = Residual sum of squares (RSS) + λ * ( Sum of the squares of the weights ),
where this can be represented in the form of an equation as:
In this equation,
λ denotes the shrinkage performed
If λ = 0 denotes that all of the features are being considered in the equation, this is equivalent to that of the linear regression in which only the residual sum of squares is taken for building a predictive model.
And if λ = ∞ will denote that there is no feature to be considered, which in simple terms means that as λ closes to ∞, the features will be eliminated.
Finally, the variance of the model will increase with the decrease in the value of λ, and the bias will increase will the increase in the value of λ.
Difference between lasso regression and ridge regression
Earlier in this blog, we discussed that the lasso regression algorithm works by adding the penalty, and the ridge regression algorithm also does the same. So, what is the difference between the working of these two regression algorithms?
The answer is simple, if a regression model uses the L1 regularization technique to add the penalty, then it is the Lasso regression, and if the model uses the L2 regularization technique to perform the tasks of adding the penalty, then that algorithm is the Ridge regression algorithm.
The L1 regularization technique adds the penalty equivalent to that of the absolute value of the coefficient’s magnitude. The L1 regularization results in the sparse models with a few numbers of coefficients where some of them among these few may become zero and, as a result, will be eliminated from the model. In the case where the values are closer to zero, it will result in larger penalties.
Implementation of Lasso regression
To apply the Lasso regression algorithm in accordance with the machine learning problem, we can do it with the help of some simple steps:
- The first step to applying the Lasso regression in simple terms is to import all the required libraries and then import the dataset on which we are going to work.
- The second step is to analyze the dataset, and then we divide the dataset into the training and the testing dataset, where the training dataset will be used for training the model, and the testing of the performance of the models will be done through the test dataset. The scaling of the training and testing dataset will also be performed here to check the fitting of the dataset.
- The third step is to perform the exploratory data analysis that will help us in studying the background of the dataset and will help us in calculating the mean, median, quartiles, etc.
- Finally, we can apply the Lasso regression algorithm using the commands and directions being implemented in the below implementation. After that, we can use the various testing parameters such as the RMSE values and the R square values, which will help us in knowing the efficiency of the algorithm. We can also calculate the mean validation score to know the efficiency of the model.
Uses of the Lasso regression algorithm
- Lasso regression is a modified form of linear regression used to solve complex problems.
- It is mainly used when regularization is needed to improve model performance.
- Helps select a subset of predictors that minimizes prediction error.
- Works well when only a few predictors are important and others are close to zero.
- Shrinks coefficients toward zero, improving performance on new data.
- Reduces overfitting, allowing the use of more complex models safely.
- Can be used for classification tasks through generalized linear models.
Feature selection in the Lasso regression model
Lasso regression is very efficient for the purposes of using feature selection methods and therefore is an important machine learning algorithm when we need to use the regularization methods. In this lasso regression model, we need to calculate the cost function, which can be done with the help of the equation provided above. In the above mathematical equation, we calculate the cost function, and then the regression line is obtained through this.
Implementation of Lasso regression
The first step is to install glmnet and load the library.
install.packages("glmnet")
library(glmnet)The second step is to load the dataset.
df <- data(mtcars)
head(df)The third step is to load the features and the target variables.
target <- mtcars$mpg
features <- as.matrix(mtcars[, -which(names(mtcars) == "mpg")]) Next implement the lasso regression and plot the lasso graph.
lasso<- glmnet(x = features, y = target, alpha = 1)
# Plot coefficient paths
plot(lasso, xvar = "lambda", label = TRUE)cv_lasso <- cv.glmnet(x = features, y = target, alpha = 1)
# Plot cross-validation error
plot(cv_lasso)# Best lambda value
best_lambda <- cv_lasso$lambda.min
print(paste("Best lambda:", best_lambda))coef(cv_lasso, s = "lambda.min")coef(cv_lasso, s = "lambda.min")Advantages and disadvantages of using the Lasso regression model
Advantages:
- The Lasso regression can be efficiently used for regularization purposes, and therefore it can avoid overfitting and can be applied even if the number of features is greater than that of the number of data.
- The Lasso regression model can perform feature selection and is faster when compared in terms of fitting and inference.
- The results obtained through the Lasso regression model are much better than the other methods of automatic variable selection like that of the forward, backward, and stepwise variable selection methods.
Disadvantages:
- One of the main problems that occur while using the lasso regularization model is that the Lasso model is automatic and therefore avoids the case where the data analyst does need to think, and hence this model works without the guidance of the analyst.
- Being automatic is why the lasso regularization model produces unnecessary models and does not follow any hierarchy principles.
- The lasso regression model ignores the non-significant variable, which may not be of any importance.
- The models which are being selected by the lasso model may or may not be stable.
- In case when we are using the lasso regression model and we have a highly correlated feature, then the lasso model may select a part of them or one of them, and this result depends on the implementation of the model.
Conclusion
In this blog, we have learned about what is lasso regression, implementation in r programming, use cases, advantages, and disadvantages.
If you like the article and would like to support me, make sure to:
