Your Career Platform for Big Data

Be part of the digital revolution in the UK and Ireland


View all Jobs

DataCareer Blog

Use your extra time at home (and your data skills) for a good cause: Check out the  Kaggle  COVID-19 Open Research Dataset Challenge. In response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). This dataset is a resource of over 29,000 scholarly articles, including over 13,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. Today we, along with the White House and global health organizations, are asking for your help to develop text and data mining tools that can help the medical community develop answers to high priority scientific questions.
Based on the public interest, media coverage and the general interest in the topic, data science is still on the rise. Google Trends is an excellent tool to analyse the fever curve of the public interest. The screenshot below shows the Google search traffic in the UK for Data Science and Machine Learning over the last 5 years. After years  with a strong rise in interest, the curve has flattened - but is still increasing. We analyze in this blog post the job market dynamics based on job openings from Indeed to get a better understanding of what employers are looking for.  Data overview  We use job openings from Indeed's job search site using a python script based on the Indeed API. We run the following search queries based on existing trends in Data Science today: Data Scientist Data Analytics Data Engineer Big Data Machine Learning Neural and Data Science Deep Learning and Data Science Python and Data Science Researcher and Data Science R and Data Science Mathematics and Data Science Statistic and Data Science. Python and Data Analytics in high demand We obtained 1316 job openings from UK employers at the moment of writing. Their distribution in the selected directions is shown below. As we can see, Python is the leading trend in job offers in the UK. This is because almost every data-related role requires knowledge of Python. It has become the must-have language for data scientists / data analysts. A large number of requests have a query for 'Data Science', which is not surprising since this direction is general and can include all the others. The smallest request numbers belong to the 'Deep Learning' and 'Math' and 'Neural' categories. This is due to their complexity, and the fact that most employers do not include these directions in separate categories. We explain this feature in more detail below. Note that the R, Statistic, and Researcher directions have a good query rating since universities and research institutions are one of the main employers in the sphere of Data Science (in the UK). Also, many large companies conduct their own research programs. Geographic overview: Which places offer the most jobs? The distribution of requests for the UK states shows that most of the requests are concentrated in England (91%), while other states account only for 14% (Wales 2% and Scotland 8%). Note that there are no job offers for Data Science vacancies for Northern Ireland - which might be related to the way Indeed assigns jobs to regions. The same situation remains for the regions – most of the job offers are in those regions where there are large IT-companies, business, production. Not surprisingly, London accounts for more than 50% of the job openings.    Among them, we can highlight: London – 55% Eastern – 12% Scotland –8% South East – 6% South West – 5% North West – 5%. The most popular UK cities for Data Science job offers are London, Cambridge, Edinburgh, and Manchester. So apart from London, when compared to the overall population the two university cities Oxford and Cambridge definitely stand out. Which employers offer the most jobs? The chart below shows the leading employers. Companies with less than 5 job offers are included in the “other” position. Among the top employers, we can highlight: Harnham University of Edinburgh Arup GSK Source Code Personnel Expedia Group University of Oxford. It is necessary to say that Harnham is a recruitment agency specialized in data-related jobs.  As we see, many employers are universities, research institutions.    Overview of junior vs. senior roles Junior and Senior positions usually differ in the following criteria: Experience Hard-skills Level of solving tasks Soft-skills (responsibility, communicability). Let's look at the employers' offers distribution according to these positions. We’ll present graphs that show the distribution of the roles of the Junior and Senior positions. As you can see, most Data Science vacancies in the UK require a Senior level. Senior positions are opened slightly longer, and their number is larger than for other levels. Also, Senior positions are more actual in England and large regions. Conclusion Summing up, we can see that the Data Science job market in the UK covers all the main trends. A large number of offerings is from scientific and research institutions, which explains the popularity of Mathematics, Statistics, and R directions. England and large cities, where large IT-companies are concentrated, usually offer more Data Science vacancies. Comparison of the Junior and Senior positions shows that there are more job offers for Seniors, but at the same time, they remain open longer.
For many machine learning problems with a large number of features or a low number of observations, a linear model tends to overfit and variable selection is tricky. Models that use  shrinkage  such as Lasso and Ridge can improve the prediction accuracy as they reduce the estimation variance while providing an interpretable final model. In this tutorial, we will examine Ridge and Lasso regressions, compare it to the classical linear regression and apply it to a dataset in Python. Ridge and Lasso build on the linear model, but their fundamental peculiarity is regularization. The goal of these methods is to improve the loss function so that it depends not only on the sum of the squared differences but also on the regression coefficients.  One of the main problems in the construction of such models is the correct selection of the regularization parameter. Сomparing to linear regression, Ridge and Lasso models are more resistant to outliers and the spread of data. Overall, their main purpose is to prevent overfitting. The main difference between Ridge regression and Lasso is how they assign a penalty term to the coefficients. We will explore this with our example, so let's start. We will work with the Diamonds dataset, which is freely available online: . It contains the prices and other attributes of almost 54,000 diamonds. We will be predicting the price using the available attributes and compare the results for Ridge, Lasso, and OLS. # Import libraries import numpy as np import pandas as pd # Upload the dataset diamonds = pd.read_csv('diamonds.csv') diamonds.head()     Unnamed: 0 carat cut color clarity depth table price x y z 0 1 0.23 Ideal E SI2 61.5 55.0 326 3.95 3.98 2.43 1 2 0.21 Premium E SI1 59.8 61.0 326 3.89 3.84 2.31 2 3 0.23 Good E VS1 56.9 65.0 327 4.05 4.07 2.31 3 4 0.29 Premium I VS2 62.4 58.0 334 4.20 4.23 2.63 4 5 0.31 Good J SI2 63.3 58.0 335 4.34 4.35 2.75 # Drop the index diamonds = diamonds.drop(['Unnamed: 0'], axis=1) diamonds.head()     carat cut color clarity depth table price x y z 0 0.23 Ideal E SI2 61.5 55.0 326 3.95 3.98 2.43 1 0.21 Premium E SI1 59.8 61.0 326 3.89 3.84 2.31 2 0.23 Good E VS1 56.9 65.0 327 4.05 4.07 2.31 3 0.29 Premium I VS2 62.4 58.0 334 4.20 4.23 2.63 4 0.31 Good J SI2 63.3 58.0 335 4.34 4.35 2.75 # Print unique values of text features print(diamonds.cut.unique()) print(diamonds.clarity.unique()) print(diamonds.color.unique()) ['Ideal' 'Premium' 'Good' 'Very Good' 'Fair'] ['SI2' 'SI1' 'VS1' 'VS2' 'VVS2' 'VVS1' 'I1' 'IF'] ['E' 'I' 'J' 'H' 'F' 'G' 'D'] As you can see, there are a finite number of variables, so we can transform these categorical variables  to  numerical variables. # Import label encoder from sklearn.preprocessing import LabelEncoder categorical_features = ['cut', 'color', 'clarity'] le = LabelEncoder() # Convert the variables to numerical for i in range(3): new = le.fit_transform(diamonds[categorical_features[i]]) diamonds[categorical_features[i]] = new diamonds.head()     carat cut color clarity depth table price x y z 0 0.23 2 1 3 61.5 55.0 326 3.95 3.98 2.43 1 0.21 3 1 2 59.8 61.0 326 3.89 3.84 2.31 2 0.23 1 1 4 56.9 65.0 327 4.05 4.07 2.31 3 0.29 3 5 5 62.4 58.0 334 4.20 4.23 2.63 4 0.31 1 6 3 63.3 58.0 335 4.34 4.35 2.75 Before building the models, let's first scale data. Lasso and Ridge put constraints on the size of the coefficients associated  to  each variable. But, this value depends on the magnitude of each variable and it is  therefore  necessary to center and reduce, or standardize, the variables. # Import StandardScaler from sklearn.preprocessing import StandardScaler # Create features and target matrixes X = diamonds[['carat', 'depth', 'table', 'x', 'y', 'z', 'clarity', 'cut', 'color']] y = diamonds[['price']] # Scale data scaler = StandardScaler() X = scaler.transform(X) Now, we can basically build the Lasso and Ridge models. But for now, we will train it on the whole dataset and look at an R-squared score and on the model coefficients. Note, that we are not setting the alpha, it is defined as 1. # Import linear models from sklearn import linear_model from sklearn.metrics import mean_squared_error # Create lasso and ridge objects lasso = linear_model.Lasso() ridge = linear_model.Ridge() # Fit the models, y), y) # Print scores, MSE, and coefficients print("lasso score:", lasso.score(X, y)) print("ridge score:",ridge.score(X, y)) print("lasso MSE:", mean_squared_error(y, lasso.predict(X))) print("ridge MSE:", mean_squared_error(y, ridge.predict(X))) print("lasso coef:", lasso.coef_) print("ridge coef:", ridge.coef_) lasso score: 0.8850606039595762 ridge score: 0.8850713120355513 lasso MSE: 1829298.919415987 ridge MSE: 1829128.4968064611 lasso coef: [ 5159.45245224 -217.84225841 -207.20956411 -1250.0126333 16.16031486 -0. 496.17780105 72.11296318 -451.28351376] ridge coef: [[ 5.20114712e+03 -2.20844296e+02 -2.08496831e+02 -1.32579812e+03 5.36297456e+01 -1.67310953e+00 4.96434236e+02 7.26648505e+01 -4.53187286e+02]] These two models give very similar results. So, we will split the data into training and test sets, build Ridge and Lasso, and choose the regularization parameter with the help of GridSearch. For that, we have to define the set of parameters for GridSearch. In this case, the models with the highest R-squared score will give us the best parameters. # Make necessary imports, split data into training and test sets, and choose a set of parameters from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV import warnings warnings.filterwarnings("ignore") X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=101) parameters = {'alpha': np.concatenate((np.arange(0.1,2,0.1), np.arange(2, 5, 0.5), np.arange(5, 25, 1)))} linear = linear_model.LinearRegression() lasso = linear_model.Lasso() ridge = linear_model.Ridge() gridlasso = GridSearchCV(lasso, parameters, scoring ='r2') gridridge = GridSearchCV(ridge, parameters, scoring ='r2') # Fit models and print the best parameters, R-squared scores, MSE, and coefficients, y_train), y_train), y_train) print("ridge best parameters:", gridridge.best_params_) print("lasso best parameters:", gridlasso.best_params_) print("ridge score:", gridridge.score(X_test, y_test)) print("lasso score:", gridlasso.score(X_test, y_test)) print("linear score:", linear.score(X_test, y_test)) print("ridge MSE:", mean_squared_error(y_test, gridridge.predict(X_test))) print("lasso MSE:", mean_squared_error(y_test, gridlasso.predict(X_test))) print("linear MSE:", mean_squared_error(y_test, linear.predict(X_test))) print("ridge best estimator coef:", gridridge.best_estimator_.coef_) print("lasso best estimator coef:", gridlasso.best_estimator_.coef_) print("linear coef:", linear.coef_) ridge best parameters: {'alpha': 24.0} lasso best parameters: {'alpha': 1.5000000000000002} ridge score: 0.885943268935384 lasso score: 0.8863841649966073 linear score: 0.8859249267960946 ridge MSE: 1812127.0709091045 lasso MSE: 1805122.1385342914 linear MSE: 1812418.4898094584 ridge best estimator coef: [[ 5077.4918518 -196.88661067 -208.02757232 -1267.11393653 208.75255168 -91.36220706 502.04325405 74.65191115 -457.7374841 ]] lasso best estimator coef: [ 5093.28644126 -207.62814092 -206.99498254 -1234.01364376 67.46833024 -0. 501.11520439 73.73625478 -457.08145762] linear coef: [[ 5155.92874335 -208.70209498 -208.16287626 -1439.0942139 243.82503796 -28.79983655 501.31962765 73.93030707 -459.94636759]] Our score raises a little, but with these values of alpha, there is only a small difference. Let's build coefficient plots to see how the value of alpha influences the coefficients of both models. # Import library for visualization import matplotlib.pyplot as plt coefsLasso = [] coefsRidge = [] # Build Ridge and Lasso for 200 values of alpha and write the coefficients into array alphasLasso = np.arange (0, 20, 0.1) alphasRidge = np.arange (0, 200, 1) for i in range(200): lasso = linear_model.Lasso(alpha=alphasLasso[i]), y_train) coefsLasso.append(lasso.coef_) ridge = linear_model.Ridge(alpha=alphasRidge[i]), y_train) coefsRidge.append(ridge.coef_[0]) # Build Lasso and Ridge coefficient plots plt.figure(figsize = (16,7)) plt.subplot(121) plt.plot(alphasLasso, coefsLasso) plt.title('Lasso coefficients') plt.xlabel('alpha') plt.ylabel('coefs') plt.subplot(122) plt.plot(alphasRidge, coefsRidge) plt.title('Ridge coefficients') plt.xlabel('alpha') plt.ylabel('coefs') As a result, you can see that when we raise the alpha in Ridge regression, the magnitude of the coefficients decreases, but never attains zero. The same scenario in Lasso influences less on the large coefficients, but the small ones Lasso reduces to zeroes. Therefore Lasso can also be used to determine which features are important to us and keeps the features that may influence the target variable, while Ridge regression gives uniform penalties to all the features and in such way reduces the model complexity and prevents multicollinearity. Now, it’s your turn!
View all blog posts