Regression

Ordinary Least Squares and Ridge Regression

class mlpy.RidgeRegression(alpha=0.0)

Ridge Regression and Ordinary Least Squares (OLS).

Initialization.

Parameters:

alpha : float (>= 0.0)

regularization (0.0: OLS)

New in version 2.2.0.

beta()

Return b_1, ..., b_p.

beta0()

Return b_0.

learn(x, y)

Compute the regression coefficients.

Parameters:

x : numpy 2d array (n x p)

matrix of regressors

y : numpy 1d array (n)

response

pred(x)

Compute the predicted response.

Parameters:

x : numpy 2d array (nxp)

matrix of regressors

Returns:

yp : 1d ndarray

predicted response

selected()

Returns the regressors ranking.

Note

The predicted response is computed as:

\hat{y} = \beta_0 + X \boldsymbol\beta

Example (requires matplotlib module):

>>> import numpy as np
>>> import mlpy
>>> import matplotlib.pyplot as plt
>>> x = np.array([[1], [2], [3], [4], [5], [6]]) # p = 1
>>> y = np.array([0.13, 0.19, 0.31, 0.38, 0.49, 0.64])
>>> rr = mlpy.RidgeRegression(alpha=0.0) # OLS
>>> rr.learn(x, y)
>>> y_hat = rr.pred(x)
>>> plt.figure(1)
>>> plt.plot(x[:, 0], y, 'o') # show y
>>> plt.plot(x[:, 0], y_hat) # show y_hat
>>> plt.show()

>>> rr.beta0()
0.0046666666666667078
>>> rr.beta()
array([ 0.10057143])

Kernel Ridge Regression

class mlpy.KernelRidgeRegression(kernel, alpha)

Ridge Regression and Ordinary Least Squares (OLS).

Initialization.

Parameters:alpha : float (> 0.0)

New in version 2.2.0.

learn(x, y)

Compute the regression coefficients.

Parameters:

x : numpy 2d array (n x p)

matrix of regressors

y : numpy 1d array (n)

response

pred(x)

Compute the predicted response.

Parameters:

x : numpy 2d array (n x p)

matrix of regressors

Returns:

yp : 1d ndarray

predicted response

Example (requires matplotlib module):

>>> import numpy as np
>>> import mlpy
>>> import matplotlib.pyplot as plt
>>> x = np.array([[1], [2], [3], [4], [5], [6]]) # p = 1
>>> y = np.array([0.13, 0.19, 0.31, 0.38, 0.49, 0.64])
>>> kernel = mlpy.KernelGaussian(sigma=0.01)
>>> krr = mlpy.KernelRidgeRegression(kernel=kernel, alpha=0.01)
>>> krr.learn(x,y)
>>> y_hat = krr.pred(x)
>>> plt.figure(1)
>>> plt.plot(x[:, 0], y, 'o') # show y
>>> plt.plot(x[:, 0], y_hat) # show y_hat
>>> plt.show()

Least Angle Regression (LAR)

Least Angle Regression is described in [Efron04].

Covariates should be standardized to have mean 0 and unit length, and the response should have mean 0:

\sum_{i=1}^n{x_{ij}} = 0, \hspace{1cm} \sum_{i=1}^n{x_{ij}^2} = 1, \hspace{1cm} \sum_{i=1}^n{y_i} = 0 \hspace{1cm} \mathrm{for} \hspace{0.2cm} j = 1, 2, \dots, p.

class mlpy.Lar(m=None)

LAR.

Initialization.

Parameters:

m : int (> 0)

max number of steps (= number of features selected). If m=None -> m=x.shape[1] in .learn(x, y)

New in version 2.2.0.

beta()

Return b_1, ..., b_p.

learn(x, y)

Compute the regression coefficients.

Parameters:

x : numpy 2d array (nxp)

matrix of regressors

y : numpy 1d array (n)

response

pred(x)

Compute the predicted response.

Parameters:

x : numpy 2d array (nxp)

matrix of regressors

Returns:

yp : 1d ndarray

predicted response

selected()

Returns the regressors ranking.

steps()

Return the number of steps really performed.

LASSO (LARS implementation)

It implements simple modifications of the LARS algorithm that produces Lasso estimates. See [Efron04] and [Tibshirani96].

Covariates should be standardized to have mean 0 and unit length, and the response should have mean 0:

\sum_{i=1}^n{x_{ij}} = 0, \hspace{1cm} \sum_{i=1}^n{x_{ij}^2} = 1, \hspace{1cm} \sum_{i=1}^n{y_i} = 0 \hspace{1cm} \mathrm{for} \hspace{0.2cm} j = 1, 2, \dots, p.

class mlpy.Lasso(m)

LASSO computed with LARS algoritm.

Initialization.

Parameters:

m : int (> 0)

max number of steps.

New in version 2.2.0.

beta()

Return b_1, ..., b_p.

learn(x, y)

Compute the regression coefficients.

Parameters:

x : numpy 2d array (nxp)

matrix of regressors

y : numpy 1d array (n)

response

pred(x)

Compute the predicted response.

Parameters:

x : numpy 2d array (nxp)

matrix of regressors

Returns:

yp : 1d ndarray

predicted response

selected()

Returns the regressors ranking.

steps()

Return the number of steps really performed.

Gradient Descent

class mlpy.GradientDescent(kernel, t, stepsize)

Gradient Descent Method

Initialization.

Parameters:

kernel: kernel object

kernel

t : int (> 0)

number of iterations

stepsize: float

step size

New in version 2.2.0.

learn(x, y)

Compute the regression coefficients.

Parameters:

x : numpy 2d array (n x p)

matrix of regressors

y : numpy 1d array (n)

response

pred(x)

Compute the predicted response.

Parameters:

x : numpy 2d array (n x p)

matrix of regressors

Returns:

yp : 1d ndarray

predicted response

[Efron04]

(1, 2) Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani. Least Angle Regression. Annals of Statistics, 2004, volume 32, pages 407-499.

[Tibshirani96]

Robert Tibshirani. Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., 1996, volume 58, number 1, pages 267-288.