Articles of regression

Asymptotic Distribution of the LS Estimator using an Extra Ratio of the Coefficients

Assume $\boldsymbol{y}=\boldsymbol{\iota}\beta_1+\boldsymbol{x}\beta_2+\boldsymbol{u}$ where $\boldsymbol\iota$ is the n-vector of ones and $\{u_i\}$ are i.i.d. with $E(u_i)=0$ and $E(u_i^2)=\sigma^2$. Now, assume that $\boldsymbol{x’x}/n\to c>0$ and $\boldsymbol{\iota’x}/n\to 0$ as $n\to\infty.$ Suppose there is an estimator $\hat\gamma$ (independent from $\boldsymbol{u}$) for the ratio of the coefficients $\gamma=\beta_1/\beta_2$, and it follows that $$ \sqrt{n}(\hat\gamma-\gamma)\overset{A}{\sim}\mathcal{N}(0,\lambda^2). $$ Define $$ \hat\beta_2=\frac{(\boldsymbol{x}+\boldsymbol{\iota}\hat\gamma)’\boldsymbol{y}}{(\boldsymbol{x}+\boldsymbol{\iota}\hat\gamma)'(\boldsymbol{x}+\boldsymbol{\iota}\hat\gamma)}. $$ What […]

How to work out orthogonal polynomials for regression model

I put this question here as it has a pure maths element to it, even though it has a statistical twist. Basically, I have been given the table of data: $$\begin{matrix} i & \mathrm{Response} \, y_i & \mathrm{Covariate} \, x_i \\ 1 & -1 & -0.5 \\ 2 & -0.25 &-0.25 \\ 3 & 1 […]

derivation of simple linear regression parameters

I know there are some proof in the internet, but I attempted to proove the formulas for the intercept and the slope in simple linear regression using Least squares, some algebra, and partial derivatives (although I might want to do it wituout partials if it’s easier). I’ve posted my attempt below. I don’t know what […]

Fit exponential with constant

I have data whic would fit to an exponential function with a constant. So y=aexp(bt) + c Now I can solve an exponential without a constant using least square by taking log of y and making the whole equation linear. Is it possible to use least square to solve it with a constant too ( […]

Why use the kernel trick in an SVM as opposed to just transforming the data?

Why use the kernel trick in a support vector machine as opposed to just transforming the data and then using a linear classifier? Certainly, we’ll approximately double the amount of memory required to hold the data, original plus transformed, but beyond that it seems like the amount of computation remains about the same. What are […]

Why is minimizing least squares equivalent to finding the projection matrix $\hat{x}=A^Tb(A^TA)^{-1}$?

I understand the derivation for $\hat{x}=A^Tb(A^TA)^{-1}$, but I’m having trouble explicitly connecting it to least squares regression. So suppose we have a system of equations: $A=\begin{bmatrix}1 & 1\\1 & 2\\1 &3\end{bmatrix}, x=\begin{bmatrix}C\\D\end{bmatrix}, b=\begin{bmatrix}1\\2\\2\end{bmatrix}$ Using $\hat{x}=A^Tb(A^TA)^{-1}$, we know that $D=\frac{1}{2}, C=\frac{2}{3}$. But this is also equivalent to minimizing the sum of squares: $e^2_1+e^2_2+e^2_3 = (C+D-1)^2+(C+2D-2)^2+(C+3D-2)^2$. I […]

Machine Learning: Linear Regression models

I’m currently in a course learning about neural networks and machine learning, and I came across these two formulas in this textbook page on linear regression: 1) $y(x) = a + bx$ and 2) $y(x) = w^{T}\phi(x)$ What is the difference between these two formulas? How are they related? They seem to perform the same […]

Fitting an exponential function to data

I have a noisy data set (the grey line in the graph below) that corresponds roughly to $y=m(1-2^{-x/k})$ where m and k are unknown constants. How can I determine the best-fit value of m and k? I can get an approximate value for k by guessing m and then doing linear regression on $-\log_2(1-y/m)$… by […]

Deriving cost function using MLE :Why use log function?

I am learning machine learning from Andrew Ng’s open-class notes and I am trying to understand how the cost function for the logistic regression is derived. I will start with the cost function for linear regression and then get to my question with logistic regression. (Btw a similar question was asked here, which answers […]

Two dimensional (discrete) orthogonal polynomials for regression

This question How to work out orthogonal polynomials for regression model and the answer explain how to build orthogonal polynomials for regression. However they only consider one dimensional functions. How can we use (discrete) orthogonal polynomials for regression with two dimensional functions (i.e., $z = f(x, y)$)?