Articles of least squares

Unclear about matrix calculus in least squares regression

The loss function of a Least Squares Regression is defined as (for example, in this question) : $L(w) = (y – Xw)^T (y – Xw) = (y^T – w^TX^T)(y – Xw)$ Taking the derivatives of the loss w.r.t. the parameter vector $w$: \begin{align} \frac{d L(w)}{d w} & = \frac{d}{dw} (y^T – w^TX^T)(y – Xw) \\ […]

Least square best solution

The statement is “This least square problem can be solved efficiently, when A is of full rank. We can prove that the best vector has to satisfy the equation (A T A)x = A T b.” Can anybody explain this?

Checking my understanding of projection onto subspace vs least square approximation

I have trouble understanding the topic of projection vs. least square approximation in an Introductory Linear Algebra class. I know this question has already been asked (Difference between orthogonal projection and least squares solution), but I want to check my understanding. PROJECTION ONTO SUBSPACE In projection, the purpose is to find the point where the […]

derivation of simple linear regression parameters

I know there are some proof in the internet, but I attempted to proove the formulas for the intercept and the slope in simple linear regression using Least squares, some algebra, and partial derivatives (although I might want to do it wituout partials if it’s easier). I’ve posted my attempt below. I don’t know what […]

Minimal number of points to define a rotated ellipse?

What is the minimal number of points $N$ to uniquely define the semi-major axis $a$, the semi-minor axis $b$ and the rotation angle $\omega$ of an ellipse whose the center is known/fixed (this is related to ellipse fitting procedures). In other words, if we consider that the center is known/fixed, on this image what is […]

Minimize $\|AXBd -c \|^2$, enforcing $X$ to be a diagonal block matrix

Currently, I am minimizing the quadratic objective $\|\mathbf{A}\mathbf{X}\mathbf{B}\mathbf{d} -\mathbf{c} \|^2$ using CVX, as follows echo on cvx_begin variable xx(num_triangles*3,num_triangles*3) minimize( norm( A * xx * B * d – c ) ) cvx_end echo off However, $\mathbf{X}$ is a very large matrix (about $50,000 \times 50,000$, which is too big). Good news is that $\mathbf{X}$ […]

Why is this weighted least squares cost function a function of weights?

Here is a picture from my book regarding weighted least squares: Totally lost here, so I extracted the main nested issues confusing me: First Question: I know that in any LSE we want to minimize the cost function. The cost function is a function of the variable, $x$. In WLSE, I (thought?) the weights were […]

Why is minimizing least squares equivalent to finding the projection matrix $\hat{x}=A^Tb(A^TA)^{-1}$?

I understand the derivation for $\hat{x}=A^Tb(A^TA)^{-1}$, but I’m having trouble explicitly connecting it to least squares regression. So suppose we have a system of equations: $A=\begin{bmatrix}1 & 1\\1 & 2\\1 &3\end{bmatrix}, x=\begin{bmatrix}C\\D\end{bmatrix}, b=\begin{bmatrix}1\\2\\2\end{bmatrix}$ Using $\hat{x}=A^Tb(A^TA)^{-1}$, we know that $D=\frac{1}{2}, C=\frac{2}{3}$. But this is also equivalent to minimizing the sum of squares: $e^2_1+e^2_2+e^2_3 = (C+D-1)^2+(C+2D-2)^2+(C+3D-2)^2$. I […]

Least squares fit for an underdetermined linear system

I want fitting my data using bicubic interpolation: $$f(x,y)=\sum_{i=0}^{3}\sum_{j=0}^{3}a_{ij}x^iy^j$$ Let known $$f(0, 0)=1; f(2, 0)=1;f(1, 1)=0;f(0, 2) = 1; f(2, 2)=1$$ I used least squares method, $$min\sum_{k=1}^{5}(f(x_k, y_k)-\sum_{i=0}^{3}\sum_{j=0}^{3}a_{ij}(x_k)^i(y_k)^j)^2$$ Receiving this system: $$\forall t, s \in [0, 3]: \sum_{k=1}^{5}\sum_{i=0}^{3}\sum_{j=0}^{3}a_{ij}(x_k)^{i+t}(y_k)^{j+s} = \sum_{k=1}^{5}(x_k)^{t}(y_k)^{s}f(x_k, y_k)$$ If present as $Ax = b$: $$A = \begin{pmatrix} 1 & 1 & 1 […]

How to calculate the lens distortion coefficients with a known displacement vector field?

I have this vector field full of displacement vectors, which indicates radial distortions by a lens system. (Source) I know where each of the displacement vectors starts $(x,y)$ and ends $(x’,y’)$ and I know the distortion equations look like $$ x’ = (1 + k_1r^2 + k_2r^4)x\\ y’ = (1 + k_1r^2 + k_2r^4)y $$ […]