Preface

“What does it actually mean to take derivative of a loss function with respect to a matrix ?” I was wondering some day during my PhD.

“A matrix is nothing but a bunch of numbers structured in a 2D format. So, you just take derivative of loss with respect to each of these numbers and then put them in the same format at the corresponding position. That’s what the derivative of a loss with respect to that matrix is”.

Okay, that is very convenient, but at the same time quite tedious thing to do if we have to do it by hand.

“I’m sure there should be some smart way to go about this. What is that?”

This blog is what I discovered as an early PhD student and today I decided I share that with others who might be wondering too.

Differentiating with respect to Vector

Before going into matrix derivative, let’s think about derivative vector. Let $y=f\left(x\right)$ be a scalar valued function of a vector $x$; i.e., $f:R^m\to R$. Think of this as a loss function depending on some vector of dimension $m$.

The derivative $\frac{dy}{dx}$ is a vector $[\frac{dy}{d{x}_j}]$. By convention, we assume all vectors are column vectors and transpose whenever we need row vectors. Also, $\left[a_j\right]$ denotes a vector with individual elements $a_j$.

Example:

Let $x=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$ and $y=x_1^2+3x_2^4$. Then, $\frac{dy}{dx}=\left[\begin{array}{c}2x_1\\ 12x_2^3\end{array}\right]$.

Note that the column vector $[\frac{dy}{d{x}_j}]$ is constructed by putting the derivative of $y$ with respect to the $j^{th}$ element of $x$ at the $j^{th}$ position.

We are interested in developing tools where we can perform such derivation directly. For example, if $A$ is any matrix and $y=x^{T}Ax$, then we want to be able to show $\frac{dy}{dx}=(A+A^T)x$. This can be shown with cumbersome calculation as follows:

Let $A=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)$. Then, $y=\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$ $=a_{11}{x}_1^2+a_{21}{x}_1{x}_2+a_{12}{x}_1{x}_2+a_{22}{x}_2^2$.

$\frac{dy}{dx}=\left[\begin{array}{c}2a_{11}{x}_1+(a_{21}+a_{12})x_2\\ (a_{21}+a_{12})x_1+2a_{22}{x}_2\end{array}\right]=(A+A^T)x$.

However, we want to do this in an easier way. Moreover, we also want the technique to be useful for deriving derivative with respect to matrices.

Use of Differentials

Using Taylor expansion: $f(x+dx)=f\left(x\right)+(\frac{df}{dx}{)}^{T}dx+$ higher terms

The key idea here is that if we take $dx$ sufficiently small, then $dy=f(x+dx)-f\left(x\right)=(\frac{df}{dx}{)}^{T}dx$. Why? Because all the rest of the terms are higher powers of $dx$ and will converge to zero in the limiting case. In fact, the very definition of derivative comes from this limiting case of Taylor expansion if we think about it. So, we can use this trick to compute derivative. Hence, given any function $f\left(x\right)$, we calculate $dy=D^{T}dx$ and then obtain $\frac{dy}{dx}=D$.

Example:

$y=f\left(x\right)=x^{T}Ax$

$dy=(x+dx{)}^{T}A(x+dx)-x^{T}Ax$$=x^{T}Ax+x^{T}Adx+d{x}^{T}Ax+d{x}^{T}Adx-x^{T}Ax$

In the limiting case, as $dx\to 0$:

$dy=x^{T}Adx+d{x}^{T}Ax$

Using $d{x}^{T}Ax=x^T{A}^{T}dx$ (because $d{x}^{T}Ax$ is a scalar, so it’s equal to its transpose),

$dy=x^{T}Adx+x^T{A}^{T}dx=x^T(A+A^T)dx$. Therefore, $\frac{dy}{dx}=(A+A^T)x$.

Using the same procedure, it is easy to show $\frac{d}{dx}\left(w^{T}x\right)=w$.

This technique of using differentials is powerful because we can use the same technique to find derivative with respect to matrices.

Derivative with respect to matrices

Let $f$ be a function of matrices. We denote $y=f\left(X\right)$, where $X=\left[x_{ij}\right]$. Note that $y$ is scalar. What does it mean by derivative w.r.t. matrix? $f:R^{m\times n}\to R$. The derivative ${\nabla }_{X}y=\left[\begin{array}{c}\frac{\partial y}{\partial x_{ij}}\end{array}\right]$. That is, we differentiate $y$ with respect to each element of matrix $X$, $\frac{\partial y}{\partial x_{ij}}$, and then we fill the $(i,j{)}^{th}$ position of the matrix with this derivative to obtain the required derivative.

Interestingly, we can use the same procedure of differentials to find derivative with respect to matrices.

Steps to calculate derivative w.r.t. matrix X:

1. $dy=f(X+dX)-f\left(X\right)$
2. Express $dy=tr\left(D^{T}dX\right)$
3. ${\nabla }_{X}y=D$

Why is there a trace of $D$ and $dx$? Because at this point we need to define the inner product of two matrices. The Taylor series equivalent with matrix as argument requires us to define inner product in the matrix space. For now we can think of that $tr\left(D^{T}dX\right)$ is inner product of two matrices $D$ and $dX$. To see this you can take two matrices of $2\times 2$ and see that this expression gives us a scalar and satisfies inner product properties like linearity and norm etc.

Example 1:

$\frac{d}{dX}tr\left(XB{X}^T\right)=X(B+B^T)$

Proof:

$dy=tr\left[\right(X+dX\left)B\right(X+dX{)}^T]-tr(XB{X}^T)$

Setting higher terms of $dX$ to zero:

$dy=tr(dXB{X}^T+XBd{X}^T)$

Using property $tr\left(AB\right)=tr\left(BA\right)$ and $tr(A+B)=tr\left(A\right)+tr\left(B\right)$:

$dy=tr\left(dXB{X}^T\right)+tr\left(XBd{X}^T\right)$

$dy=tr\left(B{X}^{T}dX\right)+tr\left(B^T{X}^{T}dX\right)$

$dy=tr\left[\right(B+B^T\left)X^{T}dX\right]$

Therefore, $\frac{dy}{dX}=X(B+B^T)$.

Example 3:

$\frac{d}{dX}\left[tr\right(A{X}^{-1}\left)\right]=-(X^{-1}A{X}^{-1}{)}^T$

Proof: By now you must have realized from above exercise that the differential operator is linear. So, we can apply some tricks like moving differential operator inside another linear operator like trace. A direct application of something like this is as follows:

$d\left[tr\right(A{X}^{-1}\left)\right]=tr\left(Ad{X}^{-1}\right)$.

At this point, we use the following identity: $d{X}^{-1}=-X^{-1}dX{X}^{-1}$ —(1)

The proof will be provided later. For now using this identity:

$dy=tr\left(A\right(-X^{-1}dX{X}^{-1}\left)\right)$

$=tr(-A{X}^{-1}dX{X}^{-1})$

Using the cyclic property of trace, $tr\left(ABC\right)=tr\left(BCA\right)=tr\left(CAB\right)$:

$=tr(-X^{-1}A{X}^{-1}dX)$.

Therefore, $\frac{dy}{dX}=-(X^{-1}A{X}^{-1}{)}^T$.

Proof of equation (1):

$d\left(I\right)=0$.

Or, $d\left(X{X}^{-1}\right)=0$.

Or, $Xd{X}^{-1}+dX{X}^{-1}=0$.

$d{X}^{-1}=-X^{-1}dX{X}^{-1}$.

Example 4:

$\frac{d}{dX}\ln |X|=(X^{-1}{)}^T$.

Proof of example 4 includes the identity $d\ln |X|=tr\left(X^{-1}dX\right)$ — (2).

For proof of this identity, see the Convex Optimization book [2].

Using (2):

$\frac{d}{dX}\ln |X|=(X^{-1}{)}^T$.

References

1. Caltech EE150 Lecture 5 Slides](http://www.systems.caltech.edu/dsp/ee150_acospc/lectures/EE_150_Lecture_5_Slides.pdf)
2. https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf