## Tensor calculus and linear algebra In this chapter, we show that many linear algebra problems with matrix notation can be transformed into tensor forms. ### Contraction and matrix multiplication Matrix and tensor notations are two complementary ways to express linear algebra operations. A vector can be represented as $T^i$ or $T_i$, depending on whether it is contravariant or covariant. A matrix, as a second-rank tensor, may appear as $T^i_j$, $T_{ij}$, or $T^{ij}$, with **the first index being the row** and **the second being the column**; For higher order tensors, there is no single universally accepted convention, and index positions must be interpreted within context. The matrix notation works best for two-dimensional systems. Matrix multiplication can be directly compared to tensor contraction using the following correspondences: | Matrix Notation | Tensor Notation (indicial notation) | Notes | | --------------- | ----------------------------------- | ---------------------------------------- | | $C = AB$ | $C_{ij} = A_{ik} B_{kj}$ | Standard matrix multiplication | | $C = AB^T$ | $C_{ij} = A_{ik} B_{jk}$ | Multiply with transpose on the right | | $C = BA$ | $C_{ij} = A_{kj} B_{ik}$ | Swap multiplication order | | $C^T = AB$ | $C_{ji} = A_{ik} B_{kj}$ | Transpose the result of standard product | The trace of a matrix provides another example of contraction. In matrix notation, it is expressed as $ \text{Tr}(A) = \sum_k A_{kk} $ Tensor notation is particularly effective when emphasizing structure, coordinate invariance, or higher-rank objects. Matrix notation, on the other hand, is often more convenient for algebraic manipulation, symbolic calculation, and logical deduction. The choice between the two should be guided by the nature of the problem: whether one is focusing on geometric structure or on algebraic operations. ### Fundamental elements of linear algebra #### Change of basis and contravariant components A vector $\mathbf{v}$ in an $n$-dimensional linear space can be expressed in terms of a basis $\{ \mathbf{e}_i \}$ as $ \mathbf{v} = v^i \mathbf{e}_i $ where $v^i$ are the components of $\mathbf{v}$ in that basis. These components are contravariant: they transform in the opposite direction to the basis vectors under a change of basis. Consider a new basis $\{ \mathbf{e}_{i'} \}$ related to the old one by $ \mathbf{e}_{i'} = J^i_{i'} \mathbf{e}_i, \quad \mathbf{e}_i = J^{i'}_i \mathbf{e}_{i'} $ where $J^i_{i'}$ and $J^{i'}_i$ are inverses of each other. Define a matrix $X$ such that $X_{i i'} = J^i_{i'}$. Then $X$ is an $n \times n$ invertible matrix, and its inverse satisfies $X^{-1}_{i' i} = J^{i'}_i$. If we organize basis vectors into column matrices $ \mathbf{E} = [\, \mathbf{e}_1 \ \cdots \ \mathbf{e}_n \,], \quad \mathbf{E}' = [\, \mathbf{e}_{1'} \ \cdots \ \mathbf{e}_{n'} \,] $ then the basis transformation is expressed in matrix notation as $ \mathbf{E}' = X^T \mathbf{E} $ This use of the transpose arises from the convention that in tensor notation the **first index is the row** and the **second is the column**, while $\mathbf{E}$ is a matrix of column vectors. The transpose ensures the correct contraction structure to recover $\mathbf{e}_{i'} = J^i_{i'} \mathbf{e}_i$. Correspondingly, the components of the vector transform as $ v^{i'} = J^{i'}_i v^i, \quad v^i = J^i_{i'} v^{i'} $ and in matrix form (with $v$ and $v'$ as column vectors): $ \mathbf{v}' = X^{-1} \mathbf{v}, \quad \mathbf{v} = X \mathbf{v}' $ This illustrates the contravariant behavior: the components change in the **opposite** direction to the basis. For completeness, the vector expression $\mathbf{v} = v^i \mathbf{e}_i$ can be written in matrix form as $ \mathbf{v} = \mathbf{v}^T \mathbf{E} $ which shows how the components (as a row vector) contract with basis vectors (as columns) to reconstruct the full geometric object. > [!Note]- > To confirm the consistency of the transformation rules, we can insert an identity matrix in the form $(X^T)^{-1} X^T$ into the matrix expression of the vector: > $ > \mathbf{v} = \mathbf{v}^T (X^T)^{-1} X^T \mathbf{E} > $ > Then group the terms as > $ > \mathbf{v} = (X^{-1} \mathbf{v})^T (X^T \mathbf{E}) > $ > Since $X^T \mathbf{E} = \mathbf{E}'$, and $X^{-1} \mathbf{v} = \mathbf{v}'$, this becomes > $ > \mathbf{v} = (\mathbf{v}')^T \mathbf{E}' > $ > which verifies that the transformed components and basis vectors reconstruct the same geometric object $\mathbf{v}$ in the new basis. This not only confirms equation $v' = X^{-1} v$ but also illustrates the internal consistency of the matrix and tensor representations of change of basis. #### Inner product and the Gram matrix In an inner product space, the inner product of two vectors $\mathbf{u}$ and $\mathbf{v}$ can be written in tensor notation as $ (\mathbf{u}, \mathbf{v}) = (u^i \mathbf{e}_i, v^j \mathbf{e}_j) = (\mathbf{e}_i, \mathbf{e}_j) u^i v^j $ Define the matrix $M$ with entries $ M_{ij} = (\mathbf{e}_i, \mathbf{e}_j) $ This matrix is called the **Gram matrix**, also known as the **metric tensor** $Z_{ij}$. Then the inner product becomes $ (\mathbf{u}, \mathbf{v}) = M_{ij} u^i v^j $ In matrix notation, using column vectors $u$ and $v$, this expression reads $ (\mathbf{u}, \mathbf{v}) = u^T M v $ When the basis $\{ \mathbf{e}_i \}$ is orthonormal, the Gram matrix is simply the identity matrix. For a general change of basis, $M_{ij}$ transforms as a doubly covariant tensor, i.e., both indices follow the covariant transformation rule. In tensor notation, the transformation is: $ M_{i' j'} = M_{ij} J^i_{i'} J^j_{j'} $ or equivalently: $ M_{i' j'} = J^i_{i'} M_{ij} J^j_{j'} $ If we let $X$ be the matrix representation of the Jacobian $J^i_{i'}$, then the matrix form of the transformation becomes $ M' = X^T M X $ This formula ensures that the inner product remains invariant under a change of basis, and is consistent with the tensor transformation rules for covariant quantities. #### Linear transformation > [!Note] > A linear transformation $A$ is a map from a vector space to itself (or to another vector space) that preserves the structure of vector addition and scalar multiplication. Formally, a map $A: V \to W$ is **linear** if for all vectors $\mathbf{u}, \mathbf{v} \in V$ and scalars $c \in \mathbb{R}$ (or $\mathbb{C}$): > $ > A(\mathbf{u} + \mathbf{v}) = A(\mathbf{u}) + A(\mathbf{v}), \quad > A(c \mathbf{v}) = c A(\mathbf{v}) > $ > Once a basis is chosen, every linear transformation can be represented by a matrix. In this case, the action of $A$ on a vector $\mathbf{v}$ is written as matrix-vector multiplication: > $ > A(\mathbf{v}) = A \cdot \mathbf{v} > $ > This representation is central to linear algebra and forms the bridge between abstract transformations and concrete matrix computation. Consider a linear transformation $A$ that maps a vector $\mathbf{u}$ to another vector $\mathbf{v}$: $ \mathbf{v} = A(\mathbf{u}) $ In matrix notation, this becomes the familiar matrix-vector multiplication: $ \mathbf{v} = A \mathbf{u} $ In tensor notation, since both $u^j$ and $v^i$ are contravariant components, the transformation is expressed using a tensor $A^i_{j}$ that is **contravariant in the first index and covariant in the second**: $ v^i = A^i_{j} u^j $ This shows that $A^i_{\ j}$ is a **(1,1)-type tensor**, i.e., a linear operator. Under a change of basis, such a tensor transforms according to the rule: $ A^{i'}_{j'} = A^i_{j} J^{i'}_i J^j_{j'} $ where $J^{i'}_i$ is the Jacobian matrix describing the basis change. In matrix notation, this becomes: $ A' = X^{-1} A X $ This formula ensures the consistency of the transformation of operators under change of basis. As a direct consequence, we recover a classical result from linear algebra: the eigenvalues of a tensor $A^i_{j}$ (or of the matrix $A$) are invariant under a change of coordinates. ### Self-adjoint transformations and symmetry > [!Note] > In finite-dimensional complex vector spaces, a **self-adjoint operator** is equivalent to a **Hermitian matrix**. Specifically, a linear operator $A$ is called self-adjoint if it satisfies > $ > \langle A\mathbf{v}, \mathbf{w} \rangle = \langle \mathbf{v}, A\mathbf{w} \rangle > $ > for all vectors $\mathbf{v}, \mathbf{w}$ in the space. In matrix terms, this means: > $ > A^\dagger = A > $ > where $A^\dagger$ denotes the **conjugate transpose** of $A$. In real vector spaces, self-adjointness reduces to **symmetry**: $A^T = A$. Therefore, "Hermitian" is simply the matrix representation of a self-adjoint operator in a complex inner product space. > >[!Notice]- > >Here $\langle \cdot,\cdot\rangle$ stands for inner product. While in the textbook $(\cdot, \cdot)$ is applied. They are the same. A matrix $A$ represents a self-adjoint transformation if and only if the following condition holds in matrix notation: $ MA = A^T M $ Equivalently, $ A = M^{-1} A^T M $ This is a general statement characterizing a matrix $A$ that represents a self-adjoint transformation. It generally implies that $A$ is not symmetric unless the basis is orthonormal (i.e., $M = I$). To confirm this result, we turn to tensor notation. Express the inner products $(A\mathbf{u}, \mathbf{v})$ and $(\mathbf{u}, A\mathbf{v})$ as: $ (A\mathbf{u}, \mathbf{v}) = M_{ij} A^i_k u^k v^j $ $ (\mathbf{u}, A\mathbf{v}) = M_{ij} u^i A^j_k v^k $ Relabel the second expression: $ (\mathbf{u}, A\mathbf{v}) = M_{ki} A^i_j u^k v^j $ Since these identities hold for any $u^k$ and $v^j$, we must have: $ M_{ij} A^i_k = M_{ki} A^i_j $ Now relabel so that $i$ and $j$ are the free indices and use symmetry of $M_{ij}$: $ M_{ki} A^k_j = M_{kj} A^k_i $ This expression corresponds to the matrix identity: $ MA = A^T M $ which confirms the general condition. We can also use the Gram matrix to lower the superscript index, expressing the condition in the form: $ A_{ij} = A_{ji} $ This identity states that, after lowering the index, the tensor $A_{ij}$ becomes symmetric. However, this does **not** imply that the original matrix $A$ is symmetric, because the matrix has entries $A^i_j$, not $A_{ij}$. If we raise the index back, we recover: $ A^i_{\cdot j} = A^{\cdot i}_{j} $ Only the left-hand side has the same entries as the matrix $A$; the right-hand side generally does not. It is important to maintain proper index order. > [!Note]- > The set of all linear transformations on a finite-dimensional vector space $V$ over a field $F$ forms a non-commutative [[Quick overview of algebraic structures#Ring|ring]] and associative [[Quick overview of algebraic structures#Algebra|algebra]], typically denoted as $\mathrm{End}(V)$ or $M_n(F)$. Matrix multiplication is generally not commutative, and many linear maps are not invertible. Therefore, this ring does not form a [[Quick overview of algebraic structures#Group|group]] under composition. > > The subset of all **invertible** linear transformations forms a group under composition, known as the **general linear group**: > > $ > \mathrm{GL}(n, F) = \{ A \in M_n(F) \mid \det A \ne 0 \} > $ > > Several important subgroups of $\mathrm{GL}(n, F)$ preserve specific geometric or algebraic structures: > > - $\mathrm{SL}(n, F)$: special linear group (volume-preserving, $\det A = 1$) > - $\mathrm{O}(n)$: orthogonal group (preserves Euclidean inner product, $A^T A = I$) > - $\mathrm{SO}(n)$: special orthogonal group (orthogonal and orientation-preserving, $\det A = 1$) > - $\mathrm{U}(n)$: unitary group (preserves Hermitian inner product, $A^\dagger A = I$) > - $\mathrm{SU}(n)$: special unitary group (unitary with $\det A = 1$) > > These linear transformation groups (these are also [[Lie groups]]) play a fundamental role in geometry, physics, and representation theory. Not every linear transformation belongs to one of these groups; only those that satisfy the corresponding structural constraints (e.g., length, angle, orientation, or inner product preservation) do. ### Quadratic form optimization We consider the optimization of the quadratic function $ f(x) = \frac{1}{2} x^T A x - x^T b $ where $A$ is a **symmetric positive definite matrix** and $b$ is a given vector. This form generalizes the scalar quadratic function to vector spaces. We want to recover the result $Ax = b$ in matrix notation by tensor notation. > [!Note]- > The result can be obtained from matrix notation. In matrix notation, the function > $ > f(x) = \frac{1}{2} x^T A x - x^T b > $ > can be analyzed using matrix calculus. This approach makes use of two key facts: > > - The gradient of a quadratic form is > $ > \nabla f(x) = \frac{1}{2}(A + A^T)x - b. > $ > When $A$ is symmetric, this simplifies to: > $ > \nabla f(x) = Ax - b. > $ > > - The Hessian, which is the second derivative of $f(x)$ with respect to $x$, is > $ > \nabla^2 f(x) = A. > $ > If $A$ is constant and symmetric, this confirms that the function is quadratic and its curvature is fully characterized by $A$. > > To find the critical point, we set $\nabla f(x) = 0$, giving the condition: > $ > Ax = b. > $ > > When $A$ is **symmetric and positive definite**, this system has a **unique solution**, and $f(x)$ is strictly convex, ensuring a **global minimum** at that solution. > [!Warning]- > In quadratic form optimization, the assumption that the matrix $A$ is symmetric and positive definite is crucial: > > - If $A$ is **not symmetric**, the function $f(x) = \frac{1}{2} x^T A x - x^T b$ is not guaranteed to be real-valued or differentiable in a simple form. The gradient becomes: > $ > \nabla f(x) = \frac{1}{2}(A + A^T)x - b > $ > and the Hessian is not simply $A$. > > - If $A$ is **not positive definite**, then $f(x)$ may not be convex. It could have: > - multiple minimizers (if $A$ is positive semi-definite), > - or **no minimizer at all** (if $A$ has negative eigenvalues, i.e., $f(x)$ has saddle points). > > The assumption that $A$ is **symmetric positive definite** guarantees that: > - The quadratic form is **strictly convex**, ensuring a **unique global minimum**. > - The gradient simplifies to $\nabla f(x) = Ax - b$, and the critical point satisfies the linear system: > $ > Ax = b > $ > which always has a unique solution. > >[!Notice] > >However, the lack of symmetry or positive definiteness does **not** make the problem unsolvable—it simply requires more advanced tools, such as analyzing the spectral decomposition or using constrained optimization techniques. In tensor notation, the function is written as $ f(x) = \frac{1}{2} A_{ij} x^i x^j - b_i x^i $ To take the derivative with respect to $x^k$, we use the identities $ \frac{\partial x^i}{\partial x^k} = \delta^i_k, \quad \frac{\partial (x^i x^j)}{\partial x^k} = \delta^i_k x^j + \delta^j_k x^i $ This gives: $ \frac{\partial f}{\partial x^k} = \frac{1}{2} A_{ij} (\delta^i_k x^j + \delta^j_k x^i) - b_i \delta^i_k = \frac{1}{2}(A_{kj} x^j + A_{ik} x^i) - b_k $ If $A_{ij}$ is symmetric, this simplifies to: $ \frac{\partial f}{\partial x^k} = A_{ki} x^i - b_k $ Setting the gradient to zero gives the condition for minimization: $ A_{ki} x^i = b_k $ > [!Note]- > In tensor notation, the **Hessian** of a scalar function $f(x^i)$ is defined as: > $ > \frac{\partial^2 f}{\partial x^i \partial x^j}. > $ > For the quadratic form > $ > f(x) = \frac{1}{2} A_{kl} x^k x^l - b_k x^k, > $ > the Hessian is > $ > \frac{\partial^2 f}{\partial x^i \partial x^j} = A_{ij}. > $ > This result follows because the second derivatives eliminate the variables and yield the constant symmetric matrix $A_{ij}$. > > [!Proof] > > Starting from > > $ > > f(x) = \frac{1}{2} A_{kl} x^k x^l - b_k x^k, > > $ > > we compute the first derivative: > > $ > > \frac{\partial f}{\partial x^j} = \frac{1}{2} A_{kl} \left( \delta^k_j x^l + x^k \delta^l_j \right) - b_j = A_{kj} x^k - b_j. > > $ > > Then the second derivative is: > > $ > > \frac{\partial^2 f}{\partial x^i \partial x^j} = A_{kj} \delta^k_i = A_{ij}. > > $ ### The eigenvalue problem We now consider the generalized eigenvalue problem: $ A x = \lambda M x, $ where $A$ and $M$ are symmetric matrices and $M$ is also **positive definite**. This problem can be reformulated as a **variational problem**: find the extrema of the quadratic form $ f(x) = A_{ij} x^i x^j, $ subject to the constraint $ M_{ij} x^i x^j = 1. $ This constraint enforces that the vector $x^i$ has unit “length” under the inner product defined by $M_{ij}$. Introducing a Lagrange multiplier $\lambda$, we define the augmented objective: $ E(x, \lambda) = A_{ij} x^i x^j - \lambda (M_{ij} x^i x^j - 1). $ Taking the derivative with respect to $x^i$ and setting it to zero: $ \frac{1}{2} \frac{\partial E(x)}{\partial x^i} = A_{ij} x^j - \lambda M_{ij} x^j, $ which leads to the generalized eigenvalue equation: $ A_{ij} x^j = \lambda M_{ij} x^j. $ This can also be expressed as: $A_{ij} x^j = \lambda x_i$ $A^i_{\;j} x^j = \lambda x^i$ > [!Note]- > **Rayleigh Quotient Expression for Eigenvalues** > > The eigenvalues $\lambda$ of the generalized eigenvalue problem > $ > A_{ij} x^j = \lambda M_{ij} x^j > $ > can be expressed using the **Rayleigh quotient**: > $ > \lambda = \frac{A_{ij} x^i x^j}{M_{ij} x^i x^j}. > $ > This form is valid for any non-zero vector $x$. > Under the normalization condition > $ > M_{ij} x^i x^j = 1, > $ > it simplifies to: > $ > \lambda = A_{ij} x^i x^j. > $