8.1 Introduction to Tensors

Multi-way arrays and why flattening loses structure.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

Figures and notes follow Introduction to Tensor Decompositions and their Applications in Machine Learning.

1. Brief history

Tensors generalize matrices to higher dimensions: multidimensional arrays.

They and their decompositions appear in 1927, then sit unused in computer science until late in the 20th century. Cheaper compute and better multilinear algebra — especially in the last decade — moved them into statistics, data science, and machine learning.

2. Tensor basics

A tensor is a multi-way collection of numbers, usually from $ℝ$ . The simplest high-dimensional case is a three-dimensional array: a data cube. The same notation extends to more modes.

A three-way tensor as a data cube

3. Tensor order

The order of a tensor is the number of dimensions.

Object	Order	Notation
Scalar	0	$x \in ℝ$
Vector	1	$𝐱 \in ℝ^{I_{1}}$
Matrix	2	$𝐗 \in ℝ^{I_{1} \times I_{2}}$
Higher-order tensor	$N$	$𝒳 \in ℝ^{I_{1} \times I_{2} \times \dots \times I_{N}}$

Each $I$ is the length of that mode.

Scalars, vectors, matrices, and a 3-way tensor

4. Tensor indexing

Fix some indices to get a subarray.

Fibers: fix every index but one.
Slices (slabs): fix every index but two.

For a third-order tensor, fibers are $x_{: j k} = x_{j k}$ (column), $x_{i : k}$ (row), $x_{i j :}$ (tube). Slices are $𝒳_{: : k} = 𝒳_{k}$ (frontal), $𝒳_{: j :}$ (lateral), $𝒳_{i : :}$ (horizontal).

Fibers and slices of a 3-way tensor

5. Outer and inner product

The vector outer product is the product of the entries, written $⊚$ . For two length- $n$ vectors $𝐚$ and $𝐛$ ,

𝐗 = 𝐚 ⊚ 𝐛 = 𝐚 𝐛^{⊤} .

For $N$ vectors the same product is a tensor:

𝒳 = 𝐚^{(1)} ⊚ 𝐚^{(2)} ⊚ \dots ⊚ 𝐚^{(N)},

with $x_{i_{1} i_{2} \dots i_{N}} = a_{i_{1}}^{(1)} a_{i_{2}}^{(2)} \dots a_{i_{N}}^{(N)}$ .

The inner product of two length- $n$ vectors is a scalar:

x = ⟨ 𝐚, 𝐛 ⟩ = 𝐚^{⊤} 𝐛 = \sum_{i = 1}^{n} a_{i} b_{i} .

6. Rank-1 tensors

An $N$ -way tensor is rank-1 if it is exactly the outer product of $N$ vectors. Adding a mode is a new scaling of a sub-tensor.

A rank-1 matrix: $𝐗 = 𝐚 ⊚ 𝐛$ . A rank-1 3-way tensor: $𝒳 = 𝐚 ⊚ 𝐛 ⊚ 𝐜$ . The $N$ -way form is the product above.

Rank-1 tensor as an outer product of vectors

7. Tensor rank

The rank $rank (𝒳) = R$ is the smallest number of rank-1 tensors that sum to $𝒳$ :

𝒳 = \sum_{r = 1}^{R} λ_{r} 𝐚_{r}^{(1)} ⊚ 𝐚_{r}^{(2)} ⊚ \dots ⊚ 𝐚_{r}^{(N)} .

The factor matrices $𝐀$ store those vectors as columns:

𝐀 = [\begin{matrix} 𝐚_{1} & 𝐚_{2} & \dots & 𝐚_{R} \end{matrix}] .

The weights $λ_{r}$ (with $λ \in ℝ^{R}$ ) often absorb column norms, typically so each column has squared length one.

Practice

A matrix is a 2-way array. What is a tensor adding?
What is a fiber, and what is a slice?
What does $rank (𝒳) = R$ mean in terms of rank-1 tensors?

Ahmad Mousavi