Steerable Convolution

These notes are being transcribed, condensed, and illustrated. They may lag or omit details, please refer to the handwritten notes for the full lecture.

This note continues from the Group Convolution note. In group convolution, equivariance is enforced by summing over the group: we apply every group element to the filter and integrate. This works, but it means every forward pass explicitly loops over |G| group elements.

Here we take a different approach. We show that group convolution filters can be decomposed into basis functions — fixed spatial patterns that each transform as a specific irrep of the group. This decomposition reveals that the filter's group structure is entirely captured by Clebsch-Gordan coefficients (the change-of-basis solutions between the tensor product of input and filter representations and the output representation). Once we see this, we no longer need to sum over the group at all: we stay in the irrep basis, where equivariance is built into the basis functions themselves.

The convolution over space (translation) is unchanged — it's still a standard 2D convolution with learnable filters. What changes is how those filters are parameterized: instead of learning arbitrary pixel values and enforcing symmetry by group averaging, we learn scalar weights on the precomputed equivariant basis. The interaction between input and filter is mediated by CG coefficients, which determine which couplings are allowed (the selection rules).

1) Introduction and Roadmap

This note has four parts:

Part 0: Recap and group convolution in the irrep basis (§2 Recap and §3 Group Convolution in the Irrep Basis). We briefly recall the group convolution framework from the previous note, then change basis from the regular representation into irreps using C_4 as a concrete example. This shows that group convolution in the irrep basis becomes pointwise multiplication — a direct consequence of Schur's lemma.

Part 1: From group convolution to basis functions (§4 Spatial Basis Functions and Filter Decomposition and §6 Steerable Filters). We add spatial extent to the picture: decomposing the pixel permutation representation of D_4 into irreps gives us spatial basis functions — fixed 3 \times 3 patterns that transform as specific irreps. Group conv filters decompose into scalar weights on these basis elements (Schur's lemma). The regular representation acts as a coupling tensor providing all possible irrep interaction paths. We recover the same 72 parameters of group convolution, but now organized by irrep content.

Part 2: Steerable convolution (§5 Steerable Convolution: Working in the Irrep Basis and §7 Tensor Products, Linear Layers, and Schur's Lemma). We relax the requirement that features live in the regular representation. Instead, we choose arbitrary irrep content for input and output features, and think of convolution as an interaction between input and filter that produces an output — mediated by CG coefficients. The spatial convolution (translation) is unchanged, but we stay in the irrep basis rather than summing over the group. This is steerable convolution: equivariant by construction, with a flexible (typically smaller) parameter space.

Part 3: Activation functions (§8 Activation Functions). In the irrep basis, pointwise nonlinearities like ReLU break equivariance. We develop alternatives — norm-based and gated activations — that respect the irrep structure.

The note concludes with §9 Putting It All Together (a worked example on the Tetris classification problem) and §10 Summary.

2) Recap: Group Convolution on Tetris

Before spring break, we built a D_4-equivariant CNN that classifies tetrominos (the Tetris classification problem). Let's recall the key ingredients:

1. Lifting convolution: A standard image (scalar field on \mathbb{Z}^2) was "lifted" to carry a group index — the input was repeated |G| times to embed it in the regular representation.
2. Group convolution layers: Each layer stored a filter \psi_{dcj}(\mathbf{y}) with a group index j \in G and spatial positions \mathbf{y}. Equivariance was enforced by rotating and reflecting the filter via D^{\text{reg}}(k), producing a filter bank of |G| rotated copies.
3. Regular representation: Features at each spatial position formed a vector indexed by group elements — one component per element of D_4. The group acted by permuting these components.
4. Invariant pooling: At the output, we averaged over the group index (and spatial positions) to produce a group-invariant classification.

This architecture works, and pointwise nonlinearities (ReLU) are equivariant because the regular representation acts by permutation — reshuffling components without changing values.

3) Group Convolution in the Irrep Basis

We've built group convolution using the regular representation — storing one filter per group element and enforcing equivariance through D^{\text{reg}}(k). Now we change basis into irreps. This is the bridge from group convolution to steerable convolution: instead of working with group elements directly, we decompose everything into irreducible representations where the structure becomes transparent.

We'll do this in two steps:

1. Group features only (C_4): First, consider functions on the group f: G \to \mathbb{R} — no spatial dimensions yet. We use C_4 because it's small and abelian, so all irreps are 1D and the decomposition is the familiar DFT. Changing to the irrep basis (the group Fourier transform) diagonalizes the regular representation and turns convolution into pointwise products — each irrep contributes independently.

2. Spatial + group features (D_4): When the filter also has spatial extent (pixels), we additionally need to decompose the pixel permutation representation into irreps. We switch to D_4 (rotations and mirrors of the square, |D_4| = 8) because it has a 2D irrep with real-valued matrices, making the spatial basis functions easy to visualize as 3 \times 3 patterns. This gives symmetry-adapted spatial basis functions — fixed patterns that transform as specific irreps of the group — and reduces the learnable parameters to one scalar weight per basis function.

3.1) Group Features: C_4 Worked Example

We derived in the group convolution note that cross-correlation becomes a dot product in the irrep basis: [\widehat{f \star \psi}]_j = \hat{f}_j^* \cdot \hat{\psi}_j. Let's see this concretely for group convolution on C_4.

Since C_4 is abelian, all irreps are 1-dimensional, with characters \Gamma_j(k) = \omega^{jk} where \omega = e^{2\pi i/4} = i:

The change-of-basis matrix Q that diagonalizes all D^{\text{reg}}(k) simultaneously is Q_{jn} = \frac{1}{2}\omega^{jn} — the DFT matrix for N = 4. Let's see explicitly how inserting Q into the cross-correlation formula transforms it into the irrep basis.

Starting point. Cross-correlation in the regular rep is:

Insert I = Q^{-1}Q. Since Q diagonalizes the regular rep — Q \, D^{\text{reg}}(k) \, Q^{-1} = \Lambda(k) where \Lambda(k) = \text{diag}(\omega^{0 \cdot k},\, \omega^{1 \cdot k},\, \omega^{2 \cdot k},\, \omega^{3 \cdot k}) — we insert I = Q^{-1}Q on both sides:

where \hat{\psi}_j = \sum_n Q_{jn} \psi_n = \frac{1}{2}\sum_n \omega^{jn} \psi_n is the DFT of \psi, and \tilde{f}_j = \sum_n Q^{-1}_{jn} f_n. Since \Lambda(k) is diagonal, the triple product collapses to a single sum:

Each irrep j contributes independently: the product c_j = \tilde{f}_j \cdot \hat{\psi}_j is computed once (independent of k), and the output at group element k is just the weighted sum \sum_j c_j \, \omega^{jk} — an inverse DFT. This is the convolution theorem: convolution becomes pointwise multiplication in the irrep (Fourier) basis.

The interactive visualization below compares the two views side-by-side for f = [1, 0, 2, 1] and \psi = [3, 1, 0, 2]. The left panel shows the regular representation computation: permute f by k, dot with \psi. The right panel shows the irrep basis: the product c_j = \tilde{f}_j \cdot \hat{\psi}_j is computed once, then weighted by the eigenvalue \omega^{jk} and summed. Click each output element k to see how only the eigenvalue step changes — both views produce the same output [5, 4, 7, 8]:

3.2) From Abelian to Non-Abelian to Continuous

4) Spatial Basis Functions and Filter Decomposition

4.1) Spatial + Group Features: D_4 on a Pixel Grid

Everything above decomposed the regular representation alone — treating the group channels of a function f: G \to \mathbb{R} as our vector space. For 2D images, the full convolution involves more structure. Recall the spatial group convolution equation:

The filter \psi_{dcj}(\mathbf{y}) lives in the space of (pixels) \times (group elements), which carries the tensor product representation \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}} — the same tensor product structure we identified in The Tensor Product Structure. To fully decompose convolution into independent irrep blocks, we need to decompose both factors:

• Group channels (j \in G): Decompose \Gamma^{\text{reg}} into irreps — this is the group Fourier transform we just worked through above.
• Pixel positions (\mathbf{y} \in \mathbb{Z}^2): Decompose \Gamma^{\text{pix}} (the permutation representation on kernel pixels) into irreps — the same decomposition technique from vibrational modes, applied to pixel coordinates instead of atomic sites.

In the pure group case, changing to the irrep basis diagonalized (or block-diagonalized) the regular representation, turning convolution into pointwise products (or small matrix multiplications). Here, we additionally decompose the spatial structure — the result is a set of symmetry-adapted spatial basis functions for the filter, each labeled by an irrep. These are the 2D image analog of the irrep components \hat{f}_j we computed above: instead of a single scalar per irrep, each basis function is a spatial pattern on the pixel grid that transforms according to a specific irrep.

Why D_4 instead of C_4? We've been using C_4 as our running example, but C_4 is abelian, so all its irreps are 1D and necessarily complex (\omega = i). This makes the basis functions complex-valued — hard to visualize as pixel patterns. The dihedral group D_4 (rotations and reflections of the square, |D_4| = 8) has 5 irreps — four 1D and one 2D — and all of them admit a real change of basis. This means every symmetry-adapted filter pattern will be a real-valued 3 \times 3 image that we can directly visualize and interpret.

4.2) D_4 Irreps

D_4 has 8 elements (4 rotations + 4 reflections) and 5 conjugacy classes, giving 5 irreps:

Check: 1^2 + 1^2 + 1^2 + 1^2 + 2^2 = 8 = |D_4|. The regular representation decomposes as \Gamma^{\text{reg}} = A_1 \oplus A_2 \oplus B_1 \oplus B_2 \oplus 2E (each irrep appearing with multiplicity equal to its dimension).

All D_4 irreps can be realized with real matrices. The 2D irrep E acts on (x, y) via the 2 \times 2 rotation and reflection matrices we've been using throughout.

4.3) Spatial Basis Functions for a 3 \times 3 Kernel

In the group convolution notes, we built the pixel permutation representation \Gamma^{\text{pix}} — a set of 9 \times 9 matrices describing how each group element shuffles the pixels of a 3 \times 3 kernel. This is the same kind of permutation representation we used in vibrational modes to describe how symmetry operations permute atomic sites.

Recall that in vibrational modes, decomposing the site permutation representation \Gamma^{\text{a.s.}} into irreps told us which groups of atoms transform together under each irrep — which sites are symmetry-equivalent and how they mix.

Here we do the same thing for pixels. Decomposing \Gamma^{\text{pix}} into irreps tells us which groups of pixels transform together, and the resulting basis functions are scalar patterns on the kernel grid labeled by irreps:

Just as we tensored \Gamma^{\text{a.s.}} \otimes \Gamma^{\text{vec}} to account for the displacement vectors living at each atom, the full filter space here will be \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}} — combining the spatial structure with the representation of the data at each pixel. We'll get to that tensor product in the full derivation below. For now, let's focus on the spatial part alone: decomposing \Gamma^{\text{pix}} to find the basis functions.

A 3 \times 3 kernel has 9 pixels. Under D_4, these fall into 3 orbits at distinct radii:

• Center (1 pixel): |\mathbf{r}| = 0, fixed by all of D_4
• Edges (4 pixels): |\mathbf{r}| = 1, at (\pm 1, 0) and (0, \pm 1)
• Corners (4 pixels): |\mathbf{r}| = \sqrt{2}, at (\pm 1, \pm 1)

Since |\mathbf{r}| is invariant under D_4, the group acts independently within each orbit. This is the discrete analog of separating a function into radial and angular parts: W(\mathbf{r}) = R(|\mathbf{r}|) \, \Theta(\hat{\mathbf{r}}), where R is a scalar (the learnable parameter) and \Theta is a fixed angular pattern (the basis function).

We decompose the 9-dimensional permutation representation \Gamma^{\text{pix}} into irreps using the same tools from decomposition. The permutation representation is built by image2D_permutation_representation in group_conv.py, and the decomposition uses linalg.infer_change_of_basis:

import torch
from symm4ml import groups, rep, linalg, group_conv

# Generate D4 from two generators: mirror and 90-degree rotation
D4 = groups.generate_group([
    [[-1, 0], [0, 1]],                          # mirror across x
    [[0, -1], [1, 0]],                           # 90-degree rotation
])
D4_table = groups.make_multiplication_table(D4)
D4_irreps = rep.infer_irreps(D4_table)           # 5 irreps (four 1D + one 2D)

# Build perm rep for 3x3 kernel under D4
D4_torch = torch.tensor(D4, dtype=torch.float32)
perm_rep = group_conv.image2D_permutation_representation(D4_torch, [3, 3])

# Find basis functions for each irrep
for irrep in D4_irreps:
    basis = linalg.infer_change_of_basis(irrep, perm_rep.numpy())
    # basis.shape = (n_copies, d_irrep, 9)

Total: 3(1) + 0 + 1(1) + 1(1) + 2(2) = 9 ✓

These are symmetry-adapted angular basis functions for the discrete group D_4. (We will later see that continuous groups have their own versions of these — but that's a topic for future notes.) Each basis function is a real-valued 3 \times 3 pattern that transforms as a specific irrep of D_4:

• A_1: Isotropic (same value at all pixels of the same radius). Three copies correspond to three radii — the center pixel, the average over edges, and the average over corners.
• B_1: Positive on edges along one axis, negative on the other (x^2 - y^2 symmetry). Only from the edge orbit.
• B_2: Responds to diagonals (xy symmetry). Only from the corner orbit.
• E: Pairs of patterns that transform as a 2D vector (x, y). Two pairs — one from edges and one from corners — corresponding to two radii.
• A_2: Zero copies. The 3 \times 3 grid doesn't have enough angular resolution to support this irrep. (A larger kernel would.)

4.4) Properties of the Spatial Basis Functions

Each spatial basis function is written B^\alpha_{u,i}(\mathbf{y}) with three indices:

With this notation, the basis functions have three key properties:

Completeness. The basis functions span the full pixel space. Any function on the kernel grid can be expanded as:

where the coefficients are w_{\alpha u i} = \sum_{\mathbf{y}} B^\alpha_{u,i}(\mathbf{y}) \, \psi(\mathbf{y}). There are 9 basis functions spanning the 9-dimensional pixel space — no information is lost.

Orthonormality. Basis functions from different irreps, different copies, or different components are orthogonal:

This follows from the irrep decomposition: the rows of the change-of-basis matrix Q_{\text{pix}} are orthogonal vectors (Schur's lemma), and each basis function is a row of Q_{\text{pix}} reshaped into a 3 \times 3 grid.

Haar measure (orthogonality of irreps). The orthogonality above is a consequence of the Haar measure principle: the sum over pixel positions is an integration over the spatial domain. For the product of two basis functions transforming as different irreps to be nonzero, their product would need to contain the trivial irrep A_1. But D_\alpha \otimes D_\beta^* \supset A_1 only when \alpha = \beta — the same selection rule from spectroscopy, applied to spatial patterns.

4.5) Steerable \leftrightarrow Regular Rep Equivalence

We now have two ways to build an equivariant convolutional filter:

1. Group conv (regular rep): Store |G| \times (\text{pixels}) free parameters. The filter lives in \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}}. Equivariance is enforced by D^{\text{reg}}(k) acting as a coupling tensor. Decomposing this tensor product into irreps reveals 72 scalar weights \times basis elements (Schur's lemma).

2. Steerable conv: Build the filter directly from spatial basis functions B^\alpha_{u,i}(\mathbf{y}), with scalar weights as learnable parameters. Input and output features are collections of chosen irreps. The tensor product of input irreps with filter (spatial) irreps, decomposed via CG coefficients, determines the output irreps. Equivariance comes from the CG coupling structure.

The filter lives in \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}}, with dimension 9 \times 8 = 72. Before jumping to the full regular representation, let's build intuition by tensoring the spatial basis functions with a smaller representation.

4.6) Full tensor product: \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}}

Now let's do the full case. We already know both pieces separately:

Each learnable weight w_{\alpha j,\, \beta u} pairs a group-irrep component (\alpha, j) with a spatial basis function (\beta, u). The total number of weights is the product of these dimensions: 8 \times 9 = 72. To see which irrep of D_4 each weight transforms as overall, we use the CG decomposition of D_\alpha \otimes D_\beta for each (group irrep \alpha) \times (spatial irrep \beta) pair.

For example:

• A weight pairing A_1 (group) with A_1 (spatial) transforms as A_1 \otimes A_1 = A_1 — a fully symmetric filter component
• A weight pairing A_2 (group) with A_1 (spatial) transforms as A_2 \otimes A_1 = A_2 — this is how we get an A_2 filter component even though there is no A_2 spatial basis function
• A weight pairing E (group) with E (spatial) decomposes as E \otimes E = A_1 \oplus A_2 \oplus B_1 \oplus B_2 — a single (group, spatial) pair contributes to multiple filter irreps via CG coefficients. This is worth pausing on: in the regular rep view, the A_1, A_2, B_1, B_2 components that come from the same E \otimes E product are weight-tied — they share the same underlying filter weights, just recombined by the CG coefficients. In the steerable (irrep) view, Schur's lemma tells us each of these irrep components can have its own independent scalar weight. This is one way to see why the two views, while equivalent in total parameter count, organize the parameters differently

Counting degrees of freedom for D_4 on a 3 \times 3 kernel:

We can verify by directly decomposing the full 72-dimensional representation \Gamma^{\text{reg}} \otimes \Gamma^{\text{pix}} into irreps:

# Build the filter representation: reg ⊗ pix
D4_reg_rep = rep.regular_representation(D4_table)
filter_rep = rep.tensor_product(D4_reg_rep, perm_rep)  # shape (8, 72, 72)

# Decompose into irreps
irrep_names = ['A1', 'A2', 'B1', 'B2', 'E']
for name, irrep in zip(irrep_names, D4_irreps):
    basis = linalg.infer_change_of_basis(irrep, filter_rep)
    n_copies = basis.shape[0]
    d = irrep.shape[1]
    print(f"{name} ({d}D): {n_copies} copies × {d} = {n_copies * d} components")

A1 (1D):  9 copies × 1 =  9 components
A2 (1D):  9 copies × 1 =  9 components
B1 (1D):  9 copies × 1 =  9 components
B2 (1D):  9 copies × 1 =  9 components
E  (2D): 18 copies × 2 = 36 components
Total: 9 + 9 + 9 + 9 + 36 = 72 = |G| × pixels  ✔

The total — 72 — matches the |G| \times \text{pixels} = 8 \times 9 parameters of the regular rep filter exactly. The two parameterizations have identical expressiveness.

The interactive graphic below lets you explore these basis elements. The first tab shows the spatial basis functions from \Gamma^{\text{pix}} alone (9 elements). The second tab shows the full \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}} basis (72 elements) — each visualized as 8 copies of a 3 \times 3 pattern, one per group element. Select an irrep and copy to see the corresponding filter bank pattern:

The two parameterizations are related by a change of basis — the CG decomposition of \Gamma^{\text{reg}} \otimes \Gamma^{\text{pix}}. They produce identical convolution outputs.

5) Steerable Convolution: Working in the Irrep Basis

We've just shown that the group conv filter decomposes into scalar weights multiplying fixed basis functions (from \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}}). The natural question is: what if we start from basis functions that parameterize our convolutional filters, interacting with inputs that can also vary in irreps — building the operation directly from spatial basis functions and scalar weights, rather than constructing the regular representation and decomposing it after the fact?

In group conv, features are functions on the group (|G| channels per spatial position), and the filter lives in \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}} (72 parameters for D_4 on 3 \times 3). But the regular rep grows as |G|, and for continuous groups it is infinite-dimensional.

Steerable equivariant models store features directly as collections of irrep components — scalars (A_1), pseudoscalars (A_2), vectors (E), etc. — with chosen multiplicities. The filter is parameterized as scalar weights \times spatial basis functions from \Gamma^{\text{pix}}, with CG coefficients determining which (input irrep \times spatial irrep \to output irrep) couplings are allowed. Schur's lemma still governs everything: each allowed coupling path gets one learnable scalar.

5.1) Features as Concatenated Irrep Spaces

A steerable feature vector is a concatenation of irrep vector spaces. For example, with content A_1 \oplus A_2 \oplus B_1 \oplus B_2 \oplus 2E, the feature is an 8-dimensional vector formed by concatenating one 1D space for each of A_1, A_2, B_1, B_2 and two 2D spaces for E. The group acts on this vector via a block-diagonal representation: each block is the irrep matrix for that subspace, and different blocks don't mix.

By Schur's lemma, any equivariant linear map between two such concatenated irrep spaces must also be block-diagonal — with one scalar weight per irrep copy (see the equivariant linear map graphic below).

This block structure has practical consequences for implementation. We track the irrep content as a list of (irrep index, multiplicity) pairs and maintain slice information recording where each block sits:

hidden_content = [(A1, 1), (A2, 1), (B1, 1), (B2, 1), (E, 2)]
# → slices: [(0,1), (1,2), (2,3), (3,4), (4,6), (6,8)]
#   A1 at [0:1], A2 at [1:2], B1 at [2:3], B2 at [3:4], E at [4:6] and [6:8]

This slicing appears throughout the code:

• Building \rho(g). Place each irrep matrix into its diagonal block at the correct slice.
• Activation functions. Each block needs different treatment (ReLU for A_1, norm-based for others) — the activation loops over slices and dispatches accordingly.
• Embedding basis kernels. Each CG-contracted kernel acts on specific (input block, output block) pairs and must be embedded into the full d_{\text{out}} \times d_{\text{in}} matrix at the correct slice positions.

There are two approaches to computing the equivariant kernel basis:

1. Piecewise (see step-by-step notebook): iterate over all (input irrep, spatial irrep, output irrep) triples, compute CG coefficients for each, and embed into the full kernel. This makes each coupling path visible.

2. Dense (see dense notebook): build the full block-diagonal \rho_{\text{in}}, \rho_{\text{spatial}}, \rho_{\text{out}}, and call reduce_tensor_product once. The solutions already have the correct block structure — no manual slicing needed. Conceptually cleaner but scales worse for large representations.

Both produce the same basis elements. A key simplification: when all channels carry the same irrep content, the basis elements are identical for every channel pair, so they can be precomputed once and shared. The weight tensor [c_out, c_in, n_basis] is fully connected over channels, and the entire operation fuses into a single conv2d call.

5.2) The Steerable Convolution Layer

The explicit formula for a steerable convolution layer, adapted from the tensor field network formulation (Thomas, Smidt et al. 2018), is:

Reading from right to left:

1. Spatial sum: \sum_{\mathbf{y}} f^{\text{in}}_{\alpha, c, i}(\mathbf{x} + \mathbf{y}) \; B^\beta_{u, j}(\mathbf{y}) — project the local input patch onto spatial basis function B^\beta_{u,j}. This is the convolution with the basis function, producing a coefficient at each position \mathbf{x}.

2. CG contraction: \sum_{i,j} C^{\gamma k}_{\alpha i,\, \beta j} (\ldots) — couple the input irrep \alpha (component i) with the spatial irrep \beta (component j) to produce output irrep \gamma (component k). The CG coefficients are fixed.

3. Learned weights: w^{(\gamma, c')(\alpha, c)}_{\beta, u} — one scalar per (output irrep \gamma, output channel c', input irrep \alpha, input channel c, spatial copy u). Crucially, the weight does not depend on the irrep component indices i, j, k (Schur's lemma).

4. Sum over inputs: \sum_{\alpha, c} sums over all input irrep types and channels; \sum_{\beta, u} sums over all spatial basis function copies.

The indices are:

• \alpha, \beta, \gamma — irrep labels (input, spatial/filter, output)
• c, c' — input and output channels (multiplicity of each irrep type in the feature space)
• u — copy of the spatial basis function
• i, j, k — components within each irrep (summed over in the CG contraction)
• \mathbf{x} — spatial position, \mathbf{y} — kernel offset

Three things change compared to group conv:

1. No regular representation. Features are irrep components, not functions on the group. You choose which irreps to include and how many copies of each — giving full flexibility over the feature space design.

2. Control over coupling paths. We choose which irreps to include in our features and which CG-allowed coupling paths to use in the filter. We can include all possible paths for maximum expressivity, or use a sparse subset for efficiency. This flexibility is powerful but comes with the complexity of how to parameterize these choices. This is why dedicated software packages like escnn and e3nn exist — they provide abstractions that minimize this burden on the user while still exposing the flexibility of choosing irrep content and coupling paths.

3. Activation functions must change. In the regular rep, pointwise ReLU is equivariant (permutation matrices preserve values). In the irrep basis, the group acts by rotation/reflection matrices that mix components — so we need activations that respect the irrep structure.

5.3) What Changes: Group Conv vs. Steerable

6) Steerable Filters

6.1) How Basis Functions Transform

Recall the notation: B^\alpha_{u,i}(\mathbf{y}) where \alpha is the irrep label, u is the copy index, i = 1, \ldots, d_\alpha is the component within the irrep, and \mathbf{y} is the pixel position.

Each spatial basis function transforms as irrep D_\alpha under the group action on pixel positions. When we apply a group element g to the pixel coordinates:

The basis function at the rotated/reflected position g^{-1}\mathbf{y} is a linear combination of the components j of the same irrep \alpha and copy u, with coefficients given by the irrep matrix D_\alpha(g)_{ji}. For 1D irreps (A_1, A_2, B_1, B_2), D_\alpha(g) is a scalar (\pm 1), so this is just sign multiplication. For the 2D irrep E, the two components i = 1, 2 (corresponding to x, y) mix under rotation/reflection via the 2 \times 2 matrices of D_4.

Similarly, input and output features at each spatial position transform as their chosen irreps. A feature f^{\text{in}}_{\alpha, i}(\mathbf{x}) — the i-th component of irrep \alpha at position \mathbf{x} — transforms as:

6.2) The Kernel Constraint

For a convolution to be equivariant, the kernel K(\mathbf{y}) must satisfy the kernel constraint:

This says: rotating the kernel's spatial argument by g is the same as conjugating the kernel (viewed as a matrix from input to output features) by the corresponding representation matrices. In group convolution, this constraint is satisfied implicitly — the D^{\text{reg}} coupling tensor builds it in automatically. In steerable convolution, we state it explicitly and solve for the space of kernels that satisfy it.

The kernel constraint is a system of linear equations in the entries of K(\mathbf{y}). Its solution space — the set of all equivariant kernels — is a vector space, and we want a basis for it. The dimension of this space determines how many learnable parameters the layer has.

Concrete D_4 examples. The spatial basis functions on the 3 \times 3 grid contain 3A_1 \oplus B_1 \oplus B_2 \oplus 2E (no A_2). For a kernel mapping input irrep \alpha to output irrep \beta, we need CG paths \alpha \otimes \gamma \to \beta where \gamma is an irrep that appears in the spatial basis. The number of independent kernel basis elements equals the number of such paths (summed over copies):

• A_1 \to A_1: Spatial irreps containing A_1 in A_1 \otimes \gamma: need \gamma = A_1 (since A_1 \otimes A_1 = A_1). There are 3 copies of A_1 in the spatial basis → 3 basis elements. These are the 3 isotropic (rotationally invariant) patterns — one per orbit (center, edges, corners).

• A_1 \to A_2: Need \gamma such that A_1 \otimes \gamma \supset A_2, i.e. \gamma = A_2. But there is no A_2 in the spatial basis → 0 basis elements. It is genuinely impossible to build an equivariant A_1 \to A_2 kernel on the 3 \times 3 grid! The grid doesn't resolve cubic-order angular structure.

• E \to A_2: Need \gamma such that E \otimes \gamma \supset A_2. From the CG decomposition E \otimes E = A_1 \oplus A_2 \oplus B_1 \oplus B_2, we get \gamma = E, and there are 2 copies of E in the spatial basis → 2 basis elements. The CG coupling E \otimes E \to A_2 is the cross-product (determinant), so these basis elements are cross-products of input E-features with (x, y)-like and (x^2 - y^2, xy)-like spatial patterns.

• E \to E: Need \gamma such that E \otimes \gamma \supset E. From CG: E \otimes A_1 = E, E \otimes B_1 = E, E \otimes B_2 = E, and E \otimes E \supset E (not for D_4: E \otimes E = A_1 \oplus A_2 \oplus B_1 \oplus B_2, so no E output). Valid spatial irreps: A_1 (3 copies), B_1 (1 copy), B_2 (1 copy) → 5 basis elements.

The key insight: the spatial basis functions determine which coupling paths exist (some irrep pairs have zero paths!), while the CG coefficients determine how the irrep components couple along each path. Each path gets one scalar weight.

6.3) Parameterizing the Steerable Filter

A steerable filter is a scalar linear combination of spatial basis functions:

where the sum runs over irrep labels \alpha and copies u, with one scalar weight w_{\alpha u} per (irrep, copy) pair. Crucially, the weight does not depend on the component index i — the same scalar multiplies all d_\alpha components of the irrep. This is Schur's lemma: an equivariant map within an irrep block must be proportional to the identity, so the only freedom is one scalar per block. If the weight varied across components, the filter would break equivariance.

Because each basis function transforms as a known irrep D_\alpha, the transformation of the entire filter under the group is fully determined — it inherits the irrep structure of the basis.

For the convolution to be equivariant — transforming the input by g must transform the output by g in the corresponding way — the filter must couple three representations consistently: the input irreps D_{\text{in}}, the spatial basis function irreps D_\alpha, and the output irreps D_{\text{out}}. We achieve this by building the filter from equivariant pieces:

• The basis functions B^\alpha_{u,i}(\mathbf{y}) transform as known irreps D_\alpha (we just showed this)
• CG coefficients couple D_{\text{in}} \otimes D_\alpha \to D_{\text{out}} equivariantly (by construction)
• Scalar weights on each CG path preserve equivariance (Schur's lemma)

The entire convolution is equivariant because it is assembled from equivariant components.

7) Tensor Products, Linear Layers, and Schur's Lemma

The steerable convolution couples three representations: input features (\rho_{\text{in}}), spatial basis functions (\rho_{\text{filter}}), and output features (\rho_{\text{out}}). We already have the tool for this: reduce_tensor_product(ρ_in, ρ_filter, ρ_out), which finds the Clebsch–Gordan (CG) coefficients — all equivariant maps from the tensor product \rho_{\text{in}} \otimes \rho_{\text{filter}} into \rho_{\text{out}}.

7.1) Finding the Equivariant Basis

reduce_tensor_product returns n_{\text{sol}} CG solutions C_1, C_2, \ldots, C_{n_{\text{sol}}}, each a [d_{\text{out}}, d_{\text{in}}, d_{\text{spatial}}] tensor in the irrep basis. To get pixel-space kernels, we contract the spatial irrep index with the spatial basis functions B:

Each K_w is a [d_{\text{out}}, d_{\text{in}}, n_{\text{pixels}}] tensor — a valid equivariant kernel on its own, with the CG coupling and spatial structure combined into a single object that maps input irrep features to output irrep features at each pixel offset. The most general equivariant kernel is a learned linear combination:

The n_{\text{sol}} scalars \alpha_w are the only learnable parameters (per channel pair). The number of solutions — the dimension of the space of equivariant kernels — is determined entirely by the group and the irrep content.

Why is this equivariant?

1. Each solution K_w is equivariant. It comes from reduce_tensor_product, which finds intertwiners: maps that commute with the group action by construction.

2. Scalar combinations preserve equivariance. If K_1 and K_2 are equivariant, then \alpha_1 K_1 + \alpha_2 K_2 is too — equivariance is a linear property. Schur's lemma tells us that a scalar weight per path is the only freedom, so this parameterization is complete.

7.2) Understanding the Graphic: TP + Linear Layer

The graphic below shows a conceptually separated view of the same operation: first the tensor product exposes all n_{\text{sol}} CG paths, then an equivariant linear layer (one scalar weight per path) combines them into the desired output. This is not a different computation — it is the same K = \sum_w \alpha_w K_w drawn in two stages to make the structure visible. Both the step-by-step and dense notebooks produce the same basis elements K_w and apply the same learned linear combination in the SteerableConv2d forward pass.

It may look like there are two sets of weights — one in the tensor product and one in the linear layer — but they are equivalent to a single set. The tensor product produces n_{\text{sol}} independent equivariant solutions; the linear layer applies one scalar weight to each. Fusing these gives K = \sum_w \alpha_w K_w — one scalar per solution, the same as above.

The linear layer itself is constrained by Schur's lemma: an equivariant linear map between concatenated irrep spaces must be block-diagonal, with one scalar weight per irrep copy. The interactive graphic below (from the representations note) illustrates this for P(3) — the same principle applies to D_4 and any other group:

7.3) Channels and Connectivity

Recall the full steerable convolution equation from above: the weight w^{(\gamma, c')(\alpha, c)}_{\beta, u} has indices for output irrep \gamma and channel c', input irrep \alpha and channel c, spatial irrep \beta, and spatial copy u. In principle, one could parameterize these weights with additional structure — for example, applying a learned matrix to the filter's spatial components before contracting with CG coefficients, or allowing different irrep content per channel. This is what production libraries like e3nn and escnn do.

In our course implementation (symm4ml/steerable.py, ~250 lines), we make a pedagogical trade-off: we collapse the irrep and channel structure so that the weight tensor is simply [c_out, c_in, n_basis] — one learnable scalar per equivariant basis element per channel pair. This works because:

1. All channels carry the same irrep content, so the CG basis elements are identical for every (input channel, output channel) pair — only the scalar weights differ.
2. The channel dimension is fully connected — every output channel mixes all input channels, just like a standard conv layer.
3. The entire operation fuses into a single F.conv2d call: reshape the irrep dimension into the channel dimension, apply the convolution, and reshape back.

This design is deliberately parallel to group convolution (where the filter is [c_out, c_in, pixels, group]), making it easier to see what changes and what stays the same. The cost is that it is not the most general parameterization — but it keeps the code readable and tightly connected to the theory.

8) Activation Functions

We've seen that in the regular representation, pointwise nonlinearities like ReLU are equivariant because D^{\text{reg}}(g) is a permutation matrix — it reshuffles components without changing values. But in the irrep basis, features transform under rotation/reflection matrices that mix components and change signs. As we showed in the group convolution note, even for 1D irreps like B_1, applying ReLU before vs. after a sign flip gives different results: \text{ReLU}(-x) \neq -\text{ReLU}(x).

So how do steerable networks apply nonlinearities without breaking equivariance?

8.1) Norm-Based Activations

The key insight is that the norm of an irrep feature vector is invariant under the group action. For a d-dimensional irrep with feature vector \mathbf{x} \in \mathbb{R}^d:

because irrep matrices are orthogonal (unitary). So we can safely apply any nonlinearity \sigma to the norm, then rescale:

This is equivariant because:

1. \|\mathbf{x}\| is invariant → \sigma(\|\mathbf{x}\|) is invariant → the scale factor \sigma(\|\mathbf{x}\|)/\|\mathbf{x}\| is invariant
2. Multiplying by an invariant scalar preserves the transformation law: D(g)\bigl[\lambda\,\mathbf{x}\bigr] = \lambda\,D(g)\mathbf{x}

For 1D irreps (A_2, B_1, B_2), this reduces to:

The magnitude goes through the nonlinearity; the sign (which carries the representation information) is preserved.

For the trivial irrep (A_1), any pointwise nonlinearity is already equivariant since D(g) = 1 for all g. This means A_1 features can also have learnable biases — \text{ReLU}(x + b) is equivariant for A_1.

Biases on norms. For non-trivial irreps, the norm \|\mathbf{x}\| is an invariant scalar, so we can add a learnable bias to the norm: \sigma(\|\mathbf{x}\| + b) \cdot \mathbf{x} / \|\mathbf{x}\|. The bias shifts the activation threshold without breaking equivariance. What you cannot do is add a bias directly to the feature vector itself — \mathbf{x} + \mathbf{b} breaks the transformation law because \mathbf{b} does not transform under the group.

8.2) Gated Activations

An alternative approach is to gate each non-trivial irrep with a learned scalar from the trivial representation. The idea:

1. For each non-trivial irrep channel \mathbf{x}_\alpha, create a companion A_1 (trivial) scalar g_\alpha
2. Pass the gate through a sigmoid: \sigma(g_\alpha) \in (0, 1)
3. Multiply: \mathbf{x}_\alpha \;\mapsto\; \sigma(g_\alpha) \cdot \mathbf{x}_\alpha

This is equivariant because \sigma(g_\alpha) is a scalar that transforms trivially (unchanged by the group), and multiplying an equivariant feature by an invariant scalar preserves equivariance. The trade-off is that gated activations require extra A_1 channels — one gate per non-trivial irrep block.

9) Putting It All Together: Steerable Tetris

wget -O symm4ml_latest.zip https://symm4ml.mit.edu/_static/symm4ml_s26/symm4ml/symm4ml_latest.zip
unzip -o symm4ml_latest.zip
# The annotated source is at symm4ml/steerable.py

We return to the tetromino classification problem from the group convolution note, now using a steerable CNN instead of a regular-rep group CNN. Same task: classify the seven tetromino shapes under D_4, training on one centered image per class.

9.1) Network Architecture

The steerable CNN has the same high-level structure as the group conv version, but every component works in the irrep basis:

The key differences from the group conv architecture:

1. Features are irrep vectors, not group-indexed. Each spatial position carries a vector of irrep components (e.g. A_1 \oplus A_2 \oplus B_1 \oplus B_2 \oplus 2E = 8-dim) rather than |G| = 8 group-element-indexed values. The irrep content is a hyperparameter — you choose how many copies of each irrep to include.

2. Filters are CG basis × learned scalars. Instead of free filter weights rotated by a filter bank, the filter is a linear combination of fixed equivariant basis elements, with one learned scalar per CG path per channel pair.

3. Activations are norm-based. ReLU on A_1 scalars (with learned bias); \sigma(\|\mathbf{x}\| + b) \cdot \mathbf{x}/\|\mathbf{x}\| on non-scalar irreps.

4. Invariant pooling projects to A_1. Instead of averaging over the group dimension, we extract the A_1 component (index 0 of the irrep vector).

9.2) Computing the Equivariant Basis

There are two approaches, both implemented in companion notebooks:

• Steerable Tetris: Step-by-Step — enumerates each CG path explicitly (piecewise)
• Steerable Tetris: Dense — solves for all equivariant kernels in one call (dense)

Both produce the same set of equivariant basis elements. Each basis element is a fixed [d_{\text{out}}, d_{\text{in}}, n_{\text{pixels}}] tensor. The number of basis elements equals the number of independent CG paths — this is the number of learnable scalar weights per channel pair.

Both notebooks feed the basis elements into the same SteerableConv2d module, which stores them as a buffer and learns scalar weights of shape [c_{\text{out}}, c_{\text{in}}, n_{\text{basis}}].

9.3) Computational Trade-offs

The steerable approach introduces its own computational considerations:

• CG tensor size. The piecewise approach computes many small CG contractions (one per irrep triple). The dense approach builds a single large CG tensor. For our tetris example with uniform channels and a small hidden rep, the dense tensor fits easily in memory — but for large irrep content with many multiplicities, the piecewise approach with its sparsity (skipping zero CG paths) is essential.

• GPU efficiency. The piecewise triple-loop with many small reduce_tensor_product calls is not GPU-friendly. The dense approach vectorizes better but produces a large intermediate tensor. In our tetris example, we simplify by using uniform channel count across all irreps — every channel carries the same irrep content — which allows fusing into a single conv2d call. Production frameworks like e3nn and escnn handle the general case with irregular multiplicities, sparse CG tensors, and optimized GPU kernels.

• Numerical precision. The CG basis has float-valued entries (unlike the permutation matrices of the regular rep), so equivariance error accumulates with float32 arithmetic. For the tetris model this is ~10^{-4} (exact in float64).

Managing this complexity is the key reason to use a dedicated package like e3nn or escnn in production. They handle irrep content specification, CG path enumeration, sparse tensor operations, and activation function constraints — letting you focus on the network architecture rather than the bookkeeping.

10) Summary

The key difference between group conv and steerable conv: in group conv, equivariance is enforced by summing over the group (rotating the filter by every group element). In steerable conv, equivariance is built into the basis via CG coefficients — there is no sum over the group at runtime. The spatial convolution (translation) is identical in both; only the parameterization of the filter changes.

For continuous groups like \mathrm{SO}(3), the regular representation is infinite-dimensional, so the group-conv approach (summing over all group elements) is not directly applicable. The steerable framework is what is primarily used in practice: the spatial basis functions become spherical harmonics, the CG coefficients are the standard Wigner 3j symbols, and the learnable scalars become radial functions. This is the basis on which libraries like e3nn and escnn operate. (We will also discuss spherical convolution, which is a special case that operates on signals defined on the sphere rather than in \mathbb{R}^3.)

References:

• T. Cohen & M. Welling, Group Equivariant Convolutional Networks, ICML 2016.
• N. Thomas, T. Smidt et al., Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds, 2018.
• R. Kondor & S. Trivedi, On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups, ICML 2018.
• M. Weiler & G. Cesa, General E(2)-Equivariant Steerable CNNs, NeurIPS 2019.
• M. Weiler, P. Forré, E. Verlinde & M. Welling, Equivariant and Coordinate Independent Convolutional Networks, 2023. (See Part I, Chapters 5–6 for the kernel constraint and its solution.)

Related exercises: Steerable Convolution Exercise | Steerable Tetris Colab

Previous: Group Convolution