Group Convolution

These notes are being transcribed, condensed, and illustrated. They may lag or omit details, please refer to the handwritten notes for the full lecture.

This note develops group equivariant convolution — a generalization of the standard convolution operation that is equivariant not just to translations, but to an arbitrary symmetry group G. The key idea: instead of sliding a filter over spatial positions (translation equivariance), we also rotate and reflect the filter according to the group, achieving equivariance to the full symmetry group.

1) Motivation

1.1) Why Group Convolution Now?

Building on what we've learned about tensor product representations in vibrational modes — and more broadly, how fields of representations (representations living at points in space) transform under symmetry — we're now ready to tackle more ML operations on discrete spaces.

So far, we've seen how representation theory constrains one fundamental ML operation: the linear layer y = Wx. Schur's lemma tells us that if W commutes with a group action (i.e. is equivariant), it must be block-diagonal in the irrep basis, with each block proportional to the identity. This drastically reduces the number of free parameters.

Now we turn to the other workhorse of modern ML: convolution. Convolution is the core operation in convolutional neural networks (CNNs), which dominate image processing, signal processing, and many scientific applications.

1.2) Symmetries Beyond Translation

Standard convolutional neural networks (CNNs) are translation equivariant: shifting the input shifts the output by the same amount. This is why CNNs work so well for image recognition — a cat is a cat regardless of where it appears in the image.

But many domains have additional symmetries beyond translation:

• Image classification: A cat rotated 90° is still a cat. Standard CNNs must learn rotated versions of the same feature separately.
• Molecular property prediction: Physical properties are invariant to rotation and reflection that match the symmetry of the molecule.
• Crystallography: Crystal properties must respect the space group symmetry.
• Particle physics: Detector readouts may have discrete rotational symmetry.

Group equivariant convolution builds these symmetries directly into the architecture, so the network doesn't have to learn them from data. This was introduced by Cohen & Welling (ICML 2016) and has since become a foundational tool in geometric deep learning.

1.3) Roadmap

This note has two main parts.

Part 1: Convolution on the translation group through the lens of representation theory. We start with ordinary convolution on \mathbb{Z}_N — the familiar operation behind standard CNNs — and examine it from a group-theoretic perspective. Shifts of a signal are just the regular representation of \mathbb{Z}_N, and cross-correlation can be written as a dot product involving D^{\text{reg}}(x). When we decompose the regular representation into irreps, we recover the Fourier transform (the irreps of \mathbb{Z}_N are roots of unity, and the change-of-basis matrix Q is the DFT matrix). The convolution theorem — that convolution becomes pointwise multiplication in the Fourier basis — then follows directly from Schur's lemma and the Haar measure (orthogonality of group elements). This irrep decomposition of the translation group foreshadows what we'll do later: applying the same idea to the group index in group convolution to arrive at steerable basis functions and the steerable convolution framework.

Part 2: Group equivariant convolution. We then ask: what happens when our data carries an index over an arbitrary finite group G, not just translations? We develop group cross-correlation on the regular representation and show how to implement it as a filter bank that rotates and reflects the kernel according to G. When this group action is combined with spatial translations, we get the full group equivariant convolution operating on a tensor product of spatial and group representations.

2) From Objects to Fields

Before diving into the convolution formula, let's identify what's conceptually new. So far in this course, groups have acted on objects — atoms, vectors, displacement modes. Now we'll see groups acting on functions over a fixed domain. This distinction matters for how we think about transformations.

2.1) Active vs. Passive Transformations

When a group acts on a discrete set of objects (atoms in a molecule, for example), each object physically moves to a new location and carries its properties along. But when a group acts on a function defined over a fixed domain (pixel values on a grid, a signal on the group), the domain stays put and we must decide what it means to "transform the function." This leads to a fundamental distinction:

The interactive graphics below illustrate this distinction concretely. We start with a familiar example — the water molecule from vibrational modes, acted on by the mirror \sigma_v(yz). Toggle between the identity and the mirror to see the same transformation from both viewpoints:

Because \sigma_v is its own inverse (\sigma_v^{-1} = \sigma_v), the "look up the pre-image" step in the passive view happens to use the same operation as the active view — so the g^{-1} doesn't look any different from g. To see why the inverse really matters, here's a square acted on by a 90° rotation, where R_{90}^{-1} = R_{270} \neq R_{90}:

This active/passive distinction will show up concretely when we write the cross-correlation formula: there will be two equivalent forms — one that "moves the filter" and one that "transforms the input function" (with the characteristic inverse g^{-1}).

3) Cross-Correlation and the Translation Group

3.1) Standard Cross-Correlation Recap

In a standard CNN, the fundamental operation is cross-correlation (often loosely called "convolution"). Let's build it up carefully, because the same logic will generalize to groups.

We have two functions:

• f: \mathbb{Z} \to \mathbb{R} is the input signal — the data we want to analyze (e.g. the pixel values of an image).
• \psi: \mathbb{Z} \to \mathbb{R} is the filter (or kernel) — a small, learnable pattern that the network slides across the input to detect features. For example, \psi might be an edge detector.

Cross-correlation computes the dot product (overlap) between the filter and the input at each position x. The standard formula is:

This is symmetric in interpretation: it measures how much f and \psi overlap when one is shifted relative to the other. The conventional reading is shifting the filter — we slide \psi to position x (the argument y - x centers the filter there), multiply element-wise with f, and sum.

There is an equivalent way to write this. Substituting y' = y - x (a relabeling of the summation index):

Now the filter \psi(y') stays fixed at the origin, and instead we shift the input by +x. The two formulas produce identical products — the only question is "who moves?" This is essentially the rearrangement theorem for the translation group: summing over all y is the same as summing over all y' = y - x, because translation is a bijection on \mathbb{Z}. The visualization below shows both views side by side:

This equivalence — shifting the filter by -x vs. shifting the input by +x — is exactly the pattern that generalizes to groups, where the rearrangement theorem says that summing over all group elements h is the same as summing over all gh (because left-multiplication by g is a bijection on G).

The output [f \star \psi](x) is itself a function of position. Mathematically, it is simply the overlap between f and \psi as a function of their relative shift x. In the ML context, we think of f as data and \psi as a learnable pattern, so we interpret the output as "how strongly does the filter \psi match the input at each location x?" — but the formula itself is symmetric.

The key property is translation equivariance: if we shift the input f by some amount t, the output shifts by the same amount. Formally, defining the translation operator [T_t f](x) = f(x - t):

This works because the overlap only depends on the relative shift between f and \psi, not on any absolute position. Shifting f by t shifts the overlap function by the same amount — the operation is purely relational. Translation equivariance is built into the architecture; the network never has to learn it.

3.2) Data on the Group vs. Data in a Representation

So far in this course, our data has lived in vector spaces that groups act on — representations. A displacement vector lives in \mathbb{R}^{3n}, and the group acts on it via matrices D(g). The data lives in the vector space; the group tells us how to transform it.

For group convolution we adopt a different perspective: data as a function on the group itself — a value assigned to each group element, f: G \to \mathbb{R}.

Standard convolution, as we've just seen, is convolution on the translation group \mathbb{Z} (or \mathbb{Z}_N for periodic signals). For group convolution, we replace \mathbb{Z} with an arbitrary finite group G. Later, when we work with signals on the sphere, we'll see that convolution can also be formulated for data on spaces the group acts on (like S^2) rather than on the group directly — spherical harmonics will play the role of irrep components.

What does this mean concretely? For C_4 = \{E, R_{90}, R_{180}, R_{270}\}, a function on the group is just a list of 4 numbers: f = [f(E),\, f(R_{90}),\, f(R_{180}),\, f(R_{270})]. Each number is the value associated with that orientation. For an image processing network, f(R_{90}) might represent "how strongly is a certain feature detected at 90° orientation."

3.3) The Translation Group \mathbb{Z}_N

Before moving to general groups, let's look more carefully at what standard convolution is doing algebraically. The key insight: shifting an image is a permutation of pixel values, so translation equivariance is a special case of the representation theory we've already developed.

Shifts are permutations. Consider a 1D signal with N values: f = [f_0, f_1, \ldots, f_{N-1}]. Shifting by k positions moves the value at position n to position n + k (mod N). This is a permutation of the vector components — "shift by k" is a permutation matrix. The set of all shifts \{0, 1, \ldots, N-1\} under addition mod N forms the cyclic group \mathbb{Z}_N, and the shift matrices give a permutation representation of \mathbb{Z}_N. This is exactly the same idea as the permutation representation of atomic sites from vibrational modes — just applied to pixel positions instead.

The infinite/periodic caveat. This clean permutation picture requires that every shift is a bijection — no data falls off the boundary. That holds for infinite signals (\mathbb{Z}) or periodic signals (\mathbb{Z}_N), but not for a finite signal with a finite filter: shifting pushes data off one end, which is why practical CNNs need zero-padding or other boundary handling. The exact algebraic story we develop here assumes periodic (or infinite) signals.

Multi-dimensional images. A 2D image is a function on \mathbb{Z}_{n_1} \times \mathbb{Z}_{n_2}, with one cyclic group per spatial dimension. Translations in different dimensions commute — shifting right then down is the same as shifting down then right — so the full translation group is a direct product \mathbb{Z}_{n_1} \times \mathbb{Z}_{n_2}. Because the factors commute, we can handle each dimension independently: the irrep decomposition of \mathbb{Z}_{n_1} \times \mathbb{Z}_{n_2} is the tensor product of the decompositions for each factor, and the 2D DFT is just two 1D DFTs applied sequentially (first along rows, then along columns). For this reason, we develop the theory for a single \mathbb{Z}_N and let the multi-dimensional case follow by direct product.

3.4) \mathbb{Z}_N Acting on Pixels = the Regular Representation

Not every permutation representation is a regular representation — for example, C_{3v} acting on the 3 atoms of water gives a 3D permutation rep, but the regular rep of C_{3v} is 6-dimensional. What's special here is that \mathbb{Z}_N acts on exactly N positions by cyclic shift: the action is transitive (any position can reach any other) and free (no non-identity shift fixes any position). This means each position can be identified with a group element, and the shift action is just left multiplication — which is the definition of the regular representation.

We'll see later why this matters; for now, note that the regular rep contains every irrep of the group. And since \mathbb{Z}_N is abelian, all its irreps are 1-dimensional and each appears exactly once — so the decomposition gives a basis that diagonalizes every shift matrix simultaneously.

3.5) Irrep Decomposition and the Convolution Theorem

Cross-correlation involves the regular representation. Cross-correlation on \mathbb{Z}_N is defined as:

Substituting m = n + x (so n = m - x), this becomes \sum_m f(m)\,\psi(m - x). The factor \psi(m - x) is exactly D^{\text{reg}}(x) acting on \psi, since (D^{\text{reg}}(x)\boldsymbol{\psi})_m = \psi(m - x). So we can write cross-correlation as:

This is the key observation: computing cross-correlation at every shift x requires multiplying by the regular representation matrix D^{\text{reg}}(x) for each group element. If we could find a basis that diagonalizes all these matrices simultaneously, the triple sum over positions would factor into something much simpler. We already know such a basis exists — the irrep decomposition of \Gamma^{\text{reg}}. Let's find it explicitly for \mathbb{Z}_N.

3.5.1) Finding the Irrep Basis

From the decomposition note, we know that any representation can be decomposed into irreps via a change-of-basis matrix Q that block-diagonalizes all the representation matrices simultaneously. Let's carry this out for the regular representation of \mathbb{Z}_N. A detailed version of this derivation is here.

Step 1: Find the irreps of \mathbb{Z}_N. Since \mathbb{Z}_N is abelian, all irreps are 1-dimensional (recall: in an abelian group, every element commutes with everything, so by Schur's lemma every irrep matrix must be a scalar). A 1D representation of \mathbb{Z}_N is a map \Gamma: \mathbb{Z}_N \to \mathbb{C}^* satisfying \Gamma(a + b) = \Gamma(a)\Gamma(b). Since the generator 1 has order N, we need \Gamma(1)^N = \Gamma(0) = 1, so \Gamma(1) must be an N-th root of unity: \Gamma(1) = e^{2\pi i j/N} for some j \in \{0, 1, \ldots, N-1\}. This determines the entire representation:

There are exactly N distinct irreps \Gamma_0, \Gamma_1, \ldots, \Gamma_{N-1}, each 1-dimensional — and the regular representation is N-dimensional, so the decomposition \Gamma^{\text{reg}} = \bigoplus_j \Gamma_j accounts for all dimensions perfectly.

Step 2: Find the eigenvectors of the shift matrix. The shift-by-1 matrix P_1 (the regular representation matrix for the generator) is the cyclic permutation matrix. We need to find its eigenvectors. If P_1 \mathbf{v} = \lambda \mathbf{v}, then applying P_1 N times gives \mathbf{v} = \lambda^N \mathbf{v}, so \lambda^N = 1 — the eigenvalues are exactly the N-th roots of unity \omega^j. For eigenvalue \omega^j, the eigenvector must satisfy (P_1 \mathbf{v})_n = v_{n-1} = \omega^j v_n, giving v_n = \omega^{jn} v_0. Normalizing:

Step 3: Assemble the change-of-basis matrix. The matrix Q whose rows are \mathbf{v}_j^\dagger (equivalently, whose columns are the eigenvectors of P_1^\dagger) diagonalizes every shift matrix:

where \omega is the primitive N-th root of unity:

This is exactly the Discrete Fourier Transform (DFT) matrix — the irrep decomposition of the translation group is what the rest of the world calls the Fourier transform. The DFT didn't come from signal processing intuition; it fell out of representation theory. As we'll see, the idea of a "Fourier transform" generalizes to any group; the usual DFT is the special case for \mathbb{Z}_N.

3.5.2) Applying the Decomposition

With Q in hand, let's return to our cross-correlation formula [f \star \psi](x) = \mathbf{f}^T\, D^{\text{reg}}(x)\, \boldsymbol{\psi} and change to the irrep basis.

Step 1: Change to the irrep basis. As shown in the graphic above, Q diagonalizes all regular representation matrices: Q\, D^{\text{reg}}(x)\, Q^\dagger = \Lambda(x), where \Lambda(x) is diagonal with entries \lambda_j(x) = \Gamma_j(x) = \omega^{jx} = e^{2\pi i jx/N} (the character of irrep \Gamma_j at shift x). To use this, we insert the identity I = Q^\dagger Q on both sides of D^{\text{reg}}(x):

Step 2: Identify the irrep components. The right factor is Q\boldsymbol{\psi} = \hat{\boldsymbol{\psi}} — the irrep components of \psi. For the left factor, since \mathbf{f} is real we have \mathbf{f}^T = \mathbf{f}^\dagger, so \mathbf{f}^T Q^\dagger = \mathbf{f}^\dagger Q^\dagger = (Q\mathbf{f})^\dagger = \hat{\mathbf{f}}^\dagger. Note: although f is real in the position basis, its Fourier transform \hat{\mathbf{f}} is complex, so the conjugation in \hat{\mathbf{f}}^\dagger matters. This gives:

The triple sum over positions has collapsed to a single sum over irreps, where each term involves only the j-th irrep components of f and \psi.

Step 3: Interpret the result. Let's define c_j = \hat{f}_j^* \, \hat{\psi}_j — a single complex number for each irrep j, computed entirely from the irrep components of f and \psi. Then our formula reads:

Notice what this sum is: the factor \omega^{jx} = \Gamma_j(x) is the character of irrep \Gamma_j evaluated at group element x, and we are summing over irreps with coefficients c_j. This is the inverse Fourier transform — it converts from the irrep basis back to the group element basis. In other words, the coefficients c_j are the irrep components (Fourier coefficients) of the output:

This is the convolution theorem: cross-correlation becomes pointwise multiplication in the Fourier (irrep) basis. We didn't assume this — it fell directly out of the change of basis Q that diagonalizes the regular representation.

What changes for non-abelian groups? For a non-abelian group G, some irreps have dimension d_j > 1. The regular representation still decomposes into irreps, and convolution still separates into independent operations per irrep block. But within each block, the structure changes. In the abelian case, \Gamma_j(g) = e^{2\pi i jg/N} is a scalar, so we can compute c_j = \hat{f}_j^* \hat{\psi}_j first (independent of g) and then multiply by \Gamma_j(g) to transform back to the group element basis. For d_j > 1, the irrep matrix \Gamma_j(g) is d_j \times d_j and sits between \hat{\mathbf{f}}_j and \hat{\boldsymbol{\psi}}_j:

However, we can still separate the "combine f and \psi" step from the "transform back to group elements" step. Writing the expression in index notation:

Since these are all scalars inside the sum, we can rearrange and group the g-independent factors into an outer product [C_j]_{ab} = (\hat{\psi}_j)_a \,(\hat{f}_j)_b^*, i.e. C_j = \hat{\boldsymbol{\psi}}_j \,\hat{\mathbf{f}}_j^\dagger (a d_j \times d_j matrix):

Now the g-independent part is isolated: we first compute the outer product C_j (independent of g), then trace against the irrep matrix \Gamma_j(g) to get back to the group element basis:

We'll see this explicitly for C_4 (abelian, all irreps 1D) and then for larger groups later in the course.

4) Integration over the Group: the Haar Measure

We've seen that changing to the irrep basis diagonalizes the regular representation and turns convolution into pointwise multiplication. There's a closely related principle that will keep showing up: when you sum (or integrate) over the entire group, only the trivial irrep survives. Everything else cancels.

The finite-group statement. For a finite group G, define the "group average" of a representation matrix:

For the trivial irrep, \Gamma_{\text{trivial}}(g) = 1 for all g, so the sum is trivially 1. For any non-trivial irrep, the matrix elements oscillate and cancel out — the same way \sum_{k=0}^{N-1} e^{2\pi i jk/N} = 0 for j \neq 0.

Connection to Fourier analysis. For \mathbb{Z}_N, \Gamma_j(k) = \omega^{jk}. The sum \sum_k \omega^{jk} is the sum of all N-th roots of unity raised to power j — this vanishes unless j = 0 (the trivial irrep / zero frequency). This is the same reason sinusoids cancel when summed over a full period: a non-trivial irrep winds around the unit circle and the contributions average to zero. The Haar measure statement generalizes this to arbitrary groups.

Why "Haar measure"? For finite groups, the Haar measure is just the uniform sum \frac{1}{|G|}\sum_g. For continuous groups like SO(3), it becomes an integral \int_{SO(3)} dg with a specific measure that is invariant under the group action. The principle is the same: integration over the group kills all non-trivial irrep components.

Where this shows up. This principle will appear repeatedly in what follows:

• Invariant output: Averaging features over the group dimension projects onto A_1 — this is how we get invariant output for classification (see Invariant Output for Classification below).
• Selection rule: The spatial sum \sum_\mathbf{y} in convolution is integration over pixel positions — orthogonality of different irrep basis functions is the same principle (see The Selection Rule below).
• Convolution theorem: The block-diagonalization we just derived is intimately related — Schur's orthogonality relations (which underpin the Haar measure result) are what make the irrep basis special.

5) Group Cross-Correlation

Now we're ready for the general formula. Both the input and filter are functions on the group:

• f: G \to \mathbb{R} is the input — it assigns a value to each group element (e.g. a feature response at each orientation).
• \psi: G \to \mathbb{R} is the filter — a learnable pattern defined on the group.

There are two equivalent ways to write group cross-correlation. The first generalizes "slide the filter" from standard CNNs:

Reading this the same way as before: for each "position" g, we evaluate \psi at the relative group element g^{-1}h (the "displacement" from g to h), multiply by f(h), and sum. This is the form you will encounter in many papers.

Substituting h' = g^{-1}h (i.e. h = gh') and applying the rearrangement theorem — summing over all h \in G is the same as summing over all h' = g^{-1}h \in G, since left-multiplication by g^{-1} is a bijection on G — gives an equivalent form:

Now the filter \psi stays fixed, and we transform the input by g: evaluate f at gh' for each h', then take the inner product with \psi.

The cross-correlation avoids the inverse because we chose to transform f by the group element g directly (evaluating f(gh)), rather than shifting the filter \psi. The inverse only appears when we define what "translating f as a function" means — the operator L_g that describes how f changes under the group action. This is exactly the active vs. passive distinction from above.

6) Group Convolution via the Regular Representation

This is the central section: connecting the abstract cross-correlation formula to concrete matrix operations using the regular representation.

6.1) From Summation to Matrix Form

Starting from the act-on-input form [f \star \psi](g) = \sum_{h \in G} f(gh)\,\psi(h), we expand f(gh) using a Kronecker delta:

The key observation: \delta_{i,\,gh} is exactly an entry of the regular representation matrix. Recall that D^{\text{reg}}(g) is a permutation matrix defined by D^{\text{reg}}(g)\lvert h\rangle = \lvert gh\rangle, so its (i,h) entry is 1 when i = gh — i.e., D^{\text{reg}}(g)_{ih} = \delta_{i,\,gh}. Therefore:

No inverse or transpose was needed — the regular representation matrix appears directly. This is bilinear in f and \psi, with D^{\text{reg}}(g) providing all the structure. The regular representation encodes the group's multiplication table, and that's all we need.

6.2) Channels and Batches

So far our signals have been scalar-valued: f: G \to \mathbb{R} assigns a single number to each group element. In practice, data often has multiple channels — independent scalar values at each position. The most familiar example is a color image: each pixel has three channels (red, green, blue), so the image is a function f: \mathbb{Z}^2 \to \mathbb{R}^3. Similarly, a function on the group with c input channels is f: G \to \mathbb{R}^c.

The filter \psi maps c input channels to d output channels, so it has shape \psi: G \to \mathbb{R}^{d \times c} — for each group element, a d \times c matrix of learnable weights. Each output channel combines information from all input channels, just like in a standard CNN.

We also process multiple examples simultaneously in a batch (indexed by z). This is standard practice in neural network training — it doesn't change the math, just adds an extra index.

6.3) The Einsum Formulation

With channels and batches, our indices are:

• z = batch index (which example in the batch)
• c = input channel, d = output channel
• i, j, g = group element indices

Group convolution becomes:

This triple contraction is a single einsum call — you will implement this in the Group Convolution Exercise.

The tensor shapes:

• reg_rep has shape [|G|, |G|, |G|] — one |G| \times |G| permutation matrix per group element.
• input has shape [\text{batch}, c_{\text{in}}, |G|] — for each sample and channel, a function on the group.
• filter has shape [c_{\text{out}}, c_{\text{in}}, |G|] — learnable weights, also a function on the group.
• output has shape [\text{batch}, c_{\text{out}}, |G|] — the convolved result, again a function on the group.

Since D^{\text{reg}}(g)_{ij} = \delta_{i,\,gj}, the contraction over i collapses to f_{zc,\,gj}, recovering the act-on-input form \sum_j f(gj)\,\psi(j). This is why the code needs no inverse.

Reading the einsum. The three indices of D^{\text{reg}} are [g, i, j] where g selects which matrix, i is the row, and j is the column. Recall the regular representation is constructed so that D^{\text{reg}}(g)_{i,j} = 1 when i = g \cdot j — i.e., the indices are [\lvert g\rangle, \lvert gh\rangle, \lvert h\rangle]. In the einsum, i contracts with f and j contracts with \psi: for each output element g, the regular representation transforms f's group index (looking up f at g \cdot j), then we take the inner product with \psi. This is NOT a Clebsch–Gordan decomposition (where i \otimes j \to g would label irreps) — it is the group algebra product, where g, i, j are all group-element indices within the regular representation.

After the exercise, compare with reg_rep_group_convolution in symm4ml/group_conv.py.

6.4) Worked Example with C_4

Let's make this concrete with the cyclic group C_4 = \{E, R_{90}, R_{180}, R_{270}\} (rotations by 0°, 90°, 180°, 270°).

Multiplication table (row \circ column):

Each row of the multiplication table gives a permutation, which becomes a regular representation matrix. For example, the row for R_{90} is (R_{90}, R_{180}, R_{270}, E), meaning R_{90} sends E \to R_{90}, R_{90} \to R_{180}, R_{180} \to R_{270}, R_{270} \to E.

The interactive visualization below computes [f \star \psi](g) for f = [1, 0, 2, 1] and \psi = [3, 1, 0, 2]. Click each output element g to see both conventions side by side: the left panel acts on the filter (\psi(g^{-1}h)), the right panel acts on the input (f(gh)). Both give identical products and the same sum:

7) Input Data and the Lifting Convolution

The discussion above assumes our data is already a function on the group: f: G \to \mathbb{R}^c. But real data — like images — doesn't naturally carry a group index. An image is a function f: \mathbb{Z}^2 \to \mathbb{R}^c, with no group dimension.

How do we get from image data to group-indexed data? We need to lift the input to the regular representation.

The key insight: the input image is invariant under the group action on the group index — it doesn't change when you permute group copies, because all copies are identical. In representation-theoretic terms, the input lives in the trivial representation: the all-ones vector [1, 1, \ldots, 1] of length |G| is invariant under any permutation.

In practice, we "promote" the image by repeating it along a new group dimension:

# image: [batch, c_in, height, width]
z, c, h, w = image.shape
# Add group dimension: repeat |G| times
image_promoted = image.reshape(z, c, 1, h, w).repeat(1, 1, |G|, 1, 1)
# image_promoted: [batch, c_in, |G|, height, width]

After the first group convolution layer, the |G| copies are no longer identical — the convolution with different rotated filters produces different responses for each group element. The network has learned to distinguish orientations. This promotion step is called the lifting convolution.

8) Adding Spatial Dimensions

Now we combine spatial convolution (translation equivariance) with group convolution (rotation/reflection equivariance). This is where the full power of the approach emerges — and where we reconnect with the tensor product of representations from vibrational modes.

Recall the key pattern from vibrational modes: when we have something that transforms (a displacement vector) living at sites that permute (atom positions), the full transformation is a tensor product \Gamma^{\text{a.s.}} \otimes \Gamma^{\text{vec}}. The same structure appears here, with different ingredients.

8.1) Two Groups Acting Simultaneously

For 2D images, two groups act simultaneously:

1. The translation group \mathbb{Z}^2 acts on spatial positions (shifting pixels).
2. The point group G (e.g. C_4 or D_4) acts on both:
   • Spatial positions — rotating/reflecting pixel coordinates (a permutation representation \Gamma^{\text{pix}}, analogous to \Gamma^{\text{a.s.}} for atoms)
   • The group index — permuting group channels (the regular representation \Gamma^{\text{reg}}, analogous to \Gamma^{\text{vec}} for displacements)

Just as displacements live at atom sites, group channels live at pixel positions. The filter therefore lives in a tensor product \Gamma^{\text{pix}} \otimes \Gamma^{\text{reg}} — the same tensor product structure, with pixels playing the role of atoms and the regular representation playing the role of the vector representation.

The filter shape becomes [c_out, c_in, |G|, kh, kw]:

• c_out, c_in — output and input channels (learnable)
• |G| — the group dimension (regular representation)
• kh, kw — the spatial kernel dimensions (permutation representation)

8.2) The Filter Bank: Rotating Pixels

To apply a group element to the spatial dimensions of a filter, we need to rotate and reflect the pixel grid. This is exactly a permutation representation — the same concept we used for atomic sites in Vibrational Modes!

The procedure, step by step:

Step 1: Assign coordinates to pixels. Each pixel in the kernel gets a coordinate using centered linspace coordinates (with a y-flip to match the image convention where origin is at top-left).

Step 2: Compute the permutation representation. Apply each 2 \times 2 group element matrix to the pixel coordinates. For each group element, determine which pixel maps to which — exactly as we did for atomic sites in vibrational mode analysis. The result is |G| permutation matrices of size (k_h \cdot k_w) \times (k_h \cdot k_w).

The visualization below shows how each group element permutes the pixel coordinates of a 3 \times 3 kernel — the same permutation representation from vibrational modes, applied to pixel positions instead of atomic sites:

Step 3: Apply via einsum. Permute the spatial dimensions of the filter using these matrices, producing |G| rotated copies — one per group element.

The result is the filter bank: |G| rotated copies of the original filter.

You will implement this step-by-step in the Group Convolution Exercise. Steps 1–2 are packaged as image2D_permutation_representation and Step 3 as image2D_group_convolution_filter_bank in symm4ml/group_conv.py.

Use symm4ml.plot.filter_bank() in your notebook to interactively visualize filter banks for any group. For example, after computing the permutation representation and building a filter bank, display it with:

from IPython.display import HTML
perm_rep = image2D_permutation_representation(C4, [3, 3])
fb = image2D_group_convolution_filter_bank(perm_rep, filter)
HTML(symm4ml.plot.filter_bank(fb[:, 0, 0, 0].numpy(), rep_2D=C4.numpy(), labels=['E','R₉₀','R₁₈₀','R₂₇₀']))

8.3) The Full Convolution

The full group convolution for 2D images combines three steps:

1. Create the filter bank — apply the spatial permutation representation to generate |G| rotated copies of each filter.
2. Contract with the regular representation — rotate the group channel index using D^{\text{reg}}(k), combining spatial and group transformations.
3. Apply standard spatial convolution — reshape the combined filters and use standard 2D convolution over the spatial dimensions.

You will implement the full pipeline in the Group Convolution Exercise. The reference implementation is image2D_group_convolution in symm4ml/group_conv.py, which takes the precomputed perm_rep (from image2D_permutation_representation) so that the convolution itself is pure PyTorch. Using the act-on-input convention (consistent with the rest of these notes), the full equation is:

Here k acts on f in two ways: it rotates the spatial sampling point (\mathbf{x} + k\mathbf{y} instead of \mathbf{x} + \mathbf{y}), and it permutes the group index (since D^{\text{reg}}(k)_{ij} = \delta_{i,kj} selects f at group element kj). The filter \psi_{dcj}(\mathbf{y}) stays fixed at the origin. The spatial argument is \mathbf{x} + k\mathbf{y} (not k(\mathbf{x}+\mathbf{y})) because translations and rotations combine as a semidirect product \mathbb{Z}^2 \rtimes G, with composition rule (\mathbf{x}, k) \cdot (\mathbf{y}, h) = (\mathbf{x} + k\mathbf{y},\; kh). This is the direct generalization of f(gh) from the pure group case.

8.4) The Tensor Product Structure

The filter bank construction is a manifestation of a tensor product of representations — the same structure we saw in vibrational modes, with different ingredients:

In both cases, the tensor product combines a permutation representation (what moves where) with a representation at each site (what transforms there). Decomposing the tensor product into irreps reveals the independent, symmetry-adapted components — normal modes for molecules, and irrep-labeled filter components for convolution.

9) Nonlinearities and Invariant Output

9.1) Pointwise Nonlinearities

Between convolution layers, we apply pointwise nonlinearities like ReLU. Are these equivariant?

The answer depends on how the group acts on the features. A pointwise nonlinearity \sigma applies independently to each component: [\sigma(\mathbf{x})]_i = \sigma(x_i). For this to commute with the group action D(g), we need:

This holds if and only if D(g) does not change the sign or magnitude of individual components — i.e., D(g) is a permutation matrix. Permutation matrices only reorder entries, so applying \sigma before or after the permutation gives the same result.

The regular representation D^{\text{reg}}(g) is a permutation matrix (it permutes group elements), so pointwise nonlinearities are equivariant in the regular rep:

This is true for any pointwise nonlinearity (\text{ReLU}, \sigma, \tanh, etc.).

For any non-permutation representation, pointwise nonlinearities break equivariance. Consider a feature vector transforming under a non-trivial irrep like E of D_4. A 90° rotation acts as \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}, which mixes the two components and changes their signs. Applying ReLU before vs. after this rotation gives different results:

Even the 1D irreps A_2, B_1, and B_2 have this problem — they assign -1 to some group elements, so the group action flips signs, and \sigma(-x) \neq -\sigma(x) for nonlinear \sigma.

9.2) Invariant Output for Classification

For classification tasks, we want the final output to be invariant (unchanged under any group transformation), not just equivariant. How do we go from equivariant features to an invariant output?

Average over the group dimension. This is the Haar measure principle in action — a projection onto the trivial representation A_1:

In matrix form, this is exactly the projector:

This projector picks out the component of the feature vector that transforms trivially (is invariant) under the group action — the same projection technique we used in decomposition to extract irrep components. The Haar measure guarantees that all non-trivial components cancel in this sum, leaving only the invariant part.

10) Putting It All Together: Classifying Tetrominos

To see group convolution in action, we build a group-equivariant CNN that classifies the seven standard tetromino shapes (the pieces from Tetris). This is a toy problem, but it illustrates the key ideas powerfully: the network is trained on just one image per class (centered, canonical orientation) and generalizes perfectly to all orientations and positions — no data augmentation needed.

10.1) The Tetromino Pieces

The seven tetrominos are:

Under C_4 (rotations only), all seven are distinct — no rotation maps L to J or S to Z. Under D_4 (rotations + reflections), the reflection \sigma maps L \leftrightarrow J and S \leftrightarrow Z, merging them into the same orbit. This reduces the number of classes from 7 to 5.

The achiral pieces (I, O, T) are shown in blue. The two chiral pairs are color-coded: L and J in pink and S and Z in violet. Each chiral pair consists of mirror images — like left and right hands.

10.2) Network Architecture

The group-equivariant CNN has three stages:

1. Lifting convolution: the input image (a scalar field) is convolved with a learned 3 \times 3 filter and all its rotated copies, producing a feature map indexed by the group. This is the \ell = 0 \to \text{reg} lifting step.

2. Group convolution: group feature maps are convolved using image2D_group_convolution from the library, producing new group feature maps. Filters live on both the spatial grid and the group.

3. Invariant pooling: average over spatial dimensions, then average over the group dimension (A_1 projection). The result is a vector of invariant features fed to a linear classifier.

Because the architecture is equivariant by construction, it achieves 100% test accuracy on randomly rotated and translated pieces after training on just one centered image per class.

10.3) Beyond A_1: Projecting onto Any Irrep

The A_1 (invariant) pooling treats L and J as identical under D_4 — they're in the same orbit. But what if we want to distinguish chiral pairs? Or extract vector-valued output?

The general tool is the change-of-basis matrix Q_G that block-diagonalizes the regular representation. Applying Q_G to the group-indexed feature vector f = [f(g_1), f(g_2), \ldots] extracts all irrep components at once:

The rows of Q_G corresponding to irrep \alpha extract the components of f that transform as D_\alpha. This works because of the wonderful orthogonality theorem (also called the great orthogonality theorem): the matrix elements of different irreps are orthogonal when summed over the group:

This is what makes Q_G work — the rows corresponding to different irreps (or different components of the same irrep) are orthogonal. It is the representation-theoretic generalization of the orthogonality of Fourier modes.

For 1D irreps (like A_1 or A_2), each row of Q_G reduces to the character vector [\chi_\alpha(g_1), \chi_\alpha(g_2), \ldots], giving a scalar projection. For multidimensional irreps (like E), there are d_\alpha rows per copy, extracting a d_\alpha-dimensional vector.

The pseudoscalar A_2. For the specific case of A_2, the character is \chi_{A_2}(g) = \det(g): +1 for rotations, -1 for reflections. The projection is:

This output is invariant under rotations but flips sign under reflections — exactly the definition of a pseudoscalar. In physics, chirality, helicity, and the cross product \mathbf{a} \times \mathbf{b} are all pseudoscalars.

The choice of which irrep to project onto in the final pooling layer determines what information survives:

This illustrates a general principle: the irreps of the symmetry group are not just mathematical objects — they classify the types of physical quantities a network can compute. A_1 gives scalars. A_2 gives pseudoscalars. E gives vectors. Each choice leads to different equivariance properties in the output, and Q_G (via infer_change_of_basis) is the tool that extracts any of them.

11) Summary and Looking Ahead

11.1) Standard CNN vs. Group Equivariant CNN

11.2) Looking Ahead: Steerable Convolutions

Related exercises: Group Convolution Exercise | Tetris Colab