A quick recap on symmetry, equivariance and invariance

When people first hear about symmetry, they often think about invariance: if the input rotates, the output stays the same. That is useful at the very end of a predictive model, where we only care about the class label. But inside a deep network, we usually don’t want to throw away pose information too early. Later layers still need to know whether an edge is vertical, horizontal, or diagonal. A filter \(\psi\) is equivariant if transforming the input and then applying the network gives the same result as applying the network first and then transforming the features:

It says the feature changes predictably when the input changes. Once we know the transformation \(g\), we know exactly how the feature response should change. We do not need to learn a separate filter from examples. Here, \(f\) is the input signal or feature map, \(g\) is a transformation such as a rotation or reflection, \(\psi \in \Psi\) is the learnable filter, \(\rho(g)\) is a representation on how the input transforms, and \(\pi(g)\) is a representation on how the feature transforms.

G-CNNs: stacks of transformations

The basic transformation law is:

It says the value that ends up at position \(x\) after a transformation \(g\) comes from the old position \(g^{-1}x\).

For translations, standard convolution is already equivariant. If you shift the image, the feature map shifts in the same way. This is one of the big reasons CNNs work so well. But standard CNNs don’t naturally know what to do with rotations and reflections. Data augmentation helps, but the architecture itself still treats each pose as something it may need to learn again. G-CNNs fix this by using a group such as \(p4\) or \(p4m\). They explicitly track position and orientation, so a rotated input produces a rotated feature map with its orientation channels permuted in a predictable way.

Instead of learning only one filter, G-CNN applies all transformed versions of the filter:

The resulting filter tensor contain an additional group dimension:

where \(H\) and \(W\) are the spatial size of the kernel, \(C_{in}\) and \(C_{out}\) are the number of input and output channels and \(\|G\|\) is the number of transformations in the group \(G\). For example, a rotation-equivariant CNN with 8 discrete orientations stores \(C \times 8\) feature maps corresponding to the different orientations. Although it’s expressive, it makes every feature pay the cost of carrying a full orientation stack.

What does G-CNN learn?

In G-CNN, the network learns only a single base filter \(\psi\). The remaining filters are generated automatically by applying the group transformations:

\(\psi\), as a convolution filter, is a function defined over spatial positions \(\psi:\mathbb{R}^{2}\rightarrow\mathbb{R}\) and \(x\) represents a spatial coordinate inside the filter kernel. \(g^{-1}\) maps the transformed coordinate back to the original coordinate.

For example, for a discrete rotation group \(p4\) \(G=\{0^\circ,90^\circ,180^\circ,270^\circ\}\) the filter bank becomes \(\{\psi_0,\psi_{90},\psi_{180},\psi_{270}\}\). A \(3\times3\) learned filter can be represented as:

where \(a,\ldots,i\) are learned parameters. The group transformation learns only one filter and then deterministically generates transformed versions of the original filter by changing the spatial coordinates. A \(90^\circ\) rotation produces:

The rotated filter is therefore not an additional learned parameter. It is obtained directly from the original filter through the group operation. Therefore, a G-CNN learns only the base filter \(\psi\). The symmetry group automatically generates the set of transformed filters without requiring additional trainable parameters.

From input space to group feature space in G-CNNs

In a standard CNN, the input image is transformed into a feature map:

where \(x\) represents the spatial position, \(X(x)\) is the input value, and \(F(x)\) is a feature vector at that position.

The output feature map tenosr has the form:

where \(C\) is the number of channels and \(H\) and \(W\) are the spatial dimensions.

In a G-CNN, the first layer is different. Instead of producing only spatial features, the network produces features that are indexed by both spatial position and transformation state:

The resulting feature map tensor becomes:

For example, for the discrete rotation group \(p4\)

each spatial location contains a set of orientation-dependent features:

Thus, a feature is not only associated with a spatial location, but also with a transformation state.

The first layer of G-CNN: Lifting convolution

The first layer of a G-CNN is called a lifting convolution. It maps the input image space to a feature space that includes the group dimension. Instead of the standard convolution:

the lifting convolution computes:

where the filters are generated from the base filter through the group action:

The output therefore belongs to a space that combines spatial and transformation dimensions:

The network has learned a base filter \(\psi\), while the transformed filters \(\psi_g\) are generated deterministically by the symmetry operations.

Subsequent layers of G-CNN

After the first layer, the input is no longer a standard feature map. It is a feature field defined over both spatial positions and transformation states: \(F(x,g)\)

Subsequent layers perform group convolutions that combine information from neighboring spatial positions and different transformation states. The group convolution is defined as:

The term \(g^{-1}h\) ensures that the relationship between different orientations is treated consistently under the group transformation.

Towards Steerable CNNs

Although G-CNN is a real improvement over conventional CNNs, it raises an important question: must every feature be stored as a full stack of orientations?

Cohen & Welling (2017) addressed this problem using Steerable CNN. It keeps the promise of G-CNNs: if the input rotates or flips, the internal features change in a known, predictable way. But instead of forcing every feature to carry a full orientation stack, they let each feature transform in whatever way is natural for it. Some stay unchanged, some flip sign, some rotate together as a 2D pair, and some still need the full stack. The word steerable itself is older, coming from signal processing work on steerable filters.

Each spatial position contains a geometrically structured object (fiber)

At each spatial position \(x\), a feature map stores a structured object called the fiber. In a standard CNN, the channels inside one fiber are just a plain list of numbers. In a steerable CNN, that list is a structured object with a rule for how it transforms when the input rotates or flips. At its general concept, a fiber can hold, for example:

• a scalar that stays the same under every rotation and reflection,
• a signed value that flips sign under specific rotations or reflections,
• a 2D vector whose two components rotate together,
• an even orientation stack, whose channels permute among several poses (this is exactly what a G-CNN fiber looks like).

So a fiber does not only say “what is here!” It also says “how this should behave when the input transforms!”

The rule a steerable filter must satisfy

Let’s assume a convolution filter \(\psi \in \Psi\) maps an input feature map \(f \in F\), transforming under \(g \in G\) according to a representation \(\rho(g)\), to an output feature map transforming under g according to a representation \(\pi(g)\). For the output to be steerable, a convolution filter \(\psi\) must satisfy:

Note that \(\pi(g)\) allows us to steer the output fiber.

According to group theory \(\rho(gh) = \rho(g)\rho(h) \quad \forall g,h \in G\):

\(\Psi\) is a filter space and \(F\) is a linear feature space. Here \(G\) is the subgroup fixing the origin (i.e. rotations and reflections only). Translations are handled separately, by sliding the same filter bank across every position, as in an ordinary convolution.

As usual this says rotating and/or flipping the input feature map and then filtering must give the same result as filtering first and then rotating and/or flipping the output.

Interwiners

The filter must be compatible with the way \(g\) transforms the input and output feature maps. This results into the intertwiner equation: \(\psi(\rho(g)f) = \pi(g)\psi(f)\). Here \(\psi\) intertwines the two group actions \(\rho\) and \(\pi\). In other words, the filter must be determined in a way to have specific symmetries so transforming the input results into transforming the output in a controlled fashion. Here is an example on how interwiner equation imposes constraints on the filter:

Let the \(3\times3\) feature map window be

For a \(g=90^\circ\) clockwise rotation,

Now consider a parameterized filter

Its response to \(f\) is obtained by element-wise multiplication followed by summation:

For simplicity, we assume output is a scalar, then \(\pi(g)\) acts trivially: \(\pi(g)y=y\) and the intertwiner equation therefore becomes: \(\psi(\rho(g)f)=\psi(f)\).

For an arbitrary filter, the \(w_i\)’s are independent, and there is no reason for the intertwiner equation to hold. Requiring the interwiner equality for every possible \(f\) imposes constraints on the filter parameters: \(w_1=w_3=w_7=w_9\) and \(w_2=w_4=w_6=w_8\), while \(w_5\) remains free.

Thus, the filter must have the form

so that \(\rho(g)\psi=\psi\), and consequently \(\psi(\rho(g)f)=\psi(f)\).

More generally, when the output transforms non-trivially according to \(\pi(g)\), the constraint becomes \(\psi(\rho(g)f)=\pi(g)\psi(f)\). In this case, the parameters of \(\psi\) are constrained by both \(\rho(g)\) and \(\pi(g)\).

The filter bank \(\Psi\) that satisfies the interwiner equation form a vector space, written \(\mathrm{Hom}_G(\rho,\pi)\) and called the space of intertwiners. Training only searches inside this space, instead of the space of all possible filters.

A G-CNN turns out to be a special case of this rule. If both \(\rho\) and \(\pi\) are the regular representation, where we have one channel for every element of the group and those channels are simply permuted among the group elements (the orientation stack from the previous post), the constraint above reduces exactly to the group convolution used in G-CNNs. Steerable CNNs generalize this framework by allowing feature fields to transform according to more general representations, rather than restricting them to the regular representation.

How do filters have symmetry?

The filter \(\psi\) is \(3\times3\) matrix, containing 9 parameters.

When we rotate the filter clock-wise by \(90^\circ\) (\(r\)), we obtain

In the interwiner equation, rotating the input is equivalent to rotating the filter in the opposite direction. So the left side of the equation can be written: \(\psi(\rho(g)f) = (\rho(g^{-1})\psi)(f)\).

Now let’s define some very basic filters and see how they react to transformations of \(D_4\) group:

• \(e\) : \(0^\circ\) rotation (identity)
• \(r\) : \(90^\circ\) rotation
• \(r^2\) : \(180^\circ\) rotation
• \(r^3\) : \(270^\circ\) rotation
• \(m\) : horizontal reflection
• \(mr\) : diagonal reflection across \(y=x\)
• \(mr^2\) : vertical reflection
• \(mr^3\) : diagonal reflection across \(y=-x\)

Example 1: the center pixel

Consider a filter containing only the center pixel:

After a \(90^\circ\) rotation,

Nothing changes. \(\rho(r)\psi=\psi\) and therefore \(x\rightarrow x\). We call this a 1-dimensional \(A_1\) representation: \(\pi_{A_1}(r)=[1]\) that will be applied to the right side of the interwiner equation. For this specific filter all elements of group \(D_4\) can be represented by \([1]\) because the filter remains unchanged under all elements of this group, therefore:

\(\pi_{A_1}(g)=[1], \qquad \forall g \in D_4\).

Example 2: symmetric corners

Consider the following filter:

A \(90^\circ\) rotation leaves the filter unchanged: \(\rho(r)\psi=\psi\). This is another \(A_1\) representation type. As before, for this specific filter all elements of group \(D_4\) can be represented by \([1]\) because the filter remains unchanged under all elements of this group:

\(\pi_{A_1}(g)=[1], \qquad \forall g \in D_4\).

Example 3: symmetric edges

Consider the following filter:

Again, a \(90^\circ\) rotation leaves the filter unchanged: \(\rho(r)\psi=\psi\). This gives a third \(A_1\) representation type.

The \(3\times3\) filter space contains 3 \(A_1\). These three representation types (or components) can be understood intuitively as: centers, cornders, edges.

Example 4: Transformation-dependent representation

Consider the following filter

After a \(90^\circ\) rotation,

Thus, \(\rho(r)\psi=-\psi\) and \(x\rightarrow -x\). Consequently the convolutional output also changes sign therefore on the right side of the equation we have \(\pi(r)=[-1]\).

Meanwhile after a \(180^\circ\) rotation (\(r^2\)) the filter remains unchanged: \(\rho(r^2)\psi=\psi\) and therefore \(\pi(r^2)=[1]\).

This is another type of 1-dimensional representation, which we call \(B_1\). Depending on the transformation, it results in either the same filter or its negative, so at the right side of the equation we have \(\pi_{B_1}(r)=[-1]\) and \(\pi_{B_1}(r^2)=[1]\).

Example 5: Detection of edge and its orientation

Consider two oriented filters for detecting local gradients in the horizontal and vertical directions.

This filter detects a change from left to right.

Similarly,

This filter detects a change from top to bottom.

Applying these filters to an input feature map produces two feature maps,

Together, they form a 2-dimensional feature,

which can be interpreted as a local gradient vector. Its direction describes the orientation of the local gradient, while the corresponding edge is oriented perpendicular to this gradient.

Under a \(90^\circ\) rotation, the horizontal and vertical directions are exchanged as follows:

Consequently,

This is a 2-dimensional representation type we call \(E\). For these specific filters we need a \(2 \times 2\) matrix to represent this type of transformation at the right side of the interwiner equation:

Apply this representation on the feature map and see the result: \({\mathbf{f}_r}'=\pi_E(r)\mathbf{f'}\).

Irreducible representations

The representations associated with the examples above are irreducible representations (irreps) that cannot be decomposed further. We can construct the input representation \(\rho\) and output representation \(\pi\) as direct sums of irreducible representations. For example, if \(A_1\), \(B_1\), and \(E\) are irreducible types, then:

acting on the input feature space (\(f\)), and the representation

acting on the output feature space (\(\psi(f)\)).

In matrix form, the symbol \(\oplus\) denotes formation of block-diagonal matrix:

For example, consider the output representation \(\pi\) and suppose that the irreducible types \(A_1\) and \(B_1\) are 1-dimensional, while \(E\) is 2-dimensional:

Then the output feature vector \(\mathbf{f'} = \psi(f)\) (the fiber) can be written as

and under a transformation \(g\), it transforms according to \({\mathbf{f}_g}'=\pi(g)\mathbf{f'}\). Each block of the feature vector therefore transforms according to its own irreducible representation.

We can also have several copies of the same irreducible representation. For example,

means that the feature space contains: 3 copies of \(A_1\), 2 copies of \(B_1\), and 4 copies of \(E\). If \(A_1\) and \(B_1\) are one-dimensional while \(E\) is two-dimensional, then the total dimension of the representation is \(\dim(\pi)=3(1)+2(1)+4(2)=13\). More explicitly, the representation matrix has the block-diagonal form

In other words, the notation \(\rho = \bigoplus_i n_i\rho_i\) and \(\pi = \bigoplus_j m_j\pi_j\) means that the feature space is constructed by taking several copies of different irreducible representations and placing their representation matrices along the diagonal. This decomposition of the group representations into irreducible representations allows a steerable CNN to contain different feature types that transform differently under rotations and reflections, while still having a well-defined overall transformation rule.

Object and feature map

Domain \(\Omega\) (e.g. 2D \(\mathbb{Z}^2\) or 3D \(\mathbb{Z}^3\) Euclidean spaces) is an underlying space where we can define an object. An object can be an input image or a feature map generated within the intermediate layers of representation learning. We apply function space \(\mathbb{F}\) on domain \(\Omega\) to define object \(f\) (e.g. a 2D image or a protein 3D structure).

where \(\mathbb{C} \in \mathbb{R}^K\) is the range of values taken by objects defined on \(\Omega\).

Equivariance and symmetry group

A function \(\Phi\) is equivariant to a transformation \(g\) if transforming the input and then applying \(\Phi\) gives the same result as applying \(\Phi\) first and then transforming the output:

Here \(T_g\) and \(T'_g\) are (possibly different) operators that represent the transformation \(g\) acting on the input and output spaces respectively.

You’ve already seen this with regular convolutions: shifting an image and then convolving gives the same result as convolving and then shifting. That’s just translation equivariance. The goal of G-CNNs is to make networks equivariant to a broader group of transformations \(G\).

Why equivariance and not invariance? Invariance means \(T'_g = I\) (the output doesn’t change at all). While that sounds useful, it’s actually too strong for intermediate layers. If your intermediate features are invariant to rotation, you’ve thrown away spatial information about where things are oriented, which makes it impossible to detect relative poses between features. Equivariance preserves structure; invariance destroys it.

A symmetry group is a set of transformations under which the properties of the object remain unchanged. A Group satisfies following properties:

• Has closure and can be composed (applying two transformations gives another transformation from the set)
• Has an identity (the “do nothing” transformation)
• Has inverses (every transformation can be undone)

The simplest example is the set of 2D integer translations, \(\mathbb{Z}^2\). A translation by \((n, m)\) composed with a translation by \((p, q)\) gives \((n+p, m+q)\). The inverse of \((n, m)\) is \((-n, -m)\). This is the group that regular CNNs exploit.

For more information see my previous post on this topic:
Equivariance and Invariance: The building blocks of geometric deep learning

Here we focus on two richer groups:

• The group p4 p4 consists of all combinations of translations and 90-degree rotations in a grid. Every element can be written as a \(3 \times 3\) matrix:

  \[g(r, u, v) = \begin{bmatrix} \cos(r\pi/2) & -\sin(r\pi/2) & u \\ \sin(r\pi/2) & \cos(r\pi/2) & v \\ 0 & 0 & 1 \end{bmatrix}\]

  where \(0 \leq r < 4\) (four rotations: 0°, 90°, 180°, 270°) and \((u, v) \in \mathbb{Z}^2\) are the translation components. Group composition is just matrix multiplication.

• The group p4m p4m extends p4 with mirror reflections. Its elements are parameterized as:

  \[g(m, r, u, v) = \begin{bmatrix} (-1)^m \cos\!\left(\tfrac{r\pi}{2}\right) & -(-1)^m \sin\!\left(\tfrac{r\pi}{2}\right) & u \\ \sin\!\left(\tfrac{r\pi}{2}\right) & \cos\!\left(\tfrac{r\pi}{2}\right) & v \\ 0 & 0 & 1 \end{bmatrix}\]

  where \(m \in \{0, 1\}\) controls the flip. This gives 8 distinct orientations (4 rotations × 2 flip states), making p4m a richer symmetry group.

Functions on domains vs functions on groups

In a regular CNN, a feature map (which is an object) is a function \(f : \mathbb{Z}^2 \to \mathbb{R}^K\) . It operates on a 2D grid of pixels (domain \(\mathbb{Z}^2\)) and at each pixel \((p, q)\) it assigns a \(K\)-dimensional vector of channels. A simple example is a grayscale image itself, which can be interpreted as a signal defined over the discrete grid (domain) \(\mathbb{Z}^2\), where each pixel contains a single intensity value (i.e., one channel). In CNN, each channel is generated by a filter. One channel might detect edges, another might detect textures, and another might detect corners. The only thing indexing the feature is the spatial position, so at each pixel we store features: \((p, q) \rightarrow\) feature vector.

In a G-CNN, feature maps become functions on the group \(G\) itself: \(f : G \to \mathbb{R}^K\). Instead of indexing by position alone, each feature is indexed by a full group element: a (position},rotation) pair for p4, or a (position, rotation, flip) triple for p4m. Therefore, for each transformation (like a rotation) AND position, we store features: (position, rotation) \(\rightarrow\) feature vector. Instead of just saying “there is an edge at (10, 5),” a G-CNN represents it as “there is an edge at (10, 5) with specific orientations (e.g., 30°, 90°, etc.),” explicitly encoding both position and transformation.

Group actions

When a transformation \(g \in G\) acts on a feature map \(f\), it does so by “pulling back” the coordinates of domain \(\mathbb{Z}^2\). Instead of moving each feature forward to a new location, we ask: “to know what the transformed feature map looks like at position \(x \in \mathbb{Z}^2\), which position in the original map should I look up?” That answer is \(g^{-1}x\):

\(L_g\) is called the left action or left regular representation. It’s the operator that applies a group transformation \(g\) to a feature map by reindexing its domain coordinates.

The reason \(g^{-1}\) appears rather than \(g\) is a consistency requirement: applying transformation \(g\) and then \(h\) must equal applying \(gh\) in one shot. Using \(g^{-1}\) guarantees this — it makes \(L_g\) a proper group homomorphism (a structure-preserving map):

To visualize a p4 feature map, imagine four copies of a 2D spatial grid arranged in a circle, one per rotation. When you rotate such a feature map, each copy shifts to the next position and its contents rotate by 90°. This interplay between the “rotation index” and the spatial content is exactly what the group structure captures.

Standard CNNs fail at rotation equivariance

In a regular CNN, the correlation of a feature map \(f\) with the convolutional filter \(\psi\) is:

where \(y\) ranges over all pixel positions, \(k\) ranges over the \(K\) input channels, \(f_k(y)\) is the feature map value at pixel \(y\) in channel \(k\), and \(\psi_k(y - x)\) is the weight of filter \(\psi\) at the relative offset \((y - x)\) in channel \(k\). The output at location \(x\) is a weighted sum of the neighborhood around \(x\). As you know this is the standard convolution.

For translations, this operation is equivariant: shift the input by \(t\) pixels, and the output shifts by exactly \(t\) pixels:

But for a rotation \(r\), things break down. Working through the same algebra with a rotation gives:

Notice the extra \(L_{r^{-1}} \psi\) on the right. This says: rotating the input and then convolving with \(\psi\) is equivalent to convolving with the already-rotated filter \(L_{r^{-1}}\psi\) and then rotating the output. The filter on the right-hand side has changed. It’s now the \(r\) degree-rotated version of the original $\psi$. So convolving with the same unmodified \(\psi\) on both sides gives different results. The operation is not equivariant to rotation.

This is why a standard CNN that wants to detect a feature in all four orientations (in case of p4 group) needs to learn four separate filters. G-CNNs solve this by sharing weights across all orientations.

G-Convolutions

The G-CNN replaces the standard convolution with a G-correlation, which explicitly correlates over all transformations in \(G\).

First layer

In the first layer, both the input \(f\) and the filter \(\psi\) are ordinary functions on \(\mathbb{Z}^2\). The G-correlation produces a feature map that is a function on \(G\):

The output is indexed by elements \(g \in G\). For each \(g\), the value \([f \star \psi](g)\) is obtained by applying the transformation \(g^{-1}\) to the filter and computing its inner product with the input. As a result, the feature map is a function defined on the group \(G\), rather than on \(\mathbb{Z}^2\).

Subsequent layers

After the first layer, each feature map is no longer just a grid over \(\mathbb{Z}^2\), but a function on the group \(G\).

Instead of assigning a value to each spatial location, it assigns a value to each transformation \(g \in G\) (for example, a combination of a translation and a rotation). In typical settings, \(G\) may be a product such as \(G = \mathbb{Z}^2 \rtimes C_n\). In this case, an element \(g\) can be written as \(g = (x, r)\), where \(x \in \mathbb{Z}^2\) encodes a translation and \(r \in C_n\) encodes a discrete rotation. A feature map \(f\) therefore assigns a vector \(f(g)\) to each transformation \(g\), rather than to each position alone.

Because of this change in representation, filters must be defined in the same space. A filter \(\psi\) is therefore also a function on \(G\), so that it can be compared meaningfully with the input feature map at every group element.

Intuitively, both the input and the filter now describe patterns indexed by transformations rather than just positions.

For example, if \(G = \mathbb{Z}^2\), a feature map is a standard image-like grid. If \(G = \mathbb{Z}^2 \times C_4\), where \(C_4\) represents four rotations, then a feature map assigns a value to each location \((x,y)\) and each orientation \(r \in \{0^\circ, 90^\circ, 180^\circ, 270^\circ\}\). In this case, the filter must also specify a response for each position and orientation, so that it can detect not only where a pattern appears, but also in which orientation it appears.

The generalized correlation operator is then defined by comparing the input with transformed versions of the filter over all elements \(g \in G\):

This is similar to a standard convolution, except the “spatial domain” is now the group \(G\) itself.

Proving equivariance

The equivariance proof follows the same substitution trick as we saw before, now using \(h' = u^{-1}h\) (left multiplication by \(u\) is a bijection on G):

So the G-correlation commutes with \(L_u\) for any \(u \in G\). The network is fully equivariant.

Everything else is equivariant too

Equivariance isn’t just a property of the G-convolution layer. It propagates cleanly through all the other components of a modern network.

Pointwise nonlinearities

Let \(\nu : \mathbb{R} \to \mathbb{R}\) be a nonlinearity (e.g. ReLU). The operator $C_\nu$ applies $\nu$ pointwise:

It changes values independently at each \(g \in G\) without mixing locations.

Group transformations act by shifting the input function: \((L_h f)(g) = f(h^{-1}g)\). A key property is that these two operations commute: \(C_\nu L_h f(g) = \nu(f(h^{-1}g)) = L_h C_\nu f(g)\). This means applying a group transformation and then a nonlinearity gives the same result as applying the nonlinearity first and then transforming. Therefore, pointwise nonlinearities preserve equivariance. Here is the proof:

Since both expressions are equal for all \(g \in G\), we conclude: \(C_\nu L_h f = L_h C_\nu f\). ReLU, sigmoid, tanh: they’re all equivariant for free.

Pooling

We define max-pooling over a neighborhood \(U \subset G\) as:

We show that pooling commutes with the group action.

Now substitute \(u = hu'\), which preserves the set structure:

Therefore, \(P L_h f = L_h P f\)

So pooling is equivariant.

If pooling is taken over the entire group (e.g. all four rotations of p4) at each spatial location then the output will be a rotation-invariant feature map which is useful at the final layer if you want a fully invariant prediction. However, doing this too early (at intermediate layers) hurts performance, because it discards pose information before the network has had a chance to use it.

Batch normalization and residual connections

Batch normalization and residual connections are commonly used in CNNs. \textbf{Question:} Do you think they are also equivariant? Why?

Putting it all together

G-CNN isn’t that different from a standard network in structure: you simply replace each convolution with a G-convolution, adjust the number of channels to keep parameters in check, and let the network carry orientation-aware feature maps through its layers, optionally pooling over them at the end for invariance. But this small change leads to a much bigger shift in perspective: instead of treating features as unstructured vectors, the network now respects the symmetries of the problem by design. As a result, it doesn’t have to relearn the same pattern in multiple orientations, making it more sample-efficient, more expressive with the same number of parameters, and more reliable in how it generalizes.

Representation learning powers deep learning. Instead of relying on hand-crafted features, deep learning models learn useful ones directly from data. A good representation learning should be meaningful, invariant to irrelevant changes, and disentangled into fundamental factors.

Convolutional Neural Networks (CNN) are a great example of representation learning. Instead of learning each part of an image separately, CNN reuse the same learned filters across all spatial locations, sharing weights throughout the image. This works because a cat is still a cat wherever it appears. A cat in the top-left of an image is the same cat in the bottom-right. This is translation equivariance by design, and it’s one of the core reasons CNNs work so well. Shifting the input image by a few pixels shifts the convolution output by the same amount! The network doesn’t have to re-learn anything.

However, conventional CNNs struggle with other geometric transformations, like rotations and reflections, often relying on brute-force data augmentation to learn them. When you rotate a handwritten “6” by 180°, it looks like a “9” to both humans and machines. A classical CNN has no built-in mechanism to link these two images. It must learn each orientation independently, which means:

• Wasting model capacity on redundant rotated copies of every filter
• Requiring heavy data augmentation (rotating training images) just to generalise
• Still degrading significantly on unseen rotation angles

The figure below shows the same digit at 8 different orientations (rotation angles 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°). The pixel pattern changes completely at each angle, yet the semantic content (the identity of the digit) is identical. A classical CNN trained on upright digits must re-learn each orientation from scratch. Given this limitation, any model that bakes rotation invariance into its architecture rather than learning it from data has a fundamental advantage.

Beyond the conventional CNN

Over the past decade, researchers have been trying to find a way to make neural networks naturally understand things like rotations and symmetry, instead of forcing them to learn these patterns from large amounts of extra data. In other words, they aim to mathematically “hard-bake” geometric symmetries directly into neural network architectures. I will explore methods in representation learning that embed built-in structural mechanisms to help models learn relationships between rotated versions of images. The works that I present here are only a small sample of a much larger body of research and are by no means exhaustive.

• Chapter 0: When machine learns not to panic every time something rotates

• Chapter 1: How to learn representations under p4 and p4m symmetry groups

Disentanglement and invariance

Think of disentangling as taking a complex, blended fruit smoothie and teaching a machine to mathematically separate it back out into pure, individual piles of strawberries, bananas, and yogurt. In computer science, disentanglement means breaking down complex data (e.g. an image) into its individual ingredients (factors of variation), like separating an image into distinct elements such as object’s identity, shape, location, orientation, scale or even lighting condition. Once these factors are neatly separated, achieving invariance, which is the ability to ignore irrelevant changes (e.g. rotation of an object), becomes simple. If you want to identify an object regardless of where it is, you just focus on the identity factor and ignore the position factor, allowing the system to stay consistent even when unimportant details change.

Weyl’s principle

Physicists use this principle to figure out what the true, fundamental elementary particles of a system are, separating the real particles from the raw, messy numbers a measuring device might produce which has no physical meaning. This principle is based on the idea of symmetry. Even we change how we measure something (altering its surface appearance or measured values), the core reality of what we are observing stays exactly the same. Cohen and Welling realized that the pixels in a digital image are just like those raw physical measurements, and we need a way to find the “elementary particles” of the image. Although this principle is primarily used in physics, the concept is fully abstract and independent of the data type (such as images, optical flow, or audio), which makes it highly valuable for representation learning. More broadly, it can be applied to any context where a clear notion of symmetry exists.

Irreducibility and Lie group theory

Lie group is a mathematical framework used to describe continuous, smooth transformations. Think of the difference between a digital clock that jumps rigidly from 1:00 to 1:01, versus an old analog clock where the second hand sweeps in a perfectly smooth, continuous circle. Conventional neural networks only understood the rigid, jumping steps of a pixel grid. Lie group theory, however, provided the mathematical language to describe a smooth, unbroken sweeping motion—like rotating an image by any microscopic fraction of a degree, rather than just flipping it 90 degrees. In this context, irreducibility means breaking down these complex, continuous transformations into their absolute most fundamental, independent building blocks. Once a transformation is irreducible, it cannot be divided any further into smaller invariant parts, making it the purest elementary component of the system’s symmetry.

Applying these three ideas to computer vision, they built a smart probabilistic model called Toroidal Subgroup Analysis (TSA). Instead of relying on the brute-force method of showing a neural network thousands of manually rotated images, TSA learns the underlying mathematical rules of the transformation itself. The model was trained simply by looking at pairs of images: an original image and a transformed version of it. By observing how the image changes from the first to the second frame, the model learns to mathematically disentangle the complex, jumbled web of pixels into simple, independent pieces called irreducible representations.

Figure below shows model’s posterior distribution over a rotation angle (denoted as \(s\)) for three distinct pairs of images. The top row shows the initial and rotated images, while the corresponding blue graphs below plot the model’s calculated probability across all possible rotation angles from \(0\) to \(2\pi\). For the first pair (a plus sign rotated into an ‘X’), the distribution displays four distinct peaks due to the shape’s four-fold rotational symmetry, indicating four equally probable rotation angles. The second pair features a circular ‘O’ shape, which results in a completely flat, uniform distribution because its continuous symmetry makes every rotation angle equally valid. The third pair involves a ‘T’ shape lacking rotational symmetry, yielding a graph with a single prominent peak that represents high model certainty for a specific angle.

Souce: Cohen, T., & Welling, M. (2014). Learning the irreducible representations of commutative lie groups. In International Conference on Machine Learning

This work laid the foundation for building neural networks that can naturally handle continuous changes, such as cyclic rotation.

Continue to Chapter 1: How to learn representations under p4 and p4m symmetry groups

Proteins

Proteins are fundamental to virtually all biological processes. They fold into intricate 3D shapes, interact with other proteins or molecules, and are behind nearly every cellular machinery. The ability to understand protein sequence, structure, dynamics, interactions, and functions at the molecular level has tremendous implications for therapeutic development, drug discovery, precision medicine, and the rational design of biomolecular interventions.

Proteins exhibit complex sequence context and intricate structural organization which govern their thermodynamic stability and functional specificity. Individual proteins may adopt multiple conformational states and carry out diverse functions depending on their structural state and cellular environment.

By analyzing protein sequences and structures we can answer multiple open questions in the protein science, including:

• The stability of the protein
• The functional capacity of the protein
• The impact of mutations on protein stability and activity
• The quality of the modeled protein structure, whether obtained experimentally or predicted using computational tools and enable:
• The rational design of proteins that are geometrically and physico-chemically compatible with a desired function or target

These analyses have a wide range of applications in explaining how proteins regulate cellular mechanisms and in enabling the rational design of selective and effective therapeutic agents. Therefore, a comprehensive understanding of the molecular mechanisms behind proteins is essential.

For a machine to reason about proteins, whether predicting their structure, function, or interactions, the first step is deciding how to represent them. Protein representation learning focuses on creating dense, rich and low-dimensional representations in a way that they reflect real biological meaning while still being practical and machine-learning-friendly for computational models. Here I give a short, practical overview of main ways proteins are represented and how data-driven and deep learning-based methods represent proteins as a compact vectors (i.e. embeddings).

Why representation matters

At the same time, a protein is:

• A sequence of amino acids
• A folded 3D object with physical constraints
• A dynamic system shaped by evolution
• An interconnected network of atoms linked by physical interactions

Any representation highlights some of these aspects and ignores others. The choice of representation often determines what a model can (and cannot) learn. A protein (or a protein complex) can be represented in different ways:

• Sequence
• Multiple sequence alignment
• Volumetric map
• Gaussian surface
• Molecular surface
• 3D molecular graph
• Point clouds
• Residue interaction network

Sequence-based representations

The most basic representation is the amino acid sequence itself.

Early protein sequence representations relied on simple, interpretable methods such as one-hot encoding of individual amino acids, but these approaches scale poorly and fail to model long-range dependencies.

Inspired by advances in natural language processing, modern methods instead learn dense sequence embeddings by treating proteins like language (language of life) using models trained on massive corpora of millions of sequences with self-supervised objectives like masked token prediction.

These learned embeddings capture rich evolutionary, structural, and functional signals directly from sequence data.

See my previous post on this topic:
Protein Language Models

Here is a hands-on session on protein sequence representation learning

Evolutionary representations

Evolution leaves strong signals in protein families, and multiple sequence alignments (MSAs) make these signals explicit by revealing which residues are conserved or co-evolved across related sequences. Conserved positions often indicate functional or structural importance, while correlated mutations can suggest interacting residues that are spatially close in the folded protein or at the interface of protein-protein interaction. As a result, many structure prediction methods, including early versions of AlphaFold, have relied heavily on MSAs to extract rich evolutionary information, although this comes at the cost of expensive computation and limited applicability to orphan proteins or adaptive immune receptors such as antibodies, nanobodies, and T cell receptors (TCRs) that lack sufficient homologs.

Structure-based representations

When experimentally determined or computationally predicted 3D structures are available, proteins can be represented with a richer and more intuitive way, a geometric way. Instead of just looking at sequences, we can represent proteins using atomic- or residue-level coordinates, which capture how the protein is actually arranged in the 3D space.

3 dimensional convolutional neural networks (3D-CNN) can represent an intricate arangement of atoms in a space as a compact vector useful for classification or regression tasks.

3D volumetric maps

These coordinate-based representations are often either inherently invariant representations or processed with equivariant neural networks. An example of inherently invariant representation is 3D volumetric maps that are centered and oriented based on the common scaffold of the backbone of amino acids.

Here is a hands-on session on how to learn 3D volumetric maps of protein structures

On the other hand equvariant neural networks naturally respect rotational and translational symmetry. It means the protein’s properties don’t change just because we rotate or shift it in space.

See my previous posts on this topic:
Geometric Deep Learning and Equivariance and Invariance

Graph representation learning

Another common approach is to model the structure as a graph or point cloud, where nodes represent residues or atoms and edges are spatial proximity, covalent or molecular bonds, or other types of interaction.

Graph neural networks (GNNs) are particularly well suited to this setting and have become increasingly popular for structural modeling. Using different variants of the message-passing algorithm, GNNs iteratively update node and edge features, as well as spatial coordinates in the case of SE(3)-Transformers equivariant architecture. At each iteration, nodes aggregate information from their neighbors, progressively incorporating information from increasingly distant nodes. After several message-passing steps, each node representation captures both its local neighborhood and the broader topology and features of the graph. These node representations can then be pooled into a single compact vector that summarizes the entire graph.

Here is a hands-on session on graph representation learning

The big advantage of these geometric approaches is that they explicitly encode physical and spatial information. However, obtaining reliable 3D structures can be expensive and time-consuming, and even predicted structures may be noisy or incomplete.

Conclusion

Protein representation learning is at the intersection of biology, physics, and machine learning, aiming to turn the complexity of proteins into forms that computer models can actually understand and use. One of its biggest strengths is transferability: once a model has learned a meaningful representation, it can be reused across many downstream tasks with little additional supervision, saving both time and data. That said, there is no single “best” representation! There are only ones that fit particular questions and constraints. For instance, if we study highly flexible regions of a protein, a static, structure-based representation might miss important biophysical realities, whereas a sequence-based approach, implicitly capturing aspects of dynamics and function, or an explicitly dynamics-based representation could be more appropriate. We have only begun to scratch the surface; as models become more multimodal and grounded in biophysical principles, representations are shifting from hand-crafted features to deeply learned ones, bringing us closer to truly capturing the rich and nuanced complexity of proteins.

What are Protein Language Models?

Just like human languages, nature has its own unique language—the language of life! This language dictates the cellular mechanisms within all living organisms. By understanding it, we can gain insights into biological processes and answer essential questions, such as: What are the 3D structures of proteins? How do proteins interact? and What are the impacts of mutations on proteins and their interactions?

So how can we understand the language of life? luckily the advances in deep learning algorithms (e.g. transformers and masked language modeling) and the advent of large language models (LLM) have paved the way. Thanks to these algorithms, we now have a class of pre-trained models, known as protein language models (pLM), that help us analyze protein sequences.

Let’s briefly introduce LLMs, which share the same underlying architectures to understand better how pLMs work.

Large Language Models

Large Language Models (LLMs) have revolutionized the field of natural language processing (NLP) with their remarkable ability to understand and generate human languages. Long story short, the most recent breakthrough came with the introduction of transformer architecture [1]. This architecture led to the development of well-known models such as the BERT [2] and the T5 [3] models. These pre-trained models use extensive data and computational resources to learn and extract informative representations from language essential for many downstream tasks in NLP.

These architectures, combined with the ever-growing use of GPUs and the scaling up of data, have led to highly efficient models for text generation. This process involves predicting each word (token) based on the probability distribution derived from the context of preceding words.

The Transformer architecture [1]

Protein Language Models

Protein Language Models (pLMs) apply the same principles from NLP to capture the complex dependencies in protein “sentences”. A protein sentence is a sequence of amino acids (i.e. polypeptide) that determine protein properties. pLMs leverage transformer architectures, similar to those used in models like BERT, and help us to predict protein structures and functions. In addition, pLMs have opened up exciting possibilities in protein engineering and synthetic biology. By enabling high-throughput prediction of variant effects, they help us design de novo proteins with desired traits.

pLMs [4-8] are trained on large datasets of protein sequences, such as those from the UniRef databases. These datasets include millions of protein sequences from various organisms and evolutionary backgrounds. Besides using amino acid sequences, some recent models [8] also include structural information represented as a sequence of structural tokens [9] (a structural “sentence”) to improve their predictive power.

Two breakthroughs in this field that have significantly advanced research in computational biology and drug discovery are the ProtTrans pLMs [5] and the Evolutionary Scale Modeling (ESM) pLMs [6]. These models extract context-aware embeddings for each amino acid necessary to predict various protein properties like function, stability, structure, inverse folding, and variant effects.

pLMs is a recent class of deep learning models designed to analyze protein sequences by leveraging techniques such as mask language modeling. Here is a timeline showing the evolution of this emerging field. pLMs can be applied in multiple protein-related applications, and some articles explore their diverse uses. To simplify the illustration, I highlighted the primary application of each tool. Similarly, these tools are the result of collaborative efforts across various institutions, and I have noted the key host institutions behind them.

The secret behind protein language models

The recipe of pLM requires five ingredients:

• Large datasets of protein sequences (e.g. UniProt)
• Self-supervised learning techniques
• Mask language modeling
• Transformer architecture (or sometimes LSTM!)

and of course

• Computational power (CPU, GPU, and TPU)

The input is protein sequences tokenized into individual amino acids, sometimes accompanied by structural tokens, similar to how words are tokenized in NLP. A portion of the amino acids in the sequences is masked, and using self-supervised techniques the model is trained to predict these masked positions.

During training, the model learns from the sequence without requiring labeled examples (thanks to self-supervised learning), enabling it to capture the underlying patterns and relationships within protein sequences. The primary objective is to minimize the cross-entropy loss (i.e. log of perplexity), which measures how well the model predicts the masked amino acids based on the rest of the sequence. This metric measures how closely the predicted probability distribution of amino acids matches the desired (target) distribution.

\[ \mathcal{L}(\mathbf{Y}, \mathbf{\hat{Y}}, \mathbf{M}) = - \frac{1}{\sum_{i=1}^n m_i} \sum_{i=1}^n m_i \log P(y_i | \mathbf{S}) \]

\(\mathbf{S} = [s_1, s_2, \ldots, s_n]\)

\(\mathbf{Y} = [y_1, y_2, \ldots, y_n]\)

\(\mathbf{\hat{Y}} = [\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_n]\)

\(\mathbf{M} = [m_1, m_2, \ldots, m_n]\)

\(\mathbf{V} = [v_1, v_2, \ldots, v_{20}]\)

\(y_i \in V\)

where S represents the input sequence of amino acids, M is a binary array (0 or 1) that indicates the masked positions within S, V is the vocabulary (20 amino acids), Y denotes the ground truth sequence represented by a probability distribution over vocabulary V for each position in the form of one-hot encoding, and Ŷ is the predicted probability distribution over the vocabulary V for each position. The core of this equation is the following formula

\(- log P(y_i | \mathbf{S})\)

which is the negative log probability of the correct token yi given the input sequence S. This is the cross-entropy. The formula’s simplicity is due to the one-hot encoding of the ground truth Y. Here is why

\( \hat{y}_i = [P(v_1|S), P(v_2|S),\ldots, P(v_{20}|S)] \)

This is the predicted probability vector for position i over V.

\( o_i = (o_{i1}, \ldots, o_{i20}), \quad o_{ij} = \begin{cases} 1, & \text{if } v_j = y_i \\ 0, & \text{otherwise} \end{cases}, \quad o_i = [0, 0, 0, 1, \ldots, 0] \)

This is the representation of yi as one-hot encoding over V at position i. Then

\( CE_i = - \sum_{j=1}^{20} o_{ij} \times log P(v_j | S) = - log P(v_k | \mathbf{S}) = - log P(y_i | \mathbf{S}) \)

Applications

When used for inference, the model functions as an encoder, encoding protein sequences into vector representations (also known as embeddings) that capture their various properties.

pLMs can share their knowledge with other downstream predictive models through a process called transfer learning. This helps boost their performance, particularly for those with a limited amount of labeled data.

Conclusion

pLMs are changing the field of computational biology by providing powerful tools for protein analysis, prediction, and design.

While we are still exploring how deeply pLMs truly grasp or understand the fundamental biophysical properties of proteins, their applications span across multiple domains, making them indispensable in modern biological research. Every year, we see more and more research papers, talks, and posters highlighting how much we rely on pLMs. It’s exciting to see their impact growing!

Stay tuned for our upcoming posts, where we will go deeper into pLMs and uncover the intriguing details behind their impressive performance!

References:

[1] Paper: Attention Is All You Need (Transformer architecture)

[2] Paper: Bidirectional Encoder Representations from Transformers (BERT)

[3] Paper: Text-to-Text Transfer Transformer (T5)

[4] Paper: Protein Sequence Embeddings (ProSE), Code: Github

[5] Paper: ProtTrans, Code: Github

[6] Paper: Evolutionary Scale Modelling (ESM), Code: Github

[7] Paper: ESMFold, Code: Github

[8] Paper: Protein structure-sequence T5 (ProstT5), Code: Github

[9] Paper: FoldSeek, Code: Github

Also here is an excellent blog post on how to use pLMs for protein design and engineering with nice examples.

Geometric deep learning (GDL) addresses two main challenges in classical deep learning: the absence of a global reference frame and the curse of high dimensionality. It leverages low-dimensional geometric priors and an associated symmetry group (𝔾), such as translations, rotations, and permutations, under which certain properties of an object remain invariant.

The properties of a protein structure (object 𝑜) defined in Euclidean space (domain Ω), such as the binding affinity between two interactors, remain invariant under roto-translation transformations from SE(3) (group 𝔾). We use GDL to design neural networks 𝑓 that conserve these properties.

Let’s set the stage with a few key definitions:

Domain Ω (e.g. 3D Euclidean space) is an underlying space where we can define an object. We apply function space 𝕆 on domain Ω to define object 𝑜 (e.g. a protein 3D structure).

\( \mathbb{O}(\Omega, \mathbb{C})=\{o:\Omega \rightarrow \mathbb{C}\} \)

where

\( \mathbb{C} \in \mathbb{R}^d \)

is the range of values taken by objects defined on Ω.

Group 𝔾 is a set equipped with a binary operation ⊗, known as composition, and satisfies the following properties:

• Closure:

\( \forall g_1, g_2 \in \mathbb{G} \rightarrow g_1 \otimes g_2 \in \mathbb{G} \)

• Associativity:

\( \forall g_1, g_2, g_3 \in \mathbb{G} \rightarrow (g_1 \otimes g_2) \otimes g_3 = g_1 \otimes (g_2 \otimes g_3) \)

• Identity:

\( \forall g \in \mathbb{G}, \exists! e \in \mathbb{G} → (g \otimes e) = (e \otimes g) = g \)

• Inverse:

\( \forall g \in \mathbb{G}, \exists! g^{-1} \in \mathbb{G} → g \otimes g^{-1} = g^{-1} \otimes g = e \)

Each transformation g ∈𝔾 has a representation 𝑝⁡(g). For example, if g is a translation in Euclidean space then,

\( p(g) \in \mathbb{R}^3 \)

is a translation matrix.

If domain Ω is symmetrical under group 𝔾 then it ensures that the properties of the object 𝑜 remain unchanged despite transformations defined within 𝔾. The Euclidean space is endowed with roto-translation symmetries and the group that contains the roto-translation transformations is called the special Euclidean group or 𝑆⁢𝐸⁡(𝑛), where n is the number of dimensions. This gives us inductive bias which eventually leads to the definition of invariant and equivariant functions, the building blocks of GDL.

Equivariant block

The function 𝑓1 (e.g. a neural network) is equivariant under the transformation g if applying g to the input object 𝑜 results in the same transformation being applied to the output. GDL employs patch-wise symmetry groups to achieve this and ensures that the entire architecture remains locally equivariant by stacking multiple locally equivariant layers.

\( f_1: \mathbb{O}(\Omega) \rightarrow \mathbb{O}(\Omega) \)

\( f_1(p(g)o) = p(g)f_1(o), g \in \mathbb{G} \)

where 𝑓1 is a function defined on the object to extract its properties and is equivariant to the group 𝔾.

Invariant block

We can formulate invariant function 𝑓2 (again a neural network) that benefits from the geometric priors of domain Ω:

\( f_2: \mathbb{O}(\Omega) \rightarrow \mathbb{Y} \)

\( f_2(p(g)o) = f_2(o), g \in \mathbb{G} \)

where 𝑓2 is a function defined on the object to extract its properties and is invariant to the group 𝔾.

In most problems, the final layer is designed to make the whole process globally invariant. The integration of equivariant and invariant blocks minimizes the number of trainable parameters by utilizing kernels with shared weights (a simple example is convolutional neural networks) and removes the necessity for data augmentation. 

Pipeline

Now let’s see how we can put together a complete GDL pipeline!

If domain Ω’ is a compact (coarse-grained) version of  domain Ω (Ω′ ⊆ Ω), then we can define the building blocks of the GDL approaches:

Linear 𝔾-equivariant layer

\( f_1: \mathbb{O}(\Omega, \mathbb{C}) \rightarrow \mathbb{O}(\Omega', \mathbb{C'}) \)

\( \forall g \in \mathbb{G}, \forall o \in \mathbb{O}(\Omega, \mathbb{C}) \rightarrow f_1(p(g)o) = p(g)f_1(o) \)

𝔾-invariant layer (global pooling)

\( f_2: \mathbb{O}(\Omega, \mathbb{C}) \rightarrow \mathbb{Y} \)

\( \forall g \in \mathbb{G}, \forall o \in \mathbb{O}(\Omega, \mathbb{C}) \rightarrow f_2(p(g)o) = f_2(o) \)

Local pooling (coarsening)

\( p: \mathbb{O}(\Omega, \mathbb{C}) \rightarrow \mathbb{O}(\Omega', \mathbb{C}) \)

Nonlinearity

\( \sigma: \mathbb{O}(\Omega, \mathbb{C}) \rightarrow \mathbb{O}(\Omega, \mathbb{C'}) \)

Now we simply concatenate these blocks to create 𝔾-invariant function:

\( \mathbb{F}: \mathbb{O}(\Omega, \mathbb{C}) \rightarrow \mathbb{Y} \)

\( \mathbb{F} = f_2 \odot \sigma_n \odot f_{1,n} \odot p_{n-1} \odot ... \odot p_1 \odot \sigma_1 \odot f_{1,1} \)

Two standout GDL frameworks that incorporate equivariant and invariant blocks are Graph Neural Networks (GNNs) and Group Equivariant Convolutional Networks (G-CNN). These are widely used in computational and structural biology for many tasks, including quality assessment of protein structure models, protein structure prediction, protein interface prediction, and protein design.

I highly recommend the resources below. Back in 2021, I enjoyed reading [1]! It was so well-explained and easy to follow - I learned a lot from it.

1. Bronstein et al., 2021: “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges”

2. Geometric Deep Learning

Here are also some amazing resources from Amsterdam Machine Learning Lab (AMLab) and the University of Amsterdam:

• Hands-on notebooks: GDL - Regular Group Convolutions and Group Equivariant Neural Networks.
• Lectures on Group Equivariant Deep Learning

This blog is also nice:

Geometric Deep Learning: Group Equivariant Convolutional Networks

Often, during events or conferences, when I explain our work I have to mention Geometric Deep Learning (GDL) as a cornerstone of our research. But this usually leads to questions like, “What is Geometric Deep Learning?” or comments like, “This is the first time I’m hearing this term!” Despite being an important and well-established field, I have noticed that it remains relatively obscure, particularly outside specialized areas like structural biology, computer vision, and physics. Understanding this concept is also crucial for grasping emerging technologies, such as de novo protein design. So I thought I’d shine some light and give a simple definition of GDL. I believe this will lay a good foundation for exploring topics like the application of generative AI in therapeutic solutions.

Let’s begin with a simple question:

How can we build a model that learns and analyzes the structure of a protein, a highly complex 3D object in the Cartesian system?

We can use traditional deep learning algorithms but then we run into two major challenges:

1. The lack of a global frame of reference. Traditional deep learning algorithms are sensitive to transformations in input data (e.g., image rotation) and might perceive transformed data as a new, unseen sample. In the context of proteins, features like biological function or binding affinity are irrelevant to global translations and orientations of the protein structure.

2. The complexity of the 3D structures leading to the curse of high dimensionality.

GDL addresses these challenges by using low-dimensional geometric priors and associated symmetry groups, ensuring invariance to transformations like rotations and permutations while maintaining the properties of protein structures intact.

But GDL isn’t just about 3D objects! It extends deep learning techniques to non-Euclidean domains like graphs and manifolds. For instance, proteins can be represented as graphs, where amino acids are nodes connected by edges. They can also be represented as meshes or point clouds. Out of all the different ways to represent/learn geometric data, graph representation learning is my favorite. It is an exciting category of architectures known as Graph Neural Networks (GNNs). These networks learn intricate, high-dimensional representations of graphs using propagation (message passing) between nodes via edges. GNNs deserve their own post!

We can represent a protein structure, or any (bio-)molecule, as a graph. Nodes can be amino acids and edges can be covalent or molecular interactions. Nodes communicate with each other and update their feature vectors through a process known as message passing. This representation allows us to leverage graph learning architectures and learn the properties of the structure.

Here are some awesome visualizations that highlight the importance of equivariance in GDL. It’s a core feature where the output of an equivariant layer undergoes the same predictable transformation as its input.

Source: https://github.com/QUVA-Lab/e2cnn

In the world of protein structures, thanks to the local equivariance property, not only does GDL find structural motifs regardless of their position and orientation, but it also considers the relative orientation and position of these motifs which is necessary for the right information aggregation.

Source: https://onlinelibrary.wiley.com/doi/abs/10.1002/prot.26235

Also, check out this video from [3].

Many researchers from different groups contributed to this field. Thanks to their mathematically rigorous work, we now have a solid foundation for GDL, with numerous applications across different domains. Here are a few of these references:

1. Bronstein et al., 2021: “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges”

2. Weiler & Cesa, 2019: “General e (2)-equivariant steerable CNNs”

3. Weiler et al., 2018: “3D steerable CNNs: Learning rotationally equivariant features in volumetric data”

4. Fuchs et al., 2020: “Se (3)-transformers: 3D roto-translation equivariant attention networks”

5. Chami et al., 2022: “Machine learning on graphs: A model and comprehensive taxonomy”

In future posts, I’ll go deeper into the details and share educational resources where you can learn more about GDL.