Steerable CNNs & irreducible representations

How group convolutional networks exploit symmetry ended with a question: must every feature map be stored as a full stack of orientations? A \(p_4\) G-CNN keeps four rotated copies of every feature, and a \(p_4m\) G-CNN keeps eight. That is wasteful when a feature doesn’t need to remember every orientation separately. Those features look the same no matter how the input is rotated, for example:

Cohen & Welling (2017) addressed this problem using Steerable CNN. It keeps the promise of G-CNNs: if the input rotates or flips, the internal features change in a known, predictable way. But instead of forcing every feature to carry a full orientation stack, they let each feature transform in whatever way is natural for it. Some stay unchanged, some flip sign, some rotate together as a 2D pair, and some still need the full stack. The word steerable itself is older, coming from signal processing work on steerable filters.

Each spatial position contains a geometrically structured object (fiber)

At each spatial position \(x\), a feature map stores a structured object called the fiber. In a standard CNN, the channels inside one fiber are just a plain list of numbers. In a steerable CNN, that list is a structured object with a rule for how it transforms when the input rotates or flips. At its general concept, a fiber can hold, for example:

• a scalar that stays the same under every rotation and reflection,
• a signed value that flips sign under specific rotations or reflections,
• a 2D vector whose two components rotate together,
• an even orientation stack, whose channels permute among several poses (this is exactly what a G-CNN fiber looks like).

So a fiber does not only say “what is here!” It also says “how this should behave when the input transforms!”

The rule a steerable filter must satisfy

Let’s assume a convolution filter \(\psi \in \Psi\) maps an input feature map \(f \in F\), transforming under \(g \in G\) according to a representation \(\rho(g)\), to an output feature map transforming under g according to a representation \(\pi(g)\). For the output to be steerable, a convolution filter \(\psi\) must satisfy:

Note that \(\pi(g)\) allows us to steer the output fiber.

According to group theory \(\rho(gh) = \rho(g)\rho(h) \quad \forall g,h \in G\):

\(\Psi\) is a filter space and \(F\) is a linear feature space. Here \(G\) is the subgroup fixing the origin (i.e. rotations and reflections only). Translations are handled separately, by sliding the same filter bank across every position, as in an ordinary convolution.

As usual this says rotating and/or flipping the input feature map and then filtering must give the same result as filtering first and then rotating and/or flipping the output.

Interwiners

The filter must be compatible with the way \(g\) transforms the input and output feature maps. This results into the intertwiner equation: \(\psi(\rho(g)f) = \pi(g)\psi(f)\). Here \(\psi\) intertwines the two group actions \(\rho\) and \(\pi\). In other words, the filter must be determined in a way to have specific symmetries so transforming the input results into transforming the output in a controlled fashion. Here is an example on how interwiner equation imposes constraints on the filter:

Let the \(3\times3\) feature map window be

For a \(g=90^\circ\) clockwise rotation,

Now consider a parameterized filter

Its response to \(f\) is obtained by element-wise multiplication followed by summation:

For simplicity, we assume output is a scalar, then \(\pi(g)\) acts trivially: \(\pi(g)y=y\) and the intertwiner equation therefore becomes: \(\psi(\rho(g)f)=\psi(f)\).

For an arbitrary filter, the \(w_i\)’s are independent, and there is no reason for the intertwiner equation to hold. Requiring the interwiner equality for every possible \(f\) imposes constraints on the filter parameters: \(w_1=w_3=w_7=w_9\) and \(w_2=w_4=w_6=w_8\), while \(w_5\) remains free.

Thus, the filter must have the form

so that \(\rho(g)\psi=\psi\), and consequently \(\psi(\rho(g)f)=\psi(f)\).

More generally, when the output transforms non-trivially according to \(\pi(g)\), the constraint becomes \(\psi(\rho(g)f)=\pi(g)\psi(f)\). In this case, the parameters of \(\psi\) are constrained by both \(\rho(g)\) and \(\pi(g)\).

The filter bank \(\Psi\) that satisfies the interwiner equation form a vector space, written \(\mathrm{Hom}_G(\rho,\pi)\) and called the space of intertwiners. Training only searches inside this space, instead of the space of all possible filters.

A G-CNN turns out to be a special case of this rule. If both \(\rho\) and \(\pi\) are the regular representation, where we have one channel for every element of the group and those channels are simply permuted among the group elements (the orientation stack from the previous post), the constraint above reduces exactly to the group convolution used in G-CNNs. Steerable CNNs generalize this framework by allowing feature fields to transform according to more general representations, rather than restricting them to the regular representation.

How do filters have symmetry?

The filter \(\psi\) is \(3\times3\) matrix, containing 9 parameters.

When we rotate the filter clock-wise by \(90^\circ\) (\(r\)), we obtain

In the interwiner equation, rotating the input is equivalent to rotating the filter in the opposite direction. So the left side of the equation can be written: \(\psi(\rho(g)f) = (\rho(g^{-1})\psi)(f)\).

Now let’s define some very basic filters and see how they react to transformations of \(D_4\) group:

• \(e\) : \(0^\circ\) rotation (identity)
• \(r\) : \(90^\circ\) rotation
• \(r^2\) : \(180^\circ\) rotation
• \(r^3\) : \(270^\circ\) rotation
• \(m\) : horizontal reflection
• \(mr\) : diagonal reflection across \(y=x\)
• \(mr^2\) : vertical reflection
• \(mr^3\) : diagonal reflection across \(y=-x\)

Example 1: the center pixel

Consider a filter containing only the center pixel:

After a \(90^\circ\) rotation,

Nothing changes. \(\rho(r)\psi=\psi\) and therefore \(x\rightarrow x\). We call this a 1-dimensional \(A_1\) representation: \(\pi_{A_1}(r)=[1]\) that will be applied to the right side of the interwiner equation. For this specific filter all elements of group \(D_4\) can be represented by \([1]\) because the filter remains unchanged under all elements of this group, therefore:

\(\pi_{A_1}(g)=[1], \qquad \forall g \in D_4\).

Example 2: symmetric corners

Consider the following filter:

A \(90^\circ\) rotation leaves the filter unchanged: \(\rho(r)\psi=\psi\). This is another \(A_1\) representation type. As before, for this specific filter all elements of group \(D_4\) can be represented by \([1]\) because the filter remains unchanged under all elements of this group:

\(\pi_{A_1}(g)=[1], \qquad \forall g \in D_4\).

Example 3: symmetric edges

Consider the following filter:

Again, a \(90^\circ\) rotation leaves the filter unchanged: \(\rho(r)\psi=\psi\). This gives a third \(A_1\) representation type.

The \(3\times3\) filter space contains 3 \(A_1\). These three representation types (or components) can be understood intuitively as: centers, cornders, edges.

Example 4: Transformation-dependent representation

Consider the following filter

After a \(90^\circ\) rotation,

Thus, \(\rho(r)\psi=-\psi\) and \(x\rightarrow -x\). Consequently the convolutional output also changes sign therefore on the right side of the equation we have \(\pi(r)=[-1]\).

Meanwhile after a \(180^\circ\) rotation (\(r^2\)) the filter remains unchanged: \(\rho(r^2)\psi=\psi\) and therefore \(\pi(r^2)=[1]\).

This is another type of 1-dimensional representation, which we call \(B_1\). Depending on the transformation, it results in either the same filter or its negative, so at the right side of the equation we have \(\pi_{B_1}(r)=[-1]\) and \(\pi_{B_1}(r^2)=[1]\).

Example 5: Detection of edge and its orientation

Consider two oriented filters for detecting local gradients in the horizontal and vertical directions.

This filter detects a change from left to right.

Similarly,

This filter detects a change from top to bottom.

Applying these filters to an input feature map produces two feature maps,

Together, they form a 2-dimensional feature,

which can be interpreted as a local gradient vector. Its direction describes the orientation of the local gradient, while the corresponding edge is oriented perpendicular to this gradient.

Under a \(90^\circ\) rotation, the horizontal and vertical directions are exchanged as follows:

Consequently,

This is a 2-dimensional representation type we call \(E\). For these specific filters we need a \(2 \times 2\) matrix to represent this type of transformation at the right side of the interwiner equation:

Apply this representation on the feature map and see the result: \({\mathbf{f}_r}'=\pi_E(r)\mathbf{f'}\).

Irreducible representations

The representations associated with the examples above are irreducible representations (irreps) that cannot be decomposed further. We can construct the input representation \(\rho\) and output representation \(\pi\) as direct sums of irreducible representations. For example, if \(A_1\), \(B_1\), and \(E\) are irreducible types, then:

acting on the input feature space (\(f\)), and the representation

acting on the output feature space (\(\psi(f)\)).

In matrix form, the symbol \(\oplus\) denotes formation of block-diagonal matrix:

For example, consider the output representation \(\pi\) and suppose that the irreducible types \(A_1\) and \(B_1\) are 1-dimensional, while \(E\) is 2-dimensional:

Then the output feature vector \(\mathbf{f'} = \psi(f)\) (the fiber) can be written as

and under a transformation \(g\), it transforms according to \({\mathbf{f}_g}'=\pi(g)\mathbf{f'}\). Each block of the feature vector therefore transforms according to its own irreducible representation.

We can also have several copies of the same irreducible representation. For example,

means that the feature space contains: 3 copies of \(A_1\), 2 copies of \(B_1\), and 4 copies of \(E\). If \(A_1\) and \(B_1\) are one-dimensional while \(E\) is two-dimensional, then the total dimension of the representation is \(\dim(\pi)=3(1)+2(1)+4(2)=13\). More explicitly, the representation matrix has the block-diagonal form

In other words, the notation \(\rho = \bigoplus_i n_i\rho_i\) and \(\pi = \bigoplus_j m_j\pi_j\) means that the feature space is constructed by taking several copies of different irreducible representations and placing their representation matrices along the diagonal. This decomposition of the group representations into irreducible representations allows a steerable CNN to contain different feature types that transform differently under rotations and reflections, while still having a well-defined overall transformation rule.