Ward Gauderis

Sparse autoencoders have become a standard tool for uncovering interpretable latent representations in neural networks. Yet salient concepts often span manifolds that current linear methods cannot capture without post hoc analysis. This paper uses quadratic latents to close this gap: we implement these with bilinear autoencoders, which decompose activations into low-rank quadratic forms, compose linearly in weight space, and admit input-independent geometric analysis. This qualitative difference in what concepts quadratic latents can detect challenges the standard linear representation hypothesis. Our experiments and visualisations show that multi-dimensional geometries are highly prevalent and that composite latents capture them well, systematically improving reconstruction error in language models. Furthermore, we show that autoencoders with varying geometric priors recover the same input subspace despite their dictionary entries being distinct. Practically, these models serve as an unsupervised tool for manifold discovery, which we demonstrate through an interactive online visualizer for Qwen 3.5. This is a step toward nonlinear but mathematically tractable latent representations whose composition is expressive and interpretable by design.

Mechanistic interpretability aims to explain neural model behaviour by reverse-engineering learned computational structure into human-understandable components. Without a formal framework, however, mechanistic explanations cannot be objectively verified, compared, or composed. We introduce compositional interpretability, a category-theoretic framework grounded in the principles of compositionality and minimum description length. Compositional interpretations are pairs of syntactic and semantic mappings that must commute to enforce consistency between a model’s decomposition and its observed behaviour. We deconstruct explanation quality into measures of faithfulness and complexity to cast interpretability as a constrained optimisation problem, and introduce compressive refinement to systematically restructure models into simpler parts without altering their function. Finally, we prove a parsimony criterion under which syntactic compression theoretically guarantees more concise, human-aligned explanations. Our framework situates prominent mechanistic methods as subclasses of refinement, and clarifies why their compressibility heuristics tend to align with human interpretability. Our work provides a measurable, optimisable foundation for automating the discovery and evaluation of mechanistic explanations.

We propose χ-net, an intrinsically interpretable architecture combining the compositional multilinear structure of tensor networks with the expressivity and efficiency of deep neural networks. χ-nets retain equal accuracy compared to their baseline counterparts. Our novel, efficient diagonalisation algorithm, ODT, reveals linear low-rank structure in a multilayer SVHN model. We leverage this toward formal weightbased interpretability and model compression.

This thesis explores novel approaches to compositional reasoning in AI leveraging the mathematics of quantum theory as a general probabilistic theory. Starting from the quantum picturialism paradigm, offering a diagrammatic category-theoretic language, I show that quantum theory provides practical modelling and computational benefits for AI. A literature survey connect various applications, from quantum game theory and satisfiability to ML, NLP and cognition. How to formally represent and reason with concepts is a longstanding challenge in cognitive science and AI. My thesis studies the Quantum Model of Concepts (QMC), which provides conceptual space theory with quantum theoretical semantics. The diagrammatic language serves as a compositional framework for both, exposing common structures and facilitating insights between domains. I implement the model as a hybrid quantum-classical architecture on real quantum hardware to explore how QMCs can form practical intermediate, compositional representations for artificial agents combining symbolic and subsymbolic reasoning. Addressing the symbol grounding problem, I show that QMC representations can be learned from raw data in a (self-)supervised subsymbolic way, but that composite concepts can also be grounded in simpler ones to be interpretable and data-efficient. By transforming quantum concepts into probabilistic generative processes, the QMC can solve visual relational Blackbird puzzles involving abstraction and perceptual uncertainty, similar to Raven’s Progressive Matrices.