Google Deepmind recently released this paper: Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. In a few words: relative to a vanilla transformer, you can tra...

The following is a crosspost from the alignment forum. This work was done as part of SERI MATS, under Leo Gao’s guidance. Thank you to Erik Jenner and Johannes Treutlein for discussions and comment...

The below is a cross-post from the alignment forum. Introduction This article aims to draw a connection between recent ML research and the claim that future advanced AI systems may be homogenous....

The full write-up of my dissertation is now public here. You can also find the code on Github. Here is the abstract: Modern network architectures generalise well even when the size of the netw...

Abstract: “In many situations, given a set of observations, we would like to find the factors which cause or influence the data we observe. To learn these factors, we can use a deep generative mode...

Generally speaking, the performance of neural networks on various tasks scales with their depth. Deep learning has been wildly successful on classification and regression tasks, and most recently o...

Do flat minima generalise better? Is there a way to bias standard optimisation algorithms like stochastic gradient descent (SGD) to prefer flatter minima? Sharpness-aware minimisation (SAM), introd...

An investigation of transformers’ and RNNs’ ability to model long-term dependencies through the lens of probabilistic modelling. You can find the write-up here, and the code here.

This project uses Gaussian processes (GPs) as surrogate models for accurate climate forecasting. We emulate the UKESM1.0 climate model for precipitation, surface temperature and snow thickness. Wor...

The full write-up of my undergraduate thesis is available here. Here is an abstract: Video-games contain a simulated reality not unlike ours, where the developer has control over constraints an...