Loss landscape analysis is the study of how the values of a machine learning model’s loss function change as its parameters are adjusted. It helps researchers and engineers understand how easy or difficult it is to train a model by visualising or measuring the shape of the loss surface. A smoother or flatter loss landscape…
Quantisation-Aware Training is a method used to prepare machine learning models for running efficiently on devices with limited computing power, such as smartphones or embedded systems. It teaches the model to handle the reduced precision of numbers, which happens when large models are made smaller by using fewer bits to represent data. This approach helps…
Batch normalisation is a technique used in training deep neural networks to make learning faster and more stable. It works by adjusting and scaling the activations of each layer so they have a consistent mean and variance. This helps prevent problems where some parts of the network learn faster or slower than others, making the…
Attention Rollout is a technique used to visualise and interpret how information flows through the layers of an attention-based model, such as a transformer. It helps to track which parts of the input the model focuses on at each stage, giving insight into the decision-making process. This method combines attention maps from different layers to…
Recursive Neural Networks are a type of artificial neural network designed to process data with a hierarchical or tree-like structure. They work by applying the same set of weights recursively over structured inputs, such as sentences broken into phrases or sub-phrases. This allows the network to capture relationships and meanings within complex data structures, making…
Perceiver Architecture is a type of neural network model designed to handle many different types of data, such as images, audio, and text, without needing specialised components for each type. It uses attention mechanisms to process and combine information from various sources. This flexible design allows it to work on tasks that involve multiple data…
Masked modelling is a technique used in machine learning where parts of the input data are hidden or covered, and the model is trained to predict these missing parts. This approach helps the model to understand the relationships and patterns within the data by forcing it to learn from the context. It is commonly used…
Activation functions are mathematical formulas used in neural networks to decide whether a neuron should be activated or not. They help the network learn complex patterns by introducing non-linearity, allowing it to solve more complicated problems than a simple linear system could handle. Without activation functions, neural networks would not be able to model tasks…
Language modelling heads are the final layers in neural network models designed for language tasks, such as text generation or prediction. They take the processed information from the main part of the model and turn it into a set of probabilities for each word in the vocabulary. This allows the model to choose the most…
Diffusion models are a type of machine learning technique used to create new data, such as images or sounds, by starting with random noise and gradually transforming it into a meaningful result. They work by simulating a process where data is slowly corrupted with noise and then learning to reverse this process to generate realistic…