Masked modelling is a technique used in machine learning where parts of the input data are hidden or covered, and the model is trained to predict these missing parts. This approach helps the model to understand the relationships and patterns within the data by forcing it to learn from the context. It is commonly used…
Activation functions are mathematical formulas used in neural networks to decide whether a neuron should be activated or not. They help the network learn complex patterns by introducing non-linearity, allowing it to solve more complicated problems than a simple linear system could handle. Without activation functions, neural networks would not be able to model tasks…
Feature attribution is a method used in machine learning to determine how much each input feature contributes to a model’s prediction. It helps explain which factors are most important for the model’s decisions, making complex models more transparent. By understanding feature attribution, users can trust and interpret the outcomes of machine learning systems more easily.
Language modelling heads are the final layers in neural network models designed for language tasks, such as text generation or prediction. They take the processed information from the main part of the model and turn it into a set of probabilities for each word in the vocabulary. This allows the model to choose the most…
Diffusion models are a type of machine learning technique used to create new data, such as images or sounds, by starting with random noise and gradually transforming it into a meaningful result. They work by simulating a process where data is slowly corrupted with noise and then learning to reverse this process to generate realistic…
Capsule Networks are a type of artificial neural network designed to better capture spatial relationships and hierarchies in data, such as images. Unlike traditional neural networks, capsules group neurons together to represent different properties of an object, like its position and orientation. This structure helps the network understand the whole object and its parts, making…
Residual connections are a technique used in deep neural networks where the input to a layer is added to its output. This helps the network learn more effectively, especially as it becomes deeper. By allowing information to skip layers, residual connections make it easier for the network to avoid problems like vanishing gradients, which can…
A Mixture of Experts is a machine learning model that combines several specialised smaller models, called experts, to solve complex problems. Each expert focuses on a specific part of the problem, and a gating system decides which experts to use for each input. This approach helps the overall system make better decisions by using the…
The Neural Tangent Kernel (NTK) is a mathematical tool used to study and predict how very large neural networks learn. It simplifies the behaviour of neural networks by treating them like a type of kernel method, which is a well-understood class of machine learning models. Using the NTK, researchers can analyse training dynamics and generalisation…
LoRA Fine-Tuning is a method used to adjust large pre-trained artificial intelligence models, such as language models, with less computing power and memory. Instead of changing all the model’s weights, LoRA adds small, trainable layers that adapt the model for new tasks. This approach makes it faster and cheaper to customise models for specific needs…