DeepSeek

DeepSeek R1

An LLM trained entirely through reinforcement learning (RL), without supervised fine-tuning. This model develops reasoning behaviors like self-verification and reflection purely via just RL

• Exploration via stochastic sampling by generating multiple responses
• Doesn't rely on supervised fine-tuning (SFT) beforehand

Enhanced Version with Cold Start: To overcome issues like poor readability and language mixing, DeepSeek-R1 adds a "cold start" phase using a small, curated set of human-readable reasoning examples before applying RL, followed by additional supervised fine-tuning and a second RL stage.

• Inject formatting and readability: Cold-start examples include summaries, structured CoT formatting, markdown, etc.

Distillation to Smaller Models: They distill reasoning ability from DeepSeek-R1 into smaller dense models (1.5B--70B). These distilled models outperform both larger open-source baselines and models trained with RL alone, highlighting distillation as more efficient than direct RL for smaller models.

Evaluation Results: DeepSeek-R1 matches or exceeds performance of OpenAI's o1-mini and approaches o1-1217 on math, coding, and reasoning benchmarks (e.g., 97.3\% on MATH-500, 79.8\% on AIME), outperforming prior open-source models.

Core Insight: Reinforcement learning — especially when combined with modest supervision and structured prompt engineering — is sufficient to incentivize emergent reasoning behaviors, and these can be efficiently propagated to smaller models via distillation.

Mixture-of-Experts (MoE) architecture: only certain parts of model activated, the ones that are important

• Experts (usually a FFN after head in transformer block): Each MoE layer has 16--64 experts.
• Routing: A top-k gating mechanism determines which experts process each token (usually k=2).
• Load balancing: Includes loss penalties to ensure tokens are evenly distributed across experts.

DeepSeekMath

\ul{PPO policy objective}:

Reinforcement Learning (RL) doesn\textquotesingle t fundamentally teach the model new mathematical reasoning abilities. Instead, it improves how well the model ranks or selects outputs it already knows how to generate.

Further Improvements: advanced sampling, Weak-to-string alignment methods, generalize ability and reflect uncertainty of reward model

RL Finetuning

TinyZero

Takes a pre-trained base LM and fine-tunes it via reinforcement learning on narrowly scoped reasoning games (CountDown and simple multiplication), for under \$30 and in about 10 GPU-hours

2 tasks:

• CountDown: combine given numbers with arithmetic to reach a target.
• Multiplication: break a multi-digit multiplication into sub-steps.

Given a base LM, prompts and ground-truth reward, we run RL.

Base model matters:

PPO, GRPO, PRIME all seem to work well, but they didn't tune hyperparameters

GPT-3

Language Models are Few-Shot Learners

Introduces GPT-3 (with 175 billion parameters), can perform tasks like translation, question answering, and arithmetic without gradient updates—just by seeing a few examples (few-shot), one example (one-shot), or even none (zero-shot).

The model is trained on hundreds of billions of tokens from filtered web corpora, books, and Wikipedia

No SFT: Unlike prior models that rely on supervised fine-tuning, GPT-3 operates purely via text prompts. This removes the need for task-specific model adjustments or training datasets.

GPT-3 isn't learning in the classical meta-learning sense (no inner-loop updates), but behaves like it's doing fast adaptation by interpreting examples in context.

They focus on in-context learning, specifically the few-shot setting.

• In few-shot learning, the model is given a few demonstrations (typically 10 to 100, as many as fit in the context window) of a task at inference time as conditioning.
  • provide the model with multiple examples to increase its chance of inferring the task correctly
• Crucially, this is done without any gradient updates or fine-tuning on task-specific datasets.
• This contrasts with traditional methods that require large datasets for fine-tuning (thousands or tens of thousands of examples)

GPT-3 achieves strong performance on many NLP tasks in the few-shot setting

• sometimes competitive with or even occasionally surpasses state-of-the-art fine-tuned models

Achieving this level of in-context learning requires obscene compute: training took 300 billion tokens and thousands of petaflop/s-days

Zero-shot purely through pattern recognition from pretraining

Prompt engineering matters

Input to decoder: the input from the user. Since the decoder is masked, each next token takes context from before.

• The self-attention mechanism within its decoder layers allows it to build a rich, contextual understanding of this entire evolving sequence.

Chain-of-Thought

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Demonstrates Emergent Reasoning via Prompting
The paper shows that only sufficiently large models (e.g., PaLM-540B) exhibit meaningful gains from CoT prompting. Smaller models (e.g., GPT-2, T5-small) don't benefit, highlighting an emergent property of scale.

Improves Performance on Reasoning Tasks
CoT prompting significantly boosts performance on:

• Arithmetic reasoning (e.g., GSM8K)
• Commonsense reasoning (e.g., CommonsenseQA)
• Symbolic reasoning (e.g., date manipulation, logic puzzles)

No Fine-tuning Required
All improvements are achieved using only in-context examples—no additional training. The model just needs to see examples of how to reason step-by-step.

Supports Better Interpretability
CoT prompts not only boost accuracy but also produce explainable intermediate steps, offering more transparency into how the model arrives at an answer.

Attention

Attention Is All You Need

• Unlike RNNs, LSTMs, that run sequentially
• Self-attention

• Parallelism
• Context at every layer
• Any two tokens have constant distance

• Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions

3 types of Self-Attention

Source: "le chat est assis"
Target: "the cat is sitting"

• Encoder-Decoder Attention (Align target tokens to source)
  • Queries: From the decoder (current target-side position).
  • Keys & Values: From the encoder (entire source-side sequence).
  • decoder learns which parts of the source to attend to for each generated token
  • Outputs a context vector: weighted sum of the encoder's outputs

Input: query from Decoder Self-Attention, key + value from Encoder Self-attention

Target output (labels): {[}"the", "cat", "is", "sitting", "</s>"{]}

• Encoder Self-Attention (Encode source context)
  • Enables the model to contextualize each token with respect to the entire source sequence.
  • Not masked

Input: {[}"le", "chat", "est", "assis"{]} + positional embeddings

Tokenized using a embedding matrix (which each input is an index in the matrix)

Output: {[}z1, z2, z3, z4{]} (representation of entire source)

• Decoder Self-Attention (Generate autoregressively)
  • Masked, $-\infty$ before softmax
  • Tokens rely only on tokens before

Input: {[}"< s>", "the", "cat", "is"{]} (during training)

Output: (the query for Encoder-Decoder Attention)

Post Multi-Head:

u = LayerNorm(x + Attention(x))

Stabilizes and normalizes the output after attention, before passing it to the FFN

z = LayerNorm(u + FFN(u))

Ensures that the output of the FFN sub-layer is normalized before being passed to the next decoder/encoder block or to output projection.

Feed-Forward Networks

• Introduces non-linearity
• Higher dimensional spaces
• FFN learns to transform features at each position independently

Layer Norm (Batch Norm would normalize a feature across tokens, bad)

• Given a token, normalizes its vector of features

$\mathrm{LayerNorm}(x) = (x - \mu)/\sigma * \gamma + \beta$, where $\gamma, \beta$ are learnable features

Equivariant NN

Intro to Equivariant NNs

Equivariant layers are built using group representations

• $\rho_1(g)$ is a linear operator on V1 representing the action of $g \in G$ on V1 .
• $\rho_2(g)$ is a linear operator on V2 representing the action of $g \in G$ on V2 .

Now, an intertwiner $L:V_1 \to V_2$ is a linear map that commutes with the group action:

$L \circ \rho_1(g)=\rho_2(g) \circ L,\ \forall g \in G$

directly connected to graph neural networks.

• 1st-order actions correspond to node features.
• 2nd-order actions correspond to edge features — pairs of nodes.

N=5, weights=15, instead of 5^4=625

CNNs

$L \circ \rho(t)=L,\ \forall t \in C_N$

Occurs when L takes the sum * some weight <- 1 parameter, another is max-pooling

Steerable nonlinearities

• instead of thinking in terms of "pointwise" nonlinearities, think of nonlinearities that act on the whole representation space, respecting the group action.

• Compute invariant quantities (typically norms: \textbar\textbar v\textbar\textbar, inner products, etc.).
• Apply scalar nonlinearities to these.
• Multiply the original vector by a function of these invariants.

V: invariant under R

Phi: scalar function

f(v) = is a steerable equivariance nonlinearity

Using this steerable equivariance over ReLU helps leverage the symmetry in the data

\texorpdfstring{In practice:}{In practice:}

In an equivariant NN layer:

1. You pass feature vectors through some linear equivariant operation (convolutions, tensor products, etc.).
2. You decompose the resulting feature space into irreps + multiplicities.
3. On each irrep block V\_i :
   • You apply a steerable nonlinearity
4. On the multiplicity space (the R^m\_i ), you can apply arbitrary MLPs, ReLU, whatever.

Gated Scalar-vector Nonlinearity

feature vector that has been structured as:

• Scalar channels s, i.e. features that transform trivially under the group (invariant or trivial irrep).
• Higher-order vector channels v, which transform according to some non trivial representation $\rho$.

1. Scalar origin: The gate g\_i comes from trivial-rep channels (or invariants like norms), so it's unchanged by any group action.
2. Identity scaling: Multiplying a nontrivial feature v\_i by a scalar g\_iis equivalent to applying the linear map g\_i * I, which commutes with every group-action matrix $\rho(g)$

.

1. Expressivity vs. cost: Gating gives you per-channel “on/off” control or smooth rescaling at just O(n) cost, whereas full tensor-product nonlinearities are $O(n^2)$

Teacher Forcing

Professor Forcing: A New Algorithm for Training Recurrent Networks

Distribution of discrete time sequence

However, during generation, the RNN generates a whole sequence by itself, but the ground-truth tokens are not available anymore. So it is conditioned on its own previous predictions, which could be incorrect.

Bengio et al., 2015 : mixing two kinds, those from the ground-truth training sequence and those generated from the model (with some probability p < 1)

• But it starts feeding in the model's own predictions during training, which implicitly changing the conditioning context
• Ground truth next token is not necessarily the correct or most likely one
  • Mitigate by making self-generated subsequences short
  • scheduled sampling yields a biased estimator

Professor Forcing

• regularizer for recurrent networks
• training performance can also be improved, and conjecture that it is because longer-term dependencies can be more easily captured

Idea

• Utilize GAN framework to match the idea that it is indistinguishable whether the network is trained with its inputs clamped to a training sequence
• train a second model, which we call the discriminator, and that can also process variable length inputs

Two modes:

• Teacher forcing: inputs are ground-truth tokens.
• Free-running: inputs are model\textquotesingle s own predictions.

Discriminator: learns to classify whether a behavior sequence (hidden states + outputs) came from teacher forcing or free-running.

Behavior sequence b: extracted from the generator RNN (hidden states pre-activation and optionally outputs).

• To align the two modes
• Summarizes how the RNN is behaving internally

• Teacher-forced

• Free-running

Generator: 1-layer GRU (1024 units for character LM, 512 for MNIST, 3-layer for MNIST).

Discriminator: bidirectional GRU (except convolutional net for MNIST).

Training: Adam optimizer, Theano framework.

Gradient-based learning algorithm for RNNs that runs online but has a high computation cost as nonlocal communication is required.

Prior works

• Hopfield nets: stable associative memory.
• Backpropagation Through Time (BPTT): unfolding RNNs into feedforward nets, but with growing memory requirements.

Fully recurrent neural network:

• Every unit can be connected to every other unit, including recurrent self-connections.
  Inputs are provided at each time step.
• The network runs continually — no separate forward pass / backward pass cycle like in a standard feedforward net.

y(t+1) = f(W z(t)), where z(t) includes both inputs x(t) and previous outputs y(t)

Cycles due to recurrent connections with hidden state

Option 1: Backpropagation Through Time (BPTT) O(T) in time

• Unroll the RNN in time (vanilla sequence): treat it as a deep feedforward net where each time step is one layer
• Apply regular backpropagation through this unrolled graph.

Option 2: Real-Time Recurrent Learning (RTRL) O(n^3) in time

• Maintain online estimates of $\partial h_t / \partial W$, as the network runs.
• For each weight W\_ij, maintain an auxiliary variable:
  • $P_t = \partial h_t / \partial W_{ij}$
• Update P\_t recursively at each time step as h\_t evolves.
  • $\partial L_t / \partial W_{ij} = \partial L_t / \partial h_t \times P_t$

Option 3: Teacher-Forced Real-Time Recurrent Learning (used with BPTT)

• replacing actual outputs with desired outputs during training when available.

• $\partial L_{\text{total}} / \partial W_{hh}$
• $\partial L_{\text{total}} / \partial W_{xh}$
• $\partial L_{\text{total}} / \partial W_{hy}$
• etc.

\texorpdfstring{3. Simulation Experiments}{3. Simulation Experiments}

They test the algorithm on several tasks requiring memory and dynamics:

\paragraph{\texorpdfstring{3.1 Pipelined XOR}{3.1 Pipelined XOR}}

• XOR delayed by 2--4 time steps.
• Network learns to compute delayed XOR by configuring itself into a deeper architecture.

Critique: Simple but useful sanity check.

\paragraph{\texorpdfstring{3.2 Simple Sequence Recognition}{3.2 Simple Sequence Recognition}}

• Task: detect the first b after an a.
• Cannot be solved by feedforward networks with fixed delay.
• Network learns to implement a flip-flop mechanism.

Critique: Demonstrates that the network can learn internal state — the core point of using RNNs.

\paragraph{\texorpdfstring{3.3 Delayed Nonmatch to Sample}{3.3 Delayed Nonmatch to Sample}}

• Task: remember a bit and compare to later inputs.
• Network sometimes develops explicit memory units; sometimes develops distributed, dynamic memory.

Critique: This is one of the most insightful observations in the paper — that the network may not store information statically, but encode it dynamically in the system's trajectory. This challenges naive intuitions about neural representations.

\paragraph{\texorpdfstring{3.4 Learning to Be a Turing Machine}{3.4 Learning to Be a Turing Machine}}

• Imitate the finite-state controller of a Turing machine deciding balanced parentheses.
• Success with 12--15 units.

Critique: Impressive demonstration of the generality of the method. But no analysis of what the network actually learned — is it really equivalent to the controller or is it an ad hoc solution?

\paragraph{\texorpdfstring{3.5 Learning to Oscillate}{3.5 Learning to Oscillate}}

• Tasks: simple oscillations (periodic outputs).
• Found that teacher forcing was critical for success.

Critique: This is an important negative result: without teacher forcing, RTRL often fails to discover oscillatory solutions due to gradient flatness. Highlights the limitations of local gradient methods.

BERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers

• Which then can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks (a bit too general, might require more)
• Encoder only model

Existing Models

• Existing approaches were either feature-based (ELMo) or fine-tuning-based (GPT), but constrained by unidirectional pre-training
  • ELMo used contextualized word embeddings, but only left-to-right or right-to-left separately.
  • Though GPT uses Transformer blocks (with full attention capability), it masks future tokens during training to predict the next word — this is what makes it left-to-right.

BERT

• Bert addresses this with via masked language modeling (MLM) and next sentence prediction (NSP)
  • MLM: randomly masks some of the tokens from the input, and objective is to predict the original vocabulary id
  • NSP: during pre-training, BERT is fed pairs of sentences, and for each pair, it has to predict whether the second sentence logically follows the first one.

• Input
  • {[}CLS{]} token at beginning: final hidden state often used for classification
  • Tok 1, \ldots, Tok n: tokens for Masked Sentence A, B
    • Some tokens are masked (\textasciitilde15\%)
  • {[}SEP{]} token: token to distinguish separation between two sentences, and at the end
  • "Unlabeled Sentence A and B Pair": either consecutive sentences (NSP) or randomly paired (negative NSP)
• Embedding (yellow)
  • input embeddings for each token
    • Contains token embeddings, segment embeddings (between sentence 1 and 2), and position embeddings
• • multiple layers of self-attention and feed-forward networks to get contextual representations
  • C, T\_1, T\_N, T\_{[}SEP{]}, T\_1\textquotesingle, T\_M\textquotesingle: the output hidden states (contextualized embeddings) produced by BERT for each corresponding input token.
    • C is the output for the {[}CLS{]} token.
• Pre-training Tasks
  • NSP
    • red arrow to predict whether "Sentence B" is the actual next sentence after "Sentence A" or a randomly sampled one
    • When creating dataset, we know whether sentence B was after <- groundtruth
      • compares its predicted probability distribution with the actual (true) label
  • Masked LM
    • The red arrows pointing up indicate that these are positions where input tokens were masked
    • final hidden state corresponding to the {[}MASK{]} token is passed through a classification layer
      • Outputs a probability distribution over the entire vocabulary, cross entropy loss to update weights to increase probability of predicting ground truth word

• Types of fine-tuning tasks
  • MNLI (Multi-Genre Natural Language Inference): A classification task where BERT determines the relationship (entailment, contradiction, neutral) between two sentences.
    • Output C token put into a classification layer
  • NER (Named Entity Recognition): A token-level classification task where BERT identifies and classifies named entities (e.g., person, organization, location) in text
    • The outputs for each token are fed into a classification layer to predict the entity tag for that token.
  • SQuAD (Stanford Question Answering Dataset): A question-answering task where BERT identifies the start and end span of an answer within a given paragraph based on a question.
• Input
  • "Question Answer Pair":
    • Specifically for SQuAD, the input is {[}CLS{]} Question {[}SEP{]} Paragraph {[}SEP{]}.
• Fine-tuning Output
  • for SQuAD, BERT predicts two probabilities for each token in the paragraph: one for being the start of the answer span and one for being the end of the answer span
    • Tokens with highest probability chosen

Experiments

• BERTLARGE substantially outperformed prior baselines across tasks.
• Fine-tuning required only modest compute (\textasciitilde1h TPU).

Ablation Studies

Pretraining

• NSP helped on tasks inter-relating sentences but not on individual sentence understanding
• Deep bidirectional MLM was clearly superior to left-to-right
  • Full context has richer and more powerful word representations
• BiLSTM added to LTR helps but can\textquotesingle t match deep bidirectional
  • Considering all tokens in the sequence simultaneously from the beginning is critical

Effect of Model Size

• Larger models yield consistent gains even on small tasks.

Feature-based vs Fine-tuning

• Fine-tuning is generally better, but BERT features alone are still strong (CoNLL NER).
• This suggests both paradigms remain viable.

Bert Freezing

• Preventing Catastrophic Forgetting
  • fine-tuning all layers can quickly overwrite valuable pre-trained features in favor of overfitting to the small dataset
• Computationally Efficient
  • Updating all weights is very expensive
• Regularization
  • Prevent overfitting
• Improve Learning Effectiveness
  • lower layers of BERT typically learn more general, fundamental linguistic features
  • Higher layers are more task specific

Types of freezing:

• Freezing All Layers (very small dataset)
• Freezing Bottom N Layers (balances performance with computational efficiency)
• Gradual Unfreezing (moderate-sized datasets, more stable and controlled)

InstructGPT

Training language models to follow instructions with human feedback

OpenAI's InstructGPT paper tackles a fundamental misalignment between large language models (LLMs) and user intent

GPT-3 (175B parameters): often produces outputs that are untruthful, unhelpful, or toxic

model size alone does not guarantee alignment; instead, fine-tuning with human feedback

Techniques

Supervised Fine-Tuning (SFT):

• Data collection: Professional labelers wrote ideal “demonstrations” for a broad suite of prompts (both labeler-crafted and real API queries)
• Training: These demonstrations were used to fine-tune GPT-3 via standard supervised learning, yielding an initial “SFT model”.

Reinforcement Learning from Human Feedback (RLHF):

• Preference data: For each prompt, multiple candidate outputs were generated by the SFT model. Human annotators ranked these candidates by helpfulness and alignment.
• Reward modeling: A neural reward model was trained on these pairwise preferences to predict human judgments.
• PPO fine-tuning: Proximal Policy Optimization was used to optimize the SFT model against the learned reward, producing the final InstructGPT models.

Evaluated on 1.3 B, 6 B, and 175 B parameter variants

Findings

Human preference: Outputs from the 1.3 B InstructGPT were preferred over those from 175 B GPT-3 in blind evaluations, demonstrating that \ul{instruction fine-tuning can beat a much larger base model
}

Truthfulness & toxicity: InstructGPT showed measurable gains in producing factual content and reductions in toxic language, without significantly sacrificing performance on standard NLP benchmarks.

Trade-offs: Some narrow regression occurred on a few academic tasks, suggesting a mild specialization effect from instruction tuning

Limitations & Future Directions

• Assessing deeper values alignment: Current metrics target helpfulness and truthfulness, but don't measure subtler ethical dimensions (e.g., fairness, privacy).
• Improving reward model robustness: Research is needed on adversarial training and uncertainty estimation to detect when the reward model is unreliable.
• Automating feedback loops: Can synthetic or semi-supervised methods (e.g., model-self critique) reduce reliance on expensive human annotation?
• Does RLHF meaningfully move the needle on robust, long-term alignment, or merely patch surface behaviors for common cases?

PPO Math

REINFORCE Objective: Standard Policy gradient to be maximized

Not used extensively in PPO, but used in GRPO/TRPO

KL divergence (Kullback--Leibler divergence) is a way to measure how different two probability distributions are.

So like TRPO/GRPO:

Remember: $\pi_\theta$ and $\pi_{\theta_{\text{old}}}$ are both autoregressive LLMs, so taking ratio of softmax logits at position t for each LLM gives our importance sampling ratio

GRPO algorithm

Hopfield Networks

Hopfield Networks is All You Need

Structure

• Single Layer of Neurons: It consists of a single layer of interconnected neurons. There are no distinct input, hidden, or output layers like in feedforward networks.
• Fully Connected: Every neuron in the network is connected to every other neuron.
• No Self-Connections: A neuron is typically not connected to itself (i.e., the connection weight from a neuron to itself is zero).
• Symmetric Weights: The connections between neurons are bidirectional and symmetric. This means the weight of the connection from neuron i to neuron j (Wij ) is the same as the weight from neuron j to neuron i (Wji ). This symmetry is crucial for the network\textquotesingle s stability and its ability to converge to stable states.
• Binary (or Bipolar) Neuron States: The neurons in a Hopfield network typically have binary states, meaning they can be either "on" or "off." These states are often represented as 0 or 1, or more commonly, as -1 or +1 (bipolar).
• Recurrent Nature: It\textquotesingle s a type of recurrent neural network, meaning the output of neurons feeds back as input to other neurons, leading to dynamic evolution of the network\textquotesingle s state over time.
• Memory Storage in Weights: The "memories" or patterns that the Hopfield network stores are embedded in the connection weights between the neurons. These weights are typically calculated using a learning rule, most commonly the Hebbian learning rule, based on the patterns to be stored.
• Energy Function: A key feature of Hopfield networks is the existence of an "energy function" (also known as a Lyapunov function). The network dynamics are designed to always decrease this energy function until it reaches a local minimum, which corresponds to a stored pattern or an attractor state

\ul{Findings}

Bridges associative memory and attention: The equivalence to attention provides a new lens to understand attention mechanisms. \ul{Transformer attention heads can be viewed as gradient descent steps in a modern Hopfield energy} landscape.

Integrates memory into deep learning: Hopfield layers can be added as differentiable memory modules, with higher capacity than RNNs or LSTMs.

Fast retrieval: Unlike classical Hopfield nets, one update is typically enough to retrieve the pattern.

Broad applicability: The authors test these layers in multiple instance learning (MIL), UCI small datasets, and drug discovery — and show SOTA or near-SOTA performance in several cases

GNN

SEMI-SUPERVISED CLASSIFICATION WITH GRAPH CONVOLUTIONAL NETWORKS

Classical spectral graph convolutions are defined via the eigendecomposition of the Laplacian, but this is computationally expensive ($O(N^2)$)

Hammond et al.\textquotesingle s Chebyshev polynomial approximation allows for K-localized filters with linear complexity in \textbar E\textbar.

Hence, a neural network model based on graph convolutions can therefore be built by stacking multiple convolutional layers

Learning a function f(X, A) using a neural network that operates on the graph adjacency

Matrix, 2-layer GCN

Summary: Implemented in TensorFlow with sparse-dense matmuls. Full-batch gradient descent is used (possible because graphs are small).

Critical Insight: Full-batch training is unrealistic for large graphs

\paragraph{\texorpdfstring{Semi-Supervised Node Classification}{Semi-Supervised Node Classification}}

Summary: GCN outperforms all baselines on accuracy and efficiency.

Critical Insight: The fact that a 2-layer GCN beats DeepWalk (unsupervised) and Planetoid (more complex pipeline) is strong validation of the model's practical utility.

\paragraph{\texorpdfstring{Evaluation of Propagation Model}{Evaluation of Propagation Model}}

Summary: Various propagation models tested; renormalization trick works best.

Critical Insight: The superiority of this trick is empirical, not theoretically guaranteed. That makes it a somewhat brittle part of the model's foundation.

\paragraph{\texorpdfstring{Training Time per Epoch}{Training Time per Epoch}}

Summary: Scales linearly in \textbar E\textbar{} as expected.

Insight: The simplicity and scalability of this design is its biggest virtue.

Learning Optimizers

Learning to learn by gradient descent by gradient descent

If neural networks can learn good features, why can't they also learn good update rules?

performance of vanilla gradient descent, however, is hampered by the fact that it only makes use of gradients and ignores second-order information

• rescaling the gradient step using curvature information, typically via the Hessian matrix of second-order partial derivatives
  • General Gauss-Newton matrix
  • Fisher information matrix

replace hand-designed update rules with a learned update rule, which we call the op-

timizer g

long history

• Shared Parameters: A small LSTM network is used for each coordinate of the optimizee\textquotesingle s parameters. These LSTMs share their parameters across all coordinates.
• Separate Hidden States: Although the parameters are shared, each coordinate has its own separate hidden state, allowing the optimizer to behave differently for each parameter.
• Permutation Invariance: This design makes the optimizer invariant to the order of the parameters in the network being optimized

Results:

Quadratic Functions: On 10-dimensional quadratic functions, the learned optimizer significantly outperformed the baseline optimizers.

MNIST: The LSTM optimizer was trained to optimize a small multi-layer perceptron (MLP) on the MNIST dataset.

• It demonstrated superior performance compared to standard optimizers on the base network architecture.
• The learned optimizer showed strong generalization capabilities. It continued to outperform baselines when tested on networks with more hidden units (40 instead of 20) and more layers (2 instead of 1).
• However, when the activation function was changed from sigmoid to ReLU, the learned optimizer\textquotesingle s performance degraded, indicating that its learned strategy was specific to the training dynamics.

CIFAR-10: For a convolutional neural network on CIFAR-10, the authors used a slightly modified optimizer with two LSTMs—one for convolutional layers and one for fully connected layers.

• The LSTM optimizer learned more quickly than the baseline methods.
• It also demonstrated effective transfer learning, performing well on subsets of the CIFAR-10 dataset (CIFAR-5 and CIFAR-2). An optimizer trained only on a disjoint subset of labels also transferred well to a new dataset.

Neural Art: The task of artistic style transfer, where each image pair creates a new optimization problem, served as a natural testbed.

• An LSTM optimizer was trained on a single style and a set of content images at a 64x64 resolution.
• It outperformed standard optimizers at the training resolution and with the training style.
• Remarkably, it continued to show strong performance when generalizing to a new style and a higher resolution (128x128) simultaneously.

1. Optimizee to Optimizer: The "Error Signal"

The optimizee, which is the function being minimized (e.g., a neural network), sends an "error signal" to the optimizer. This signal is the gradient of the optimizee\textquotesingle s objective function, notated as $\nabla f(\theta_t\hspace{0pt})$. The optimizer, which is modeled as a recurrent neural network (RNN), takes this gradient as an input at each step.

2. Optimizer to Optimizee: The "Parameter Updates"

The optimizer receives the gradient and, based on this input and its own internal state, proposes a parameter update, denoted as gt . This update is then sent back to the optimizee. The optimizee applies this update to its own parameters, $\theta$, according to the formula: $\theta_{t+1}\hspace{0pt}=\theta_t\hspace{0pt}+g_t\hspace{0pt}$. This cycle repeats, with the optimizer continuously proposing updates based on the optimizee\textquotesingle s performance to improve it

Mask R-CNN

Mask R-CNN Overview

• R-CNN (Region-based CNN): The initial R-CNN approach for bounding-box object detection involves identifying a manageable number of candidate object regions within an image. A convolutional network is then independently evaluated on each of these proposed regions of interest (RoIs).
• Fast R-CNN: This extended R-CNN by allowing the model to attend to RoIs directly on feature maps using an operation called RoIPool. This led to faster processing speeds and improved accuracy.

• Faster R-CNN: This further advanced the R-CNN stream by introducing a Region Proposal Network (RPN). The RPN learns to predict candidate object bounding boxes, making the entire object detection pipeline end-to-end and more efficient. Faster R-CNN is highly flexible and has become a leading framework in various benchmarks.

\paragraph{\texorpdfstring{1. Parallel Mask Prediction}{1. Parallel Mask Prediction}}

• Adds a mask prediction branch for each RoI, in parallel to classification and box regression.
• Predicts per-instance binary masks, not just object classes.

This mask branch in Mask R-CNN is a dedicated sub-network responsible for producing a segmentation mask—a pixel-level binary map indicating the shape and extent of each detected object instance

For each Region of Interest (RoI): (can be any dimension)

• The mask branch receives the aligned feature map from that RoI via RoIAlign.
• These features are processed by a small Fully Convolutional Network (FCN).
• The output is a $m \times m$ mask per class (typically $28 \times 28$).
  • So the RoI is split into mxm grid
• only the mask for the predicted class is kept and used

RoIAlign: Fixing Spatial Slop

• RoIPool introduced quantization when dividing proposals into bins (coordinates rounded to nearest int)
  • breaks fine mask boundaries
• RoIAlign has no quantization, but bilinear interpolation to sample features
  • Pixel-accurate

\paragraph{\texorpdfstring{ Decoupled Mask and Class Prediction}{ Decoupled Mask and Class Prediction}}

• Instead of one softmax over all class masks (like FCNs), Mask R-CNN uses:
  • A per-class binary mask output
  • A sigmoid per-pixel, not softmax
    • No class competition: better for overlapping and occluded instances
    • More robust gradients

Fully Convolutional Mask Branch

• the mask branch is a small FCN: it keeps spatial layout throughout.
• No flattening or fully connected layers = fewer parameters and better spatial precision.
• This matches well with the idea of segmentation: output shape should reflect input shape spatially, not just semantically

Runtime: remains slow (especially with large backbones) and heavy, which is why real-time variants like YOLOv8-seg, SOLO, or EfficientDet with segmentation heads emerge

NTK

Neural Tangent Kernel

• understand the complex behavior of deep neural networks
  • infinitely wide neural networks to kernel methods
  • Can analyze training dynamics, convergence properties, generalization capabilities
• quantifies how a small change in the network\textquotesingle s parameters affects the predictions for two given inputs
• defined as the inner product of the Jacobians of the network\textquotesingle s output with respect to its parameters
  \strut
  
• As number of neurons in each hidden layer of a neural network approaches infinity, non-linear dynamics of the neural network\textquotesingle s training simplify dramatically
  • Under gradient descent, an infinitely wide neural network behaves like a linear model
    • Individual parameter updates are so small can be approximated with a first order Taylor's
• distribution over functions represented by the network at initialization is a GP, where the covariance function is given by the

kernel is a function that computes the similarity between two data points. It is a key component of a class of algorithms known as kernel methods, with the most famous example being the Support Vector Machine (SVM)

kernel trick: operate in a high-dimensional feature space without ever having to compute the coordinates of the data in that space

• Never have to transform as this function equates to the dot product in the transformed space

in the context of machine learning, grokking refers to a phenomenon where a neural network, long after it has perfectly memorized its training data, suddenly and dramatically learns to generalize to new, unseen data.