# Fine Tuning

## Objective

The primary goal is to **identify and quantify how a model's parameters adapt when trained on a new, specialized task**. By analyzing the changes in the trainable weights before and after fine-tuning, we aim to visualize which parts of the neural network are most significantly modified. This analysis helps reveal the "locus of learning," indicating whether the model primarily adjusts its foundational, early-layer representations or its abstract, late-layer reasoning capabilities to master a new domain.

## Methodology

The experiment was conducted using the `meta-llama/Llama-3.2-3B` model. To make the fine-tuning process computationally feasible, we employed **QLoRA (Quantized Low-Rank Adaptation)**, a highly efficient Parameter-Efficient Fine-Tuning (PEFT) technique.

The core of the methodology is a precise, step-by-step process:

1. **Model Quantization**: The base `Llama-3.2-3B` model was loaded using a 4-bit NormalFloat (`nf4`) quantization configuration via `BitsAndBytesConfig`. This dramatically reduces the VRAM footprint by loading the model's weights in 4-bit precision, making it possible to run on a single GPU. The computation, however, is performed in `bfloat16` for stability and performance.
2. **Low-Rank Adaptation (LoRA)**: Instead of training all 3 billion parameters of the model, we freeze the base model and inject small, trainable "adapter" matrices into the attention layers. Using the `peft` library, a `LoraConfig` was applied with the following key parameters:
   * `r=16`: The rank of the update matrices, determining the number of trainable parameters.
   * `lora_alpha=32`: A scaling factor for the LoRA updates.
   * `target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]`: The trainable adapters were specifically applied to the query, key, value, and output projection matrices within each self-attention block of the transformer architecture.

## Why Lora is a Valid Proxy for full training

Instead of modifying a huge, pre-trained weight matrix `W` (which has millions of parameters), LoRA keeps `W` frozen. It learns the *change* to the weights, `ΔW`, by representing this change as the product of two much smaller, "low-rank" matrices, `A` and `B`. During a forward pass, the model's output is calculated as `h = Wx + BAx`. Only `A` and `B` are trained. Because `A` and `B` are tiny compared to `W`, we are training only a fraction of a percent of the total parameters, leading to massive savings in memory and computation time.

**Why This Shows Similar Results to Full Training:** This experiment's methodology is grounded in a key research finding known as the **"low-rank hypothesis."** Studies have shown that when large pre-trained models are fully fine-tuned, the matrix representing the change in weights (`ΔW = W_final - W_initial`) tends to have a very low "intrinsic rank." This means the complex update across millions of parameters can be effectively approximated without losing significant information. LoRA is explicitly designed to create such a low-rank update. Therefore, by constraining the update to be low-rank, LoRA is not just an efficiency hack; it's a **principled approximation of the full fine-tuning process.** Analyzing where LoRA applies its largest changes gives us a strong and valid insight into which layers would have been most modified during a full fine-tuning run.

### The Experimental Process

1. **State Capture**: Before training began, a snapshot of the initial, randomly-initialized LoRA adapter weights (`A` and `B` for all targeted modules) was saved.
2. **Training**: The model was fine-tuned for a single epoch on a specialized dataset (e.g., `math_solution.json`). A batch size of 2 and 4 gradient accumulation steps were used, with a learning rate of `1e-3` and `2e-4`.
3. **Final State Capture**: After training, the final state of the LoRA adapter weights was captured.

## Analysis & Metrics

The core of the analysis is comparing the final LoRA weights to their initial state. For every trainable parameter, we calculate the **delta (Δ)**, which is the difference between the final and initial weight values:

**Δ = Weight**<sub>**after training**</sub>**&#x20;- Weight**<sub>**before training**</sub>

For each trainable LoRA adapter matrix (e.g., the `q_proj` matrix in layer 15), the following metrics were computed from its delta tensor (Δ):

* **Mean Change**: `mean(Δ)`. The average of all weight changes. It indicates the overall directional bias of the updates (i.e., did weights tend to increase or decrease?). A value near zero suggests a balanced shift.
* **Variance of Change**: `var(Δ)`. The variance of weight changes. This shows the spread or diversity of modifications within a single matrix.
* **Mean Absolute Change**: `mean(abs(Δ))`. The average of the absolute values of all weight changes. **This is the key indicator of the total magnitude of adaptation.** A higher value signifies that the layer underwent more significant changes, regardless of their direction.
* **Variance of Absolute Change**: `var(abs(Δ))`. The variance of the absolute changes, indicating how uniformly the magnitude of change was distributed across the weights in the matrix.

To create a clear, high-level overview, these individual matrix-level statistics were **aggregated by their parent transformer layer number**. This averages the metrics from the `q_proj`, `k_proj`, `v_proj`, and `o_proj` adapters as well as the `up_proj` and `down_proj` for each layer, providing a macroscopic view of how learning was distributed across the model's depth.

## Results

***

## Comparative Weight Changes Across Domains

### LR = 2e-4

|                                                                                       Mean Change Across All Domains                                                                                      |                                                                                    Mean Absolute Change Across All Domains                                                                                    |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-1dcf52a723c09d84f6cbcbee89dcef70d0b637b7%2F4_all_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-57f6ab06fa622f1069fb71b861dd8020ff845fbf%2F4_all_abs_mean.gif?alt=media) |

### LR = 1e-3

|                                                                                       Mean Change Across All Domains                                                                                      |                                                                                    Mean Absolute Change Across All Domains                                                                                    |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-15378ed11c505f2cd71f1d297f8d288afff75ce2%2F3_all_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-70960a7a12324c8154975bee754c8874eff4329d%2F3_all_abs_mean.gif?alt=media) |

***

## Weight Change Across Layers in Individual Domains

### C++ Domain

**LR = 2e-4**

|                                                                                        Aggregated Mean Change (C++)                                                                                       |                                                                                     Aggregated Mean Absolute Change (C++)                                                                                     |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-82985d8c81736c9557d94f0a66a0dcd701731956%2F4_cpp_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-491f4ceac32138adc2846b5daba1d11e7768199c%2F4_cpp_abs_mean.gif?alt=media) |

**LR = 1e-3**

|                                                                                        Aggregated Mean Change (C++)                                                                                       |                                                                                     Aggregated Mean Absolute Change (C++)                                                                                     |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-33eb35771a00113018d8fac4be55e6936f18af33%2F3_cpp_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-a8d75d79a8b91a238db68d5e37f811328aa3c577%2F3_cpp_abs_mean.gif?alt=media) |

***

### Python Domain

**LR = 2e-4**

|                                                                                        Aggregated Mean Change (Python)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Python)                                                                                     |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-0df559835f87dd34e6737f5801fd2f4f4dd4fa50%2F4_python_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-aa191c6151a95d64b6880731418df1ba27a6dd7e%2F4_python_abs_mean.gif?alt=media) |

**LR = 1e-3**

|                                                                                        Aggregated Mean Change (Python)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Python)                                                                                     |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-036bae60d6b52b19015650c2ff17333ebdbfc282%2F3_python_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-0c5e414a436f2ab17320d8725fd5192aa1e5be25%2F3_python_abs_mean.gif?alt=media) |

***

### Math Linguistic Domain

**LR = 2e-4**

|                                                                                        Aggregated Mean Change (Math Sol)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Math Sol)                                                                                     |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-ac4e357f9773c02b33dd6e7230a41674afe57b9a%2F4_math_sol_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-1569a5d91bff233cf7937dd0c0a814c531635b80%2F4_math_sol_abs_mean.gif?alt=media) |

**LR = 1e-3**

|                                                                                        Aggregated Mean Change (Math Sol)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Math Sol)                                                                                     |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-282f941b809ac0da80f62f6ed9299ed609f746b6%2F3_math_sol_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-ba93aa6baabc5081fc09685558dfc4f4dd64cbd6%2F3_math_sol_abs_mean.gif?alt=media) |

***

### Math Thinking Domain

**LR = 2e-4**

|                                                                                        Aggregated Mean Change (Math Think)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Math Think)                                                                                     |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-2851ba7c69e6b664c87535140e7a006b84906329%2F4_math_think_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-cbb18dc40b0d745527b98908e90f6fdd7346a9df%2F4_math_think_abs_mean.gif?alt=media) |

**LR = 1e-3**

|                                                                                        Aggregated Mean Change (Math Think)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Math Think)                                                                                     |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-652c14be395960e22149dee70b396990d33ccc22%2F3_math_think_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-f1ac9f4a3538af6c7713342d6a3942f96e514797%2F3_math_think_abs_mean.gif?alt=media) |

***

### Physics Domain

**LR = 2e-4**

|                                                                                        Aggregated Mean Change (Physics)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Physics)                                                                                     |
| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-a014b9acfc58501d45604f87db1618e10a0d3996%2F4_physics_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-dddf940490791aea0d005780d1e66bc5c4e8e4ef%2F4_physics_abs_mean.gif?alt=media) |

**LR = 1e-3**

|                                                                                        Aggregated Mean Change (Physics)                                                                                       |                                                                                     Aggregated Mean Absolute Change (Physics)                                                                                     |
| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-9b2ee84e6016c884081d7db55b92bdf6b99cd6f1%2F3_physics_mean.gif?alt=media) | ![](https://1411245515-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkMHqN4U56XrAewblM60r%2Fuploads%2Fgit-blob-d062764470565ee569368642c03e5244bed2e552%2F3_physics_abs_mean.gif?alt=media) |
