[
{
"id": "apple-sorting-system",
"data": {
"title": "Apple Sorting System",
"subtitle": "An automated hardware-software hybrid system for sorting apples based on surface diseases using Vision Transformers and a Raspberry Pi.",
"date": "2026-09-06T00:00:00.000Z",
"authors": [
{
"name": "Heramb Vengurlekar",
"affiliation": "Thadomal Shahani Engineering College",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"Machine Learning",
"Computer Vision",
"Vision Transformer (ViT)",
"PyTorch",
"Raspberry Pi",
"C++",
"IoT"
],
"codeUrl": "https://github.com/HerambVE/apple_sorting_system",
"image": {
"src": "/_astro/brand-icon.DEAsHZtE.png",
"width": 618,
"height": 618,
"format": "png"
},
"draft": false
},
"body": "## Overview\n\nAn automated hardware-software hybrid system for sorting apples based on surface diseases. The project uses a physical conveyor belt powered by a Raspberry Pi, which orchestrates sensors, servos, and cameras to classify and mechanically separate apples into four distinct categories: Normal, Blotch, Rot, and Scab.\n\n## 🚀 Features\n\n- **Real-Time Sorting**: Physically routes apples on a conveyor belt using servo-actuated gates.\n- **Dual-Camera Vision**: Captures two angles (top and side) using a CSI camera and a USB webcam simultaneously for higher classification accuracy.\n- **Cloud AI Processing**: Uses a Vision Transformer (ViT) deployed via ngrok on Google Colab for powerful, off-device inference.\n- **Hardware Integration**: Full C++ implementation using `pigpio` for non-blocking hardware control on the Raspberry Pi.\n- **Safety / Rejection**: Confidence thresholding to reject non-apple objects.\n\n## 🏗 System Architecture\n\nThe system operates using an edge-to-cloud architecture:\n1. **Detection**: An IR sensor detects an apple on the conveyor belt.\n2. **Capture**: The Raspberry Pi stops the belt and captures two images.\n3. **Inference**: Images are sent via `libcurl` to the Colab API.\n4. **Action**: The API returns a classification. The Raspberry Pi actuates the corresponding servo and restarts the belt.\n\n## 📦 Classification Categories\n\nThe Vision Transformer model is trained to recognize four classes:\n- 🟢 **Normal**: Healthy apples (continue on belt).\n- 🟤 **Blotch**: Apples with blotch disease (Servo 1).\n- ⚫ **Rot**: Apples with rot/decay (Servo 2).\n- 🟡 **Scab**: Apples with scab disease (Servo 3).\n\n## 🛠 Contributions\n\n- Designed and developed the **C++ Edge Controller** (`pigpio`, multi-threaded hardware orchestration, non-blocking asynchronous camera triggers, and HTTP client via `libcurl`).\n- Engineered the **Vision Transformer (ViT)** Deep Learning pipeline in PyTorch (`vit_base_patch16_224`), including transfer learning, class weighting, and data augmentations.\n- Built the **Cloud Inference Microservice** (Flask + ngrok) with dual-camera perspective aggregation and defect arbitration logic.",
"filePath": "src/content/projects/apple-sorting-system.mdx",
"assetImports": [
"../../assets/placeholder-image.png"
],
"digest": "07c3058a05f1b6a3",
"deferredRender": true,
"collection": "projects"
},
{
"id": "hospital-patient-footfall-prediction",
"data": {
"title": "Hospital Patient Footfall Predictive Model",
"subtitle": "Forecasting patient admission trends using multivariate regression for hospital resource allocation.",
"date": "2025-11-10T00:00:00.000Z",
"authors": [
{
"name": "Heramb Vengurlekar",
"affiliation": "Thadomal Shahani Engineering College",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"Machine Learning",
"scikit-learn",
"Regression",
"EDA",
"Healthcare Analytics"
],
"codeUrl": "https://github.com/HerambVE/hospital0ftfal",
"image": {
"src": "/_astro/brand-icon.DEAsHZtE.png",
"width": 618,
"height": 618,
"format": "png"
},
"draft": false
},
"body": "## Overview\n\nA data-driven machine learning system developed to forecast hospital admission volumes, enabling healthcare administrators to optimize staffing, bed occupancy, and medical inventory.\n\n## Methodology & Highlights\n\n1. **Exploratory Data Analysis (EDA):** Analyzed multi-year admission patterns, uncovering strong seasonal correlations, day-of-week variances, and anomalous footfall spikes.\n2. **Feature Engineering:** Built lag features, rolling averages, and calendar cyclical encodings.\n3. **Model Training & Evaluation:** Trained multivariate regression models using **scikit-learn** with k-fold cross-validation, optimizing for Mean Squared Error (MSE) and high R² accuracy.",
"filePath": "src/content/projects/hospital-patient-footfall-prediction.mdx",
"assetImports": [
"../../assets/placeholder-image.png"
],
"digest": "9b94cadaa039db31",
"deferredRender": true,
"collection": "projects"
},
{
"id": "expert-emergence-in-moe",
"data": {
"title": "Expert Emergence in a Small Sparse MoE Transformer Trained on Code, Math, and Prose",
"subtitle": "Training a small MoE model on code, math, and prose to observe whether expert routing patterns emerge from domain structure.",
"date": "2026-02-27T00:00:00.000Z",
"authors": [
{
"name": "sumitdotml",
"affiliation": "Independent",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"mixture of experts",
"expert specialization",
"sparse transformers",
"routing",
"load balancing"
],
"codeUrl": "https://github.com/sumitdotml/moe-emergence",
"modelUrl": "https://huggingface.co/sumitdotml/moe-emergence",
"logsUrl": "https://wandb.ai/sumit-ml/moe-emergence",
"image": {
"src": "/_astro/training_curves.BE67ecyC.png",
"width": 3560,
"height": 1460,
"format": "png"
},
"draft": true
},
"body": "import Cite from \"../../components/research/Cite.astro\";\nimport Ref from \"../../components/research/Ref.astro\";\n\n## TL;DR\n\nI trained a GPT-2-based MoE model on three domains (code, math, prose) to see whether experts develop routing preferences, and they do! The MoE model beats a dense baseline by 3.6% on aggregate eval loss, with math seeing the largest gain (14%) and prose the only regression (1.6%). Removing the load balancing loss causes full expert collapse within 500 training steps. Top-2 routing provides a negligible 0.14% improvement over top-1, confirming that a single active expert per token captures most of the benefit.\n\n## 1. Motivation\n\nMixture-of-Experts (MoE) models route each token to a subset of \"expert\" feed-forward networks instead of passing everything through one shared FFN. Different experts learn different things, and the router learns which expert to call for which input. This goes back to <Cite id=\"jacobs1991\" authors=\"Jacobs et al.\" year={1991} /> and has been scaled up in modern Transformers by <Cite id=\"shazeer2017\" authors=\"Shazeer et al.\" year={2017} /> and <Cite id=\"fedus2022\" authors=\"Fedus et al.\" year={2022} />.\n\nI wanted to see this happen from scratch, at small scale, with interpretable domains. If I train a GPT-2-sized MoE on code, math, and prose, will the experts specialize by domain? Will expert 3 end up handling math while expert 7 handles code? Or will the routing stay diffuse?\n\nThis project is pedagogical, not competitive. The goal is a trained sparse Transformer with visualizations showing routing patterns, entropy curves, and expert affinity heatmaps. Not SOTA perplexity.\n\n## 2. Architecture\n\n### Base Model\n\nI started from GPT-2 small <Cite id=\"radford2019\" authors=\"Radford et al.\" year={2019} /> (124M parameters, 12 Transformer layers). The last 4 layers (8 through 11) have their feed-forward networks replaced with MoE layers, each containing 8 experts with top-1 routing. The earlier layers (0 through 7) stay dense because they handle general linguistic features that don't need domain-specific routing.\n\n### MoE Layer Design\n\nEach MoE layer replaces the original `MLP` block. I used a learned linear projection from the hidden dimension to the number of experts as the router:\n\n```python\n# router: [batch * seq_len, d_model] -> [batch * seq_len, n_experts]\nrouter_logits = self.router(hidden_states) # shape: [B*S, 8]\nrouter_probs = F.softmax(router_logits, dim=-1)\ntopk_weights, topk_indices = torch.topk(router_probs, k=1, dim=-1)\n```\n\nFor top-1 routing, each token is dispatched to exactly one expert. The forward pass uses the hard assignment (weight = 1.0), but gradients flow through the soft probability via a straight-through estimator <Cite id=\"bengio2013\" authors=\"Bengio et al.\" year={2013} />. This lets the router learn from the loss signal even though the forward is discrete.\n\n### Expert Initialization\n\nI initialized each expert as a `deepcopy` of the original pretrained MLP, plus a small amount of Gaussian noise (`std=0.01`). The model begins training with 8 copies of a working FFN rather than 8 random networks, which avoids the cold-start problem of learning from scratch. The noise breaks symmetry so experts can diverge during training.\n\n### Auxiliary Losses\n\nTwo auxiliary losses prevent degenerate routing:\n\n1. **Load balancing loss** <Cite id=\"fedus2022\" authors=\"Fedus et al.\" year={2022} />: penalizes uneven expert utilization. Defined as `n_experts * sum(f_i * P_i)` where `f_i` is the fraction of tokens routed to expert `i` and `P_i` is the mean router probability for expert `i`. The minimum value is 1.0 at perfect balance. Coefficient: `lb_coef = 0.01`.\n\n2. **Z-loss** <Cite id=\"zoph2022\" authors=\"Zoph et al.\" year={2022} />: stabilizes router logit magnitudes. Defined as `mean(logsumexp(router_logits)^2)`. Prevents the router from producing extremely confident (or noisy) predictions. Coefficient: `z_coef = 0.001`.\n\nThe total loss is: `LM_loss + lb_coef * LB_loss + z_coef * Z_loss`.\n\n\n## 3. Training Setup\n\n### Data\n\nThree domains, each capped at 10MB of raw text:\n\n| Domain | Source | Description |\n|--------|--------|-------------|\n| Code | CodeParrot-clean | Diverse Python files |\n| Math | MathQA <Cite id=\"amini2019\" authors=\"Amini et al.\" year={2019} /> | Problem + rationale pairs |\n| Prose | C4 <Cite id=\"raffel2020\" authors=\"Raffel et al.\" year={2020} /> | Filtered web text |\n\nTexts are tokenized with the GPT-2 tokenizer, packed into 512-token blocks with domain labels, and token-balanced so each domain contributes equally to training batches. Without balancing, code produces 1.9x more tokens than prose, enough to bias expert routing toward code patterns.\n\nAfter balancing: 4,296 blocks per domain, 12,888 total (~6.6M tokens). At 10,000 steps with an effective batch size of 8, the model sees ~6 epochs of the data.\n\n### Runs\n\n| Run | Mode | Steps | Key Change |\n|-----|------|-------|------------|\n| Dense baseline | Dense GPT-2 | 5,000 | No MoE layers |\n| MoE main | Top-1, 8 experts | 10,000 | Primary experiment |\n| No-LB ablation | Top-1, lb_coef=0 | 2,000 (early-stopped at 500) | Tests necessity of load balancing |\n| Top-2 directional | Top-2, 8 experts | 10,000 | Tests whether a second expert helps |\n\nAll runs use `lr=5e-5`, cosine schedule, 10% warmup, `batch_size=2` with `grad_accum=4` (effective batch 8). All training ran on a single [PrimeIntellect](https://www.primeintellect.ai/) RTX 4090 24GB instance.\n\n\n## 4. Results\n\n### Dense vs MoE\n\n\n\n| Metric | Dense | MoE (top-1) | Delta |\n|--------|-------|-------------|-------|\n| Eval loss | 2.157 | 2.080 | -3.6% |\n| Code loss | 1.554 | 1.521 | -2.1% |\n| Math loss | 2.023 | 1.740 | -14.0% |\n| Prose loss | 3.485 | 3.541 | +1.6% |\n| Perplexity | 8.64 | 7.91 | -8.4% |\n\nThe MoE model wins on aggregate loss by 3.6%. Math sees the largest improvement at 14% because its structured problem-rationale format gives experts something concrete to specialize on. Code improves modestly at 2.1%. Prose is the one domain where MoE loses: the dense model beats it by 1.6%.\n\nC4 web text is heterogeneous: news, forums, recipes, product descriptions, everything. There's no single \"prose pattern\" for an expert to latch onto. Math and code have learnable structural regularities; prose, at this scale, does not.\n\n> Note: These runs are not step-matched (5,000 dense vs 10,000 MoE), but the dense model had plateaued by step ~3,600, so additional steps would not have closed the gap.\n\n### Training Dynamics\n\nThe MoE model crossed the dense baseline at step ~3,600 (36% of training). Both models plateaued around step 7,000-8,000, with the MoE eval loss settling at 2.080 and the dense at 2.157. The learning rate had decayed to near-zero by that point (cosine schedule), and the small dataset was fully memorized.\n\nLoad balance stayed healthy throughout the MoE main run: `lb_loss` started at 1.056 during warmup and settled to 1.011 by step 1,000, holding there for the remaining 9,000 steps. Z-loss converged from 4.3 at step 200 down to 0.99 by step 8,000, indicating the router logits stabilized completely.\n\n\n## 5. Expert Specialization\n\n### Domain-Level Routing\n\nThe central question: do the experts develop domain preferences?\n\n\n\nYes. The heatmaps show clear specialization patterns, especially in layers 8 and 10. In layer 10, expert 6 receives 0.260 of math tokens (vs the 0.125 uniform baseline), while getting only 0.075 of prose tokens. In layer 8, expert 4 gets 0.242 of math tokens but only 0.078 of code. Layer 8 shows some of the sharpest domain contrasts, while layer 11 stays more diffuse.\n\nThe routing is not perfectly disjoint (no expert handles only one domain), but the preferences are unambiguous. This is what you'd expect: the load balancing loss prevents full monopolization, so experts develop preferences rather than exclusive territories.\n\n### Router Entropy\n\n\n\nRouter entropy starts near the maximum (`ln(8) = 2.079`, representing uniform routing) and decreases over training as the router learns to make sharper decisions, moving from \"send tokens everywhere equally\" to \"send this token to expert 4 specifically.\" The decline is steepest in the first 2,000 steps and plateaus by step 5,000, mirroring the eval loss convergence.\n\nLayers 9 and 10 show the lowest final entropy, meaning the router makes the sharpest per-token decisions in those layers. Layer 8 retains higher entropy: its router distributes probability more evenly across experts for any given token, even though the aggregate domain-level preferences are strong (as the heatmaps show). Per-token confidence and domain-level preference measure different things.\n\n### Token-Level Routing\n\nBeyond domain-level patterns, I analyzed how the router handles different Python token types within the code domain.\n\n\n\nThe router distinguishes between token types within the same domain. In layer 9, expert 0 handles 0.471 of NUMBER tokens and 0.429 of OP tokens, but only 0.125 of KEYWORDs. Expert 2 handles 0.447 of NAME tokens. The experts develop finer-grained preferences for syntactic roles, not just a coarse \"this is code\" distinction.\n\nThe analysis uses majority-vote mapping from BPE token spans to Python tokenizer categories, running 200 code samples through the final model and accumulating expert assignments per token type.\n\n\n## 6. Ablations\n\n### Without Load Balancing\n\nThe no-LB ablation removes the load balancing loss (`lb_coef = 0.0`) while keeping z-loss on (`z_coef = 0.001`). Is load balancing necessary, or will the router distribute tokens naturally?\n\n\n\nExpert collapse happened fast. The first warning fired at step 100 (expert 5 in layer 11 handling 62.5% of tokens). By step 500, expert 1 in layer 9 had captured 73.6% of all tokens, triggering automatic early stopping.\n\n\n\nThe trajectory shows a classic collapse pattern: one expert begins with a slight advantage, the router reinforces that advantage by sending it more tokens, and the positive feedback loop accelerates until a single expert monopolizes the layer.\n\nThe no-LB model actually had *lower* LM loss than the MoE main run at step 400 (2.378 vs 2.493). Collapsing onto fewer experts can temporarily help optimization because the router doesn't need to learn distributed specialization. But this comes at the cost of dead experts and lost capacity.\n\nZ-loss alone does not prevent collapse. It stabilizes the *magnitude* of router logits but has no mechanism to encourage balanced *distribution*. Of the two auxiliary losses, only lb_loss prevents collapse.\n\n### Top-2 vs Top-1\n\nThe top-2 run routes each token to two experts with soft-weighted combination, doubling per-token expert compute. The main hypothesis: maybe two experts would help on prose, where top-1 routing struggles.\n\n| Metric | Top-1 | Top-2 | Delta |\n|--------|-------|-------|-------|\n| Eval loss | 2.080 | 2.077 | -0.14% |\n| Code loss | 1.521 | 1.518 | -0.19% |\n| Math loss | 1.740 | 1.734 | -0.39% |\n| Prose loss | 3.541 | 3.540 | -0.03% |\n| lb_loss | 1.012 | 1.048 | +3.6% |\n\nThe improvement is 0.14%, consistent across 50 eval checkpoints but practically negligible. Math sees the largest (still tiny) gain at 0.39%. Prose barely moved at 0.03%.\n\nThis aligns with the Switch Transformer finding <Cite id=\"fedus2022\" authors=\"Fedus et al.\" year={2022} />: top-1 routing captures most of the MoE benefit. The additional expert path adds compute without meaningfully improving expressiveness at this scale.\n\n## 7. Limitations\n\n**1.** GPT-2 small with 8 experts is far from production MoE systems like Mixtral <Cite id=\"jiang2024\" authors=\"Jiang et al.\" year={2024} /> or Switch-C. Specialization patterns at this scale may not transfer to models with hundreds of experts.\n\n**2.** Code, math, and prose are intentionally distinct. A harder test would be closely related domains (e.g., Python vs JavaScript, or algebra vs geometry) where the router needs to make finer distinctions.\n\n**3.** C4 was the wrong dataset for demonstrating prose specialization. I chose it over WikiText-103 to avoid encyclopedic bias ([decision 010](https://github.com/sumitdotml/moe-emergence/blob/main/docs/decisions/010-dataset-choices.md)), but C4's topic distribution (47% general web content, 14.5% commerce, 12% tech, and dozens of smaller categories) is too diffuse for any expert to specialize on. This showed up immediately in training: prose loss barely moved after step 1,000 and the MoE model never caught the dense baseline on this domain. A narrower corpus with structural regularity (literary fiction, scientific abstracts, news articles) would have given the router patterns to exploit, the way MathQA's problem-rationale format did for math.\n\n**4.** The dense baseline ran for 5,000 steps; MoE runs went to 10,000. The dense model had plateaued, but a perfectly controlled comparison would use identical step counts.\n\n**5.** 10MB per domain, ~6 epochs of training. Larger datasets might reveal different specialization patterns or extend the region where MoE has an advantage.\n\n**6.** Production MoE systems use a capacity factor to limit how many tokens each expert handles per batch, preventing overflow. I skipped this since the dataset is small enough that overflow wasn't a practical issue.\n\n## 8. Conclusion\n\nExpert specialization emerges reliably at small scale when the training data has domain structure. The MoE model's 14% improvement on math shows that structured, pattern-rich domains benefit most from expert routing. Prose -- heterogeneous web text with no dominant structure -- doesn't benefit, and actually regresses slightly.\n\nThe load balancing loss is essential. Without it, expert collapse happens within hundreds of steps, and z-loss alone doesn't help. Top-1 routing is sufficient: a second expert per token doubles the compute without moving the loss.\n\nAll training artifacts (checkpoints, metrics, W&B logs) and the analysis code are available in the [project repository](https://github.com/sumitdotml/moe-emergence). Models are on [HuggingFace](https://huggingface.co/sumitdotml/moe-emergence).\n\n\n## References\n\n<Ref id=\"jacobs1991\"\n authors=\"Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E.\"\n year={1991}\n title=\"Adaptive mixtures of local experts\"\n venue=\"Neural Computation\"\n volume=\"3(1)\"\n pages=\"79-87\"\n url=\"https://doi.org/10.1162/neco.1991.3.1.79\" />\n\n<Ref id=\"shazeer2017\"\n authors=\"Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J.\"\n year={2017}\n title=\"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer\"\n arxiv=\"1701.06538\" />\n\n<Ref id=\"fedus2022\"\n authors=\"Fedus, W., Zoph, B., and Shazeer, N.\"\n year={2022}\n title=\"Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity\"\n venue=\"Journal of Machine Learning Research\"\n volume=\"23(120)\"\n pages=\"1-39\"\n url=\"https://jmlr.org/papers/v23/21-0998.html\" />\n\n<Ref id=\"zoph2022\"\n authors=\"Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W.\"\n year={2022}\n title=\"ST-MoE: Designing stable and transferable sparse expert models\"\n arxiv=\"2202.08906\" />\n\n<Ref id=\"jiang2024\"\n authors=\"Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lenber, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Le Scao, T., Gerber, T., Vayer, T., Roux, T., Lavril, T., and El Sayed, W.\"\n year={2024}\n title=\"Mixtral of Experts\"\n arxiv=\"2401.04088\" />\n\n<Ref id=\"radford2019\"\n authors=\"Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.\"\n year={2019}\n title=\"Language models are unsupervised multitask learners\"\n url=\"https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf\" />\n\n<Ref id=\"bengio2013\"\n authors=\"Bengio, Y., Léonard, N., and Courville, A.\"\n year={2013}\n title=\"Estimating or propagating gradients through stochastic neurons for conditional computation\"\n arxiv=\"1308.3432\" />\n\n<Ref id=\"amini2019\"\n authors=\"Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H.\"\n year={2019}\n title=\"MathQA: Towards interpretable math word problem solving with operation-based formalisms\"\n venue=\"NAACL\"\n url=\"https://aclanthology.org/N19-1245/\" />\n\n<Ref id=\"raffel2020\"\n authors=\"Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narain, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.\"\n year={2020}\n title=\"Exploring the limits of transfer learning with a unified text-to-text transformer\"\n venue=\"Journal of Machine Learning Research\"\n volume=\"21(140)\"\n pages=\"1-67\"\n url=\"https://jmlr.org/papers/v21/20-074.html\" />\n\n\n## Appendix: Design Decisions\n\nThree design choices shaped this project. Full decision records with alternatives considered are in the [project repository](https://github.com/sumitdotml/moe-emergence/tree/main/docs/decisions).\n\n**GPT-2 as base model.** The goal is observing MoE routing behavior, not pushing performance. GPT-2 small is thoroughly documented, has stable HuggingFace integration, and fits within an $80 GPU budget. Its simplicity (no RoPE, no GQA, standard LayerNorm) means any observed effects can be attributed to the MoE routing rather than architectural quirks. More modern alternatives (Pythia-160M, custom Llama-style) were considered but added complexity without clear benefit for a pedagogical study.\n\n**MoE in the last 4 layers.** Early Transformer layers tend to learn generic features (syntax, common patterns), while later layers learn more semantic, domain-specific features. Replacing all 12 layers with MoE would cost ~3x the compute and make it harder to interpret whether specialization reflects domain structure or just low-level token statistics. Layers 8-11 are where domain-specific routing is most meaningful, and the compute savings left budget for ablations.\n\n**Warm-start via deepcopy.** Each expert starts as an exact copy of the pretrained MLP, plus tiny Gaussian noise (`std = param.std() * 1e-3`) for symmetry breaking. This means the model works identically at step 0 and experts diverge from a known baseline. Random initialization would break pretrained representations and confound \"learning to specialize\" with \"learning to function at all.\" The noise scale matters: using `param.norm()` instead of `param.std()` would corrupt weights because the norm sums over millions of elements.",
"filePath": "src/content/projects/expert-emergence-in-moe.mdx",
"assetImports": [
"../../assets/research/expert-emergence-in-moe/training_curves.png"
],
"digest": "962467cd7dc1371d",
"deferredRender": true,
"collection": "projects"
},
{
"id": "lora-and-friends",
"data": {
"title": " Is Attention Enough? A LoRA Target-Module Study on Qwen3-8B Math SFT",
"subtitle": "A controlled comparison of attention-only and all-layer LoRA on Qwen3-8B using an OpenMathInstruct-2-derived math SFT dataset and GSM8K evaluation.",
"date": "2026-05-17T00:00:00.000Z",
"authors": [
{
"name": "sumitdotml",
"affiliation": "Independent",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"LoRA",
"Qwen3-8B",
"GSM8K",
"math fine-tuning",
"target modules"
],
"codeUrl": "https://github.com/sumitdotml/lora-and-friends",
"modelUrl": "https://huggingface.co/sumitdotml/lora-and-friends",
"image": {
"src": "/_astro/fig_04_paired_seed_slope.ClcugjqE.png",
"width": 2511,
"height": 1253,
"format": "png"
},
"draft": true
},
"body": "import Cite from \"../../components/research/Cite.astro\";\nimport Ref from \"../../components/research/Ref.astro\";\n\n## TL;DR\n\nThis experiment fine-tuned `Qwen3-8B` <Cite id=\"qwen3\" authors=\"Yang et al.\" year={2025} /> on a math SFT dataset derived from `nvidia/OpenMathInstruct-2` <Cite id=\"openmathinstruct2\" authors=\"Toshniwal et al.\" year={2024} /> and compared two LoRA <Cite id=\"lora\" authors=\"Hu et al.\" year={2021} /> adapter scopes on GSM8K <Cite id=\"gsm8k\" authors=\"Cobbe et al.\" year={2021} />. Attention-only LoRA reached mean GSM8K accuracy `0.9055` across `N=3` seeds, while all-layer LoRA reached `0.9009`. The `0.455` percentage-point gap favors attention-only in this run set, but it is below the frozen `0.01` accuracy threshold for a winner claim that I had set in advance. I read this as a local result rather than a general rule, and as motivation for studying when narrower adapters are enough versus when broader adaptation is worth the extra size.\n\n**Published artifacts.**\n\n| Artifact | Location |\n| ------------------------------------ | ------------------------------------------------------------------------------------------------------ |\n| Experiment code and source artifacts | [GitHub repository](https://github.com/sumitdotml/lora-and-friends) |\n| Six selected LoRA adapter exports | [HuggingFace model repo](https://huggingface.co/sumitdotml/lora-and-friends) |\n| Frozen raw and rendered SFT dataset | [HuggingFace dataset repo](https://huggingface.co/datasets/sumitdotml/lora-and-friends-dataset) |\n| Third-party notices | [GitHub notices file](https://github.com/sumitdotml/lora-and-friends/blob/main/THIRD_PARTY_NOTICES.md) |\n\n## 1. Motivation\n\nLoRA adapts a frozen model by training low-rank updates rather than updating the full parameter set. The method is often treated as a single straightforward switch: you choose a rank, then a learning rate, and then decide which modules receive adapters. I did this study to isolate the last choice under one narrow setup.\n\nThis study was also motivated by Thinking Machines Lab's _LoRA Without Regret_ <Cite id=\"lora-without-regret\" authors=\"Schulman and Thinking Machines Lab\" year={2025} />, especially its discussion of where LoRA is applied and when broader adaptation may be worth the extra capacity.\n\nThe practical question was whether adding MLP adapters helps a small math SFT run on `Qwen3-8B`, or whether attention-only adapters are enough. I intentionally kept the scope small: one base model, one dataset recipe, one benchmark, rank `8`, and three seeds per LoRA condition.\n\n## 2. Experimental Question\n\nThe comparison was:\n\n- `attention_only`: train adapters for `q_proj`, `k_proj`, `v_proj`, and `o_proj`.\n- `all_layer`: train those attention adapters plus `gate_proj`, `up_proj`, and `down_proj`.\n\nBoth conditions used the same rendered dataset, base model, rank, selected peak learning rate, training schedule, checkpoint-selection rule, and GSM8K eval contract so that I could draw comparisons in the most unbiased way possible. The intended variable was adapter scope.\n\nThe frozen interpretation rule bounds how that mean gap can be read. A condition mean gap below `0.01` accuracy was to be reported as inconclusive rather than as a winner.\n\n## 3. Dataset And Evaluation Contract\n\n### 3.1 Dataset curation\n\nThe retained raw dataset is `openmath_original_clean`, built from the `train_1M` split of `nvidia/OpenMathInstruct-2`. It is the second candidate the project produced. An earlier candidate, `openmath_30k`, drew heavily on the augmented sources of the same split to hit a balanced 30,000-row target. Manual review packs on that candidate kept surfacing the same family of defects in the augmented rows, including impossible integer counts in physical settings, prompt-generation residue such as the literal string `A new problem:`, and chains of reasoning that silently restated the problem to match the boxed answer. After my repeated repair passes did not bring the rejection rate to zero, I decided to drop the augmented branch as I concluded that it was noisy & unreliable and thus rebuilt the dataset again from original-source rows only.\n\nThe retained build keeps rows from the `gsm8k` and `math` sources of `train_1M`. The strict acceptance gate considered `14,764` `gsm8k` candidates and `14,704` `math` candidates, for `29,468` original-source candidate rows. It accepted `14,618` `gsm8k` rows and `13,548` `math` rows, for `28,166` accepted rows. The retained reject reasons in the dataset manifest are `1,205` boxed-answer mismatches and `97` suspicious patterns across the two sources. After the [train/validation split repair](#32-trainvalidation-split-rule), the final raw split has `25,348` train rows and `2,818` validation rows.\n\nAutomated checks on the retained build returned a boxed-match rate of `1.0` on both splits, no remaining suspicious rows, and no prompt-wrapper contamination hits. The same retained dataset is the input to every result in [Results](#5-results) and every diagnostic in [Diagnostics](#6-diagnostics).\n\n### 3.2 Train/validation split rule\n\n`OpenMathInstruct-2` can include multiple accepted solutions for the same underlying problem. A row-level random split therefore tends to place answer-variants of the same problem in both train and validation, which inflates validation likelihood and weakens the checkpoint-selection signal. The retained build groups rows by canonical problem text before assigning them to a split, so all variants of a given problem land on the same side.\n\nAfter the regrouping, the contamination report recorded `row_id_overlap_count = 0` and `problem_text_overlap_count = 0` between train and validation. The benchmark gate from [Contamination and overlap gates](#34-contamination-and-overlap-gates) was applied to the same retained split.\n\n### 3.3 Rendered training contract\n\nThe rendered training dataset uses the `Qwen/Qwen3-8B` chat template with `enable_thinking=False` and the fixed system prompt:\n\n```text\nYou are a careful math solver. Solve the problem step by step. Put the final answer in \\boxed{}.\n```\n\nThe system prompt was kept because raw `OpenMathInstruct-2` problems do not always carry the boxed-answer instruction, and because the same prompt is required by the [GSM8K evaluation contract](#35-gsm8k-evaluation-contract). On a `500`-row tokenizer sample, the prompt added a constant `27` rendered tokens per example, which was small enough not to reshape the budget.\n\n### 3.4 Contamination and overlap gates\n\nThe retained contamination report passed two gates:\n\n| Check | Result |\n| --------------------------------------------------------- | -----: |\n| GSM8K test overlap against training-side `gsm8k` problems | `0` |\n| Train/validation row-id overlap | `0` |\n| Train/validation problem-text overlap | `0` |\n\nThe GSM8K overlap check used the `openai/gsm8k` `main` test split and compared the training-side `problem` field for rows where `source == \"gsm8k\"`, after Unicode-NFKC, lowercase, and whitespace-collapsed normalization. A non-zero result on either gate was treated as blocking under the failure rule in the frozen evaluation contract; both gates had to pass before any benchmark run.\n\n### 3.5 GSM8K evaluation contract\n\nGSM8K evaluation used the `1,319`-example test split. Decoding was greedy with temperature `0`, `max_new_tokens=512`, no fallback answer extractor, and exact match after boxed-answer extraction and normalization. If a model did not emit a parseable boxed answer, the example was scored incorrect.\n\n## 4. Training Setup\n\n### 4.1 Base model and render alignment\n\nThe base model I decided to go with was `Qwen/Qwen3-8B`. Qwen3 supports both thinking and non-thinking modes in its chat template; however, I decided to disable thinking mode for the rendered SFT dataset and for evaluation so the training and evaluation would be consistent with matching contracts.\n\nThe training-side render came from `AutoTokenizer.apply_chat_template(..., enable_thinking=False)` applied to each message list, which emits an empty `<think>\\n\\n</think>` block before the assistant answer. The Tinker cookbook `qwen3_disable_thinking` renderer was correct for generation prompts but did not reproduce that SFT render along the supervised-training path. The runner therefore builds Tinker training datums directly from the HuggingFace chat template and masks the loss to the assistant-answer tokens after the rendered prompt prefix. With this correction, the training render and the GSM8K evaluation render produce the same prompt structure.\n\n### 4.2 Adapter and optimizer defaults\n\nBoth LoRA conditions used Tinker with rank `r=8`, `train_unembed=false`, and peak learning rate `3e-4`. Tinker `0.18.2` did not expose local `lora_alpha` or `lora_dropout` fields through this training path. The PEFT adapter configs now published on HuggingFace record the exported adapter values as `lora_alpha=32` and `lora_dropout=0`, so I will assume that those were being applied through the Tinker API's backend. Additionally, the Adam internals (`beta1=0.9`, `beta2=0.95`, `eps=1e-12`, `weight_decay=0`, `grad_clip_norm=0`) are inherited from the Tinker `0.18.2` `AdamParams` defaults and are not tuned in this study.\n\n### 4.3 LR selection\n\nThe small LR-selection run used seed `7`, `512` train rows, `128` validation rows, and the grid `1e-4`, `3e-4`, `1e-3`. Both conditions selected `3e-4` by minimum validation mean negative log-likelihood (NLL). Seed `7` was reserved for selection work so the main comparison seeds could be `0`, `1`, and `2`.\n\n### 4.4 Main run schedule\n\nMain training used seeds `0`, `1`, and `2` for each condition:\n\n| Setting | Value |\n| -------------------- | -------------------------------------------------------------- |\n| Train rows | `25,348` |\n| Validation rows | `2,818` |\n| Epochs | `2` |\n| Effective batch size | `8` |\n| Optimizer steps | `6,338` |\n| Request shape | `batched_datums_pipelined` |\n| Warmup steps | `190` |\n| Schedule | linear warmup, cosine decay |\n| Min LR | `3e-5` |\n| Checkpoint rule | lowest validation mean NLL; exact ties choose later checkpoint |\n\nEvery main run selected the step-`3169` checkpoint by validation NLL, which was the end of the first epoch and not the final training step.\n\n### 4.5 Checkpoint export\n\nSelected training checkpoints were written under Tinker's training namespace as `weights/<run-id>-step-3169`. Before any GSM8K evaluation or HuggingFace upload, each selected checkpoint was loaded via `load_state_async(...)` and re-emitted via `save_weights_for_sampler_async(...)` to produce a sampler-format weight set as the sampler form was required by the evaluation sampling client. The six adapters published in the [HuggingFace model repo](https://huggingface.co/sumitdotml/lora-and-friends) are these sampler-converted exports, and the GSM8K numbers in [Results](#5-results) were measured against them rather than against training-state weights.\n\n## 5. Results\n\n**Figure 1. Primary GSM8K comparison.**\n\n| condition | target_modules | adapter_size_mb | seeds | mean_accuracy | min_accuracy | max_accuracy | delta_vs_baseline_pp | extraction_failures | eval_tokens |\n| -------------- | ------------------------------------------------------------- | --------------: | ----: | ------------: | -----------: | -----------: | -------------------: | ------------------: | ----------: |\n| baseline | n/a | n/a | 1 | 0.8453 | 0.8453 | 0.8453 | +0.000 | 31 | 505694 |\n| attention_only | q_proj, k_proj, v_proj, o_proj | 29.4 | 3 | 0.9055 | 0.9045 | 0.9067 | +6.015 | 13 | 1077382 |\n| all_layer | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | 83.5 | 3 | 0.9009 | 0.8992 | 0.9022 | +5.560 | 17 | 1069860 |\n\nThe untouched baseline scored `1115/1319`, or `0.8453373768006065`. Attention-only LoRA scored `1193/1319`, `1196/1319`, and `1194/1319` across seeds `0`, `1`, and `2` respectively. Meanwhile, all-layer LoRA scored `1186/1319`, `1190/1319`, and `1189/1319` across the respective seeds.\n\n\n\n_Figure 2. GSM8K accuracy by condition on the 1,319-example test set. The large markers show condition means; small markers show individual seeds. No statistical confidence interval is shown._\n\nThe attention-only range, `0.9045` to `0.9067`, sits above the all-layer range, `0.8992` to `0.9022`, in this retained run set. This non-overlap is descriptive evidence and does not remove the `N=3` limitation or the frozen inconclusive-band rule.\n\n## 6. Diagnostics\n\nValidation NLL and GSM8K accuracy did not measure the same thing; validation NLL selected checkpoints from the SFT validation set, while GSM8K accuracy measured exact boxed-answer correctness on a held-out benchmark.\n\n\n\n_Figure 3. Validation negative log-likelihood (NLL) over training. Both conditions selected step `3169`; attention-only had mean validation NLL `0.336295` at that step, and all-layer had `0.336339`._\n\nAll six main runs selected the step-`3169` checkpoint under the lowest-mean-validation-NLL rule. Step `3169` is the end of the first epoch under the frozen schedule. The per-seed selected NLLs were `0.336164`, `0.336212`, and `0.336510` for `attention_only` seeds `0`, `1`, and `2`, and `0.336119`, `0.336556`, and `0.336341` for `all_layer` seeds `0`, `1`, and `2`.\n\nAfter step `3169`, attention-only validation NLL stayed close to its minimum, while all-layer validation NLL rose more sharply. At step `4000`, the attention-only condition mean was `0.338719`, while the all-layer condition mean was `0.345773`. That divergence is consistent with the higher-capacity adapter overfitting faster on this dataset size, but this experiment treats it as a diagnostic shape rather than as proof of a mechanism.\n\n\n\n_Figure 4. LR-selection validation NLL on the small slice. Both conditions selected `3e-4`; attention-only was flatter between `3e-4` and `1e-3`, while all-layer rose more at `1e-3`._\n\nThe LR-selection sweep that was conducted before the main training run also hinted that the higher-capacity adapter was more sensitive to LR in this small slice. Attention-only moved from `0.3632619345728878` at `3e-4` to `0.3644437038722405` at `1e-3`. All-layer moved from `0.3559855057286731` at `3e-4` to `0.37397296784836664` at `1e-3`.\n\n\n\n_Figure 5. Per-seed disagreement on GSM8K. Each panel counts examples where attention-only is correct and all-layer is wrong, and the reverse._\n\nThe disagreement view shows that the mean gap did not come from one seed alone. Attention-only won more one-sided disagreements in each paired seed: `+7`, `+6`, and `+5` examples for seeds `0`, `1`, and `2`.\n\n## 7. Cost And Operational Notes\n\n**Figure 6. Token and adapter-size accounting.**\n\n| condition | adapter_size_mb | train_tokens_per_run | validation_tokens_per_run | eval_tokens_per_run | mean_accuracy | extraction_failures |\n| -------------- | --------------: | -------------------: | ------------------------: | ------------------: | ------------: | ------------------: |\n| baseline | n/a | n/a | n/a | 505,694 | 0.8453 | 31 |\n| attention_only | 29.4 | 17,227,430 | 6,556,081 | 359,127 | 0.9055 | 13 |\n| all_layer | 83.5 | 17,227,430 | 6,556,081 | 356,620 | 0.9009 | 17 |\n\nThe training-side token counts were identical across conditions because the dataset and schedule were fixed. The `adapter_size_mb` values are retained rounded reporting values. On the HuggingFace model repo main branch, the attention-only `adapter_model.safetensors` files are `30,709,160` bytes each, and the all-layer files are `87,360,552` bytes each.\n\nI had also conducted a throughput probe experiment, which can be further explored in [Appendix C](#c-throughput-probe). This provides operational context and explains why the main run used `batched_datums_pipelined`, while keeping the same nominal effective batch size of `8`. It does not indicate a model-quality result.\n\n## 8. Limitations\n\nThis is a scoped empirical case study and not a broad LoRA target-module law. Below, as per my understanding of the project, are some of this experiment's limitations:\n\n- GSM8K is a single benchmark. It is useful for grade-school math word problems, but it does not cover all math reasoning behavior.\n- Each LoRA condition has only `N=3` seeds. The observed ranges do not overlap, but that does not support a statistical significance claim here.\n- The setup uses one base model, one dataset recipe, one rank, one backend training path, and one selected LR per condition. Different ranks, datasets, optimizers, models, or answer-extraction contracts could change the result.\n- Tinker owned some backend LoRA details during training. The local runner controlled rank and layer-family switches, but it did not set `lora_alpha` or `lora_dropout`; those values are visible only in the exported adapter configs.\n- The exported HuggingFace adapter configs use PEFT's saved config format. The public directories are organized by the experimental conditions used in this study, while the exported config files themselves use PEFT's generic `target_modules` representation.\n\n## 9. Reproducibility\n\nA companion worklog gives the longer story behind this report, including the early false starts, dataset cleanup, training runs, evaluation, and figures: [LoRA Target-Module Worklog](/blog/lora-target-module-worklog).\n\nThe public HuggingFace artifacts are:\n\n| Purpose | URL |\n| ------------------------------- | ----------------------------------------------------------------------------------------------- |\n| Selected LoRA adapter exports | [HuggingFace model repo](https://huggingface.co/sumitdotml/lora-and-friends) |\n| Frozen raw and rendered dataset | [HuggingFace dataset repo](https://huggingface.co/datasets/sumitdotml/lora-and-friends-dataset) |\n\nThe experiment code and retained source-of-truth artifacts live in the project's [GitHub repository](https://github.com/sumitdotml/lora-and-friends). Third-party dataset, model, and backend notices are tracked in the repository's [third-party notices file](https://github.com/sumitdotml/lora-and-friends/blob/main/THIRD_PARTY_NOTICES.md):\n\n| Purpose | URL |\n| ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Baseline GSM8K result | [`artifacts/results/baseline-qwen3-8b-gsm8k-001/summary.json`](https://github.com/sumitdotml/lora-and-friends/blob/main/artifacts/results/baseline-qwen3-8b-gsm8k-001/summary.json) |\n| Main training runs (six `main-001-<condition>-seed-<seed>/` directories) | [`artifacts/results/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/results) |\n| Selected-checkpoint GSM8K evals (six `checkpoint-...-gsm8k-...` directories) | [`artifacts/results/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/results) |\n| Dataset manifest | [`artifacts/raw_datasets/openmath_original_clean/manifest.json`](https://github.com/sumitdotml/lora-and-friends/blob/main/artifacts/raw_datasets/openmath_original_clean/manifest.json) |\n| Rendered dataset directory | [`artifacts/rendered_datasets/openmath_original_clean_qwen3_disable_thinking/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/rendered_datasets/openmath_original_clean_qwen3_disable_thinking) |\n| Contamination report | [`artifacts/audits/contamination_check/report.json`](https://github.com/sumitdotml/lora-and-friends/blob/main/artifacts/audits/contamination_check/report.json) |\n| Frozen eval contract | [`docs/freeze/eval_contract.md`](https://github.com/sumitdotml/lora-and-friends/blob/main/docs/freeze/eval_contract.md) |\n| Frozen LoRA defaults | [`docs/freeze/lora_defaults.md`](https://github.com/sumitdotml/lora-and-friends/blob/main/docs/freeze/lora_defaults.md) |\n| Frozen run protocol | [`docs/freeze/run_protocol.md`](https://github.com/sumitdotml/lora-and-friends/blob/main/docs/freeze/run_protocol.md) |\n| Figure provenance (`fig_*/fig_*.provenance.json` under each figure directory) | [`artifacts/figures/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/figures) |\n| Main training script | [`training/run_main_training.py`](https://github.com/sumitdotml/lora-and-friends/blob/main/training/run_main_training.py) |\n\nThe HuggingFace model repo main branch contains six selected adapter exports under `checkpoints/best-checkpoints/`, split by condition, seed, and step. The HuggingFace dataset repo contains both the raw rows and the rendered `Qwen3-8B` chat-template rows used for training.\n\nThe retained main-run command is:\n\n```bash\nuv run training/run_main_training.py --run-prefix main-001\n```\n\nThe frozen protocol expanded that command across two conditions and three seeds. The final comparison did not pick the best seed. It evaluated each selected checkpoint on GSM8K and reduced each condition by the mean across the three seeds.\n\n## 10. Conclusion\n\nUnder this exact `Qwen3-8B` math SFT setup, attention-only LoRA produced the higher GSM8K mean and won more paired disagreements at every seed. The gap was small, `0.455` percentage points, and stayed below the prewritten `0.01` winner threshold.\n\nTherefore, I would keep the conclusion local to this experiment. Attention-only LoRA was smaller and slightly ahead here, but that does not mean it will always be the best option. The answer likely changes with the model, the data, and what the fine-tune is meant to improve. This points to a useful follow-up question: when are narrow adapters enough, and when is broader adaptation worth the extra size? I naturally see this as a promising research direction beyond the scope of this first experiment.\n\n## Acknowledgments\n\nI thank Thinking Machines Lab for providing Tinker credits that supported the training and evaluation runs in this experiment.\n\n## References\n\n<Ref id=\"lora\"\n authors=\"Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.\"\n year={2021}\n title=\"LoRA: Low-Rank Adaptation of Large Language Models\"\n url=\"https://arxiv.org/abs/2106.09685\"\n arxiv=\"2106.09685\" />\n\n<Ref id=\"gsm8k\"\n authors=\"Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.\"\n year={2021}\n title=\"Training Verifiers to Solve Math Word Problems\"\n url=\"https://arxiv.org/abs/2110.14168\"\n arxiv=\"2110.14168\" />\n\n<Ref id=\"qwen3\"\n authors=\"Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., and others\"\n year={2025}\n title=\"Qwen3 Technical Report\"\n url=\"https://arxiv.org/abs/2505.09388\"\n arxiv=\"2505.09388\" />\n\n<Ref id=\"openmathinstruct2\"\n authors=\"Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I.\"\n year={2024}\n title=\"OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data\"\n url=\"https://arxiv.org/abs/2410.01560\"\n arxiv=\"2410.01560\" />\n\n<Ref id=\"lora-without-regret\"\n authors=\"Schulman, J., and Thinking Machines Lab\"\n year={2025}\n title=\"LoRA Without Regret\"\n venue=\"Thinking Machines Lab Blog\"\n url=\"https://thinkingmachines.ai/blog/lora/\" />\n\n## Appendix\n\n### A. Full Disagreement Table\n\n| Seed | Attn correct, All-layer wrong | All-layer correct, Attn wrong | Delta (Attn - All) | Both correct | Both wrong |\n| ---: | ----------------------------: | ----------------------------: | -----------------: | -----------: | ---------: |\n| 0 | 54 | 47 | +7 | 1139 | 79 |\n| 1 | 49 | 43 | +6 | 1147 | 80 |\n| 2 | 46 | 41 | +5 | 1148 | 84 |\n\nEach row sums to `1,319` examples.\n\n### B. Disagreement Examples\n\n| Seed | Direction | Benchmark index | Reference | Attention-only extracted answer | All-layer extracted answer |\n| ---: | ---------------------- | --------------: | --------: | ------------------------------: | -------------------------: |\n| 0 | attention-only correct | 18 | 7 | 7 | 21 |\n| 0 | all-layer correct | 2 | 70000 | 195000 | 70000 |\n| 1 | attention-only correct | 13 | 18 | 18 | 15 |\n| 1 | all-layer correct | 39 | 18 | 48 | 18 |\n| 2 | attention-only correct | 5 | 64 | 64 | 50 |\n| 2 | all-layer correct | 40 | 8 | 12 | 8 |\n\nThe lowest-index seed-0 attention-only example asks how many dozens of eggs Claire eats after making a three-egg omelet every morning for four weeks. Attention-only computes `84 / 12 = 7`; all-layer mistakenly treats `84` as omelets and multiplies by `3` again.\n\nThe lowest-index seed-2 all-layer example asks for Brandon's iPhone age given Suzy's one-year phone, Ben's phone at two times Suzy's age, and Brandon's at four times Ben's age. All-layer extracts `8`; attention-only interprets \"two times older\" as adding two extra years and extracts `12`.\n\n### C. Throughput Probe\n\n\n\n_Figure 7. Throughput probe for attention-only LoRA at effective batch size `8`. Pipelined batching reduced mean step time from `20.2` seconds to `2.4` seconds in the retained probe._\n\nThe retained probe measured `20.22187466151081` seconds per optimizer step for sequential single-datum calls, `5.201307859155349` seconds for batched datums, and `2.4104866901249693` seconds for batched pipelined datums.\n\n### D. Source Map Notes\n\nThe public numbers in this article were checked against retained artifacts during drafting. Each source below resolves under the project repository at [github.com/sumitdotml/lora-and-friends](https://github.com/sumitdotml/lora-and-friends) or the linked HuggingFace repo:\n\n| Claim family | Source |\n| ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Baseline accuracy, extraction failures, and eval tokens | [`artifacts/results/baseline-qwen3-8b-gsm8k-001/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/results/baseline-qwen3-8b-gsm8k-001) (`summary.json` and `metrics.jsonl`) |\n| Per-seed LoRA GSM8K accuracies | Six `checkpoint-...-gsm8k-...` directories under [`artifacts/results/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/results) |\n| Training validation NLL and selected step | Six `main-001-<condition>-seed-<seed>/` directories under [`artifacts/results/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/results) |\n| Dataset rows, split rule, source counts | [`artifacts/raw_datasets/openmath_original_clean/manifest.json`](https://github.com/sumitdotml/lora-and-friends/blob/main/artifacts/raw_datasets/openmath_original_clean/manifest.json) |\n| Rendered dataset rows and system prompt | [`artifacts/rendered_datasets/openmath_original_clean_qwen3_disable_thinking/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/rendered_datasets/openmath_original_clean_qwen3_disable_thinking) |\n| Contamination and train/val overlap gates | [`artifacts/audits/contamination_check/report.json`](https://github.com/sumitdotml/lora-and-friends/blob/main/artifacts/audits/contamination_check/report.json) |\n| LR selection, run protocol, null-result rule | [`docs/freeze/run_protocol.md`](https://github.com/sumitdotml/lora-and-friends/blob/main/docs/freeze/run_protocol.md) |\n| Target modules and backend-owned LoRA fields | [`docs/freeze/lora_defaults.md`](https://github.com/sumitdotml/lora-and-friends/blob/main/docs/freeze/lora_defaults.md) |\n| Figure captions and derived tables | Per-figure subdirectories under [`artifacts/figures/`](https://github.com/sumitdotml/lora-and-friends/tree/main/artifacts/figures) (`fig_*.md` and `fig_*.data.csv`) |\n| Public adapter files and model-card README | [HuggingFace model repo](https://huggingface.co/sumitdotml/lora-and-friends) |\n| Public dataset files and dataset-card README | [HuggingFace dataset repo](https://huggingface.co/datasets/sumitdotml/lora-and-friends-dataset) |",
"filePath": "src/content/projects/lora-and-friends.mdx",
"assetImports": [
"../../assets/research/lora-and-friends/fig_04_paired_seed_slope.png"
],
"digest": "1872ee4de6857b8a",
"deferredRender": true,
"collection": "projects"
},
{
"id": "multi-agent-shopping-assistant",
"data": {
"title": "Multi-Agent AI Shopping Assistant",
"subtitle": "An intelligent e-commerce assistant coordinating multi-agent workflows and vector RAG for personalized product discovery and checkout.",
"date": "2026-04-10T00:00:00.000Z",
"authors": [
{
"name": "Heramb Vengurlekar",
"affiliation": "Thadomal Shahani Engineering College",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"Multi-Agent Systems",
"FastAPI",
"ChromaDB",
"RAG",
"Sentence-Transformers",
"Google Gemini API"
],
"codeUrl": "https://github.com/HerambVE/multi-agent-shopping-assistant",
"image": {
"src": "/_astro/brand-icon.DEAsHZtE.png",
"width": 618,
"height": 618,
"format": "png"
},
"draft": false
},
"body": "## Overview\n\nA multi-agent conversational e-commerce assistant engineered to streamline online shopping through intelligent agent coordination, preference parsing, semantic vector retrieval (RAG), and interactive cart/order management.\n\n## Architecture & Technical Highlights\n\n1. **Coordinated Multi-Agent Orchestration:** Specialized agents (Preference Agent, Recommendation Agent, and Order Agent) collaborate sequentially to understand natural language requirements, curate items, and handle state transitions.\n2. **Semantic Vector Search with ChromaDB:** Utilized **Sentence-Transformers** for dense embeddings alongside **ChromaDB** for real-time similarity search across product catalogs and categories.\n3. **LLM Context Synthesis:** Powered by **Google Gemini API** for structured query decomposition, conversational memory, and dynamic recommendation explanations.\n4. **Interactive Asynchronous Web UI:** Integrated with a **FastAPI** backend for low-latency search, real-time cart updates, and guided checkout flow.",
"filePath": "src/content/projects/multi-agent-shopping-assistant.mdx",
"assetImports": [
"../../assets/placeholder-image.png"
],
"digest": "a06ec7fa0d8861a4",
"deferredRender": true,
"collection": "projects"
},
{
"id": "mental-health-chatbot",
"data": {
"title": "Emotion-Aware Mental Health Chatbot",
"subtitle": "A hybrid NLP architecture separating emotion classification and empathetic response generation.",
"date": "2026-03-20T00:00:00.000Z",
"authors": [
{
"name": "Heramb Vengurlekar",
"affiliation": "Thadomal Shahani Engineering College",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"RoBERTa",
"NLP",
"GoEmotions",
"React",
"Flask",
"Node.js"
],
"codeUrl": "https://github.com/HerambVE/chat-bot",
"image": {
"src": "/_astro/brand-icon.DEAsHZtE.png",
"width": 618,
"height": 618,
"format": "png"
},
"draft": false
},
"body": "## Overview\n\nAn empathetic conversational AI system that detects nuanced user emotions and dynamically conditions LLM response generation to provide supportive, context-aware interactions.\n\n## Architecture & Technical Highlights\n\n1. **Emotion Classification:** Fine-tuned **RoBERTa** on the multi-label **GoEmotions** dataset to capture subtle emotional cues beyond binary sentiment.\n2. **Two-Stage Generation Pipeline:** Decoupled classification from generation to prevent hallucinated emotional assumptions and maintain therapeutic guardrails.\n3. **Full-Stack Microservices:** React UI connected to Node.js backend, communicating via REST with a dedicated Flask inference microservice for real-time sentiment scoring.",
"filePath": "src/content/projects/mental-health-chatbot.mdx",
"assetImports": [
"../../assets/placeholder-image.png"
],
"digest": "d9743555a61ba279",
"deferredRender": true,
"collection": "projects"
},
{
"id": "semantic-drift-media-engine",
"data": {
"title": "Semantic Drift Media Engine (SDME)",
"subtitle": "A containerized AI microservices platform for asynchronous media transcription, topic extraction, and contextual reference analysis.",
"date": "2026-06-15T00:00:00.000Z",
"authors": [
{
"name": "Heramb Vengurlekar",
"affiliation": "Thadomal Shahani Engineering College",
"email": "vengurlekarheramb19@gmail.com"
}
],
"keywords": [
"FastAPI",
"Celery",
"Redis",
"Whisper",
"Docker",
"Gemini API"
],
"codeUrl": "https://github.com/HerambVE/sdme",
"image": {
"src": "/_astro/brand-icon.DEAsHZtE.png",
"width": 618,
"height": 618,
"format": "png"
},
"draft": false
},
"body": "## Overview\n\nThe **Semantic Drift Media Engine (SDME)** is a production-ready asynchronous media processing backend designed to handle audio/video ingestion, speech transcription, and contextual synthesis.\n\n## Key Architecture & Decisions\n\n1. **Microservices with FastAPI & Next.js:** Decoupled client interactions from heavy CPU/GPU processing pipelines.\n2. **Distributed Queue with Celery & Redis:** Long-running transcription and analysis tasks are offloaded asynchronously, keeping web API response latencies minimal.\n3. **Local Whisper Optimization:** Implemented lazy-loading and quantization strategies to run speech-to-text transcription within strict **512MB RAM constraints**.\n4. **LLM Synthesis & BYOK Security:** Automated topic extraction and academic reference indexing via Google Gemini API with a Bring Your Own Key (BYOK) paradigm.\n5. **Container Orchestration:** Deployed using Docker Compose with isolated networks and health-checked workers.",
"filePath": "src/content/projects/semantic-drift-media-engine.mdx",
"assetImports": [
"../../assets/placeholder-image.png"
],
"digest": "1141b911706264b2",
"deferredRender": true,
"collection": "projects"
}
]