shreyansh12183/vigyan-3b-adapter-science

🧪 Vigyan 3B - Domain Expert LoRA: Science

Classification: Experimental Research Artifact / Test Model
Architecture Target: Domain Feed-Forward Expert for Sparse MoE Routing
Base Model: Qwen/Qwen2.5-3B-Instruct
Research Lab: Vigyan AI Sovereign Initiative

---

📌 1. Executive Summary & Research Context

This repository contains an experimental domain-specialized LoRA adapter trained as part of the Vigyan AI Sovereign Mixture of Experts (MoE) Upcycling Initiative.

The goal of this experimental line was to train isolated, high-rank domain experts on specialized STEM corpora and examine whether post-hoc upcycling via uncalibrated linear routers (mergekit-moe) could synthesize dense reasoning models into edge-deployable sparse MoE networks.

Experimental Research Classification:
This model is released publicly as a transparent scientific artifact. Our empirical benchmark findings demonstrated that while this standalone adapter excels in its narrow domain, naive post-hoc MoE stitching without native router pretraining causes attention-FFN desynchronization (see forensic findings below).

---

🔬 2. Domain Specialization & Dataset Curricula

• Domain Track: Natural Sciences & Physical Chemistry
• Technical Scope: Quantum mechanics, molecular spectroscopy, thermodynamic entropy, biophysical docking, and reaction kinetics.
• Training Corpus: Curated subset from shreyansh12183/vigyan-stem-experts-400k (100,000 curated CoT derivation samples).
• Alignment Protocol: Strict Prompt Loss Masking (all user prompt tokens masked with labels = -100 to prevent prompt memorization).
• Convergence Threshold: Dynamic early stopping triggered at loss 0.4820.

---

⚙️ 3. LoRA Hyperparameter Specifications

Parameter	Value	Engineering Rationale
LoRA Rank ($r$)	16	Sufficient rank capacity for technical multi-step derivations
LoRA Alpha ($lpha$)	32	Standard $2\times$ scaling factor for numerical gradient stability
LoRA Dropout	0.05	Regularization against narrow domain overfitting
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj	Full-rank adapter spanning attention and FFN projections
Precision	BFloat16 / Float16	Zero precision truncation on tensor activations
Hardware	Kaggle Dual Tesla T4 (Pinned GPU 0)	Zero multi-GPU communication overhead

---

📊 4. Scientific Post-Mortem & Forensic Findings

During post-training MoE upcycling experiments, this adapter was used in a 4-expert sparse mixture configuration. The experimental run revealed pivotal architectural insights:

1. Attention-FFN Desynchronization: Because this LoRA was trained across both attention and MLP layers, stripping the attention weights during MoE assembly deprived the specialized FFNs of their calibrated input distribution.
2. Attractor Loop Vulnerability: When routed alongside uncoordinated experts by an untrained pseudo-router, autoregressive hidden states diverged into memorized attractor loops.
3. The 70/20/10 Law Formulation: Standalone training on 100% hard STEM highlighted the necessity of the 70/20/10 Dataset Curriculum Law (70% core domain, 20% conversational alignment, 10% base distribution replay) to preserve boundary tokens and conversational priors.

---

🚀 5. Usage with 🤗 PEFT & Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_id = "shreyansh12183/vigyan-3b-adapter-science"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load domain expert adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Derive the fundamental equations governing natural sciences & physical chemistry."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

🔮 6. Strategic Next Steps

In accordance with our empirical post-mortem, our flagship production efforts have pivoted to Allen AI's natively pretrained OLMoE-1B-7B architecture (64 fine-grained experts, Top-8 routing, trained end-to-end on 5.1T Dolma tokens). This adapter remains preserved as a sovereign benchmark baseline.

---

📄 License & Attribution

Released under the Apache 2.0 License for open scientific replication and research by the global AI community.
Developer: Shreyansh (@shreyansh12183)