Prompt-baking
prompt-baking bakes system prompts into model weights via KL-divergence LoRA training. After training, the model behaves as if the system prompt is present — without needing it at inference time.
Stack: Python (transformers + PEFT). Public repo. Based on the Prompt Baking paper (arXiv 2409.13697).
Saves tokens, reduces latency, simplifies deployment.
How it works
- Teacher = base model (frozen) with system prompt in context.
- Student = base model + LoRA adapters (trainable) without system prompt.
- Objective = minimize
D_KL(P_teacher ‖ P_student).
The student learns to produce the same output distribution as the teacher, absorbing the system prompt behaviour into LoRA weights. Matches the full output distribution — more faithful than SFT on teacher generations alone.
Quick start
from prompt_baking import bake
adapter_path = bake(
model="Qwen/Qwen3-0.6B",
system_prompt="You are Andy, a friendly assistant who loves chess.",
prompts=["What is your name?", "What are your hobbies?"],
)
Load the baked adapter with PeftModel.from_pretrained — no system prompt needed at inference.
Key parameters
| Parameter | Default | Description |
|---|---|---|
lora_r |
64 | LoRA rank |
lora_alpha |
128 | LoRA scaling |
epochs |
3 | Training passes over prompts |
lr |
1e-4 | Learning rate |
temperature |
1.0 | KL divergence temperature |
num_trajectories |
4 | Teacher samples per prompt per epoch |
load_in_4bit |
False | QLoRA mode |
Sources
See also
- bakery — generalization to arbitrary prefix contexts.