Quick answer: pick the LLM fine-tuning method by your hardware and goal. Full fine-tuning updates every weight and needs the most GPU memory. LoRA trains small low-rank matrices on a frozen model, and QLoRA does the same with the base model stored in 4-bit, so it fits on much smaller GPUs. Prefix-tuning, prompt-tuning, P-Tuning v2 and adapters train even fewer parameters. Instruction fine-tuning (SFT) teaches a model to follow prompts, and RLHF aligns it with human preferences. For most projects, start with LoRA or QLoRA using the Hugging Face PEFT library.
In Part One, we talked strategy: why choosing the right base model is the foundation of every fine-tuning project. Think of it as picking the plot of land before you build a skyscraper. Get the foundation wrong, and no amount of fancy architecture will save you.
Now it’s time to roll up our sleeves. Fine-tuning isn’t just about pointing an LLM at your data and hoping for the best. It’s about knowing which technique fits your hardware, your dataset and your patience level. Some methods are heavyweight champions (full fine-tuning), while others are featherweight ninjas (prompt-tuning, prefix-tuning).
Key takeaways
- Fine-tuning means continuing to train a pre-trained LLM on your own data so it performs better on your task or domain.
- Full fine-tuning updates every weight and needs the most compute. Parameter-efficient fine-tuning (PEFT) methods such as LoRA, QLoRA, adapters and prompt-tuning train only a small extra set of parameters.
- On a single consumer GPU, QLoRA is usually the practical choice. Its authors fine-tuned a 65B-parameter model on one 48 GB GPU.
- Instruction fine-tuning and RLHF are about behaviour: teaching a model to follow instructions and prefer answers people rate highly.
The nine main fine-tuning techniques for LLMs
- Full fine-tuning
- LoRA (Low-Rank Adaptation)
- QLoRA (Quantized LoRA)
- Prefix-tuning
- Prompt-tuning
- Adapters
- P-Tuning v2
- Instruction fine-tuning (supervised fine-tuning, SFT)
- RLHF (reinforcement learning from human feedback)
Let’s dive deeper into each one.
1. Full fine-tuning
The “all-you-can-train” buffet. Full fine-tuning updates every parameter of a pre-trained model on a new dataset. It is the most powerful and the most resource-intensive way to adapt a model: it offers maximum flexibility for domain-specific tasks, but needs large compute, careful optimisation and strong data curation.
Technical workflow
- Model initialisation: start from a pre-trained checkpoint and leave every layer trainable.
- Dataset preparation: curate a large, high-quality, domain-specific dataset with deduplication and provenance tracking.
- Training set-up: a learning rate much lower than pre-training (to avoid catastrophic forgetting), large batches with gradient accumulation, AdamW, and dropout, weight decay and gradient clipping for stability.
- Compute: high-memory GPUs, distributed training frameworks such as DeepSpeed, and mixed precision (FP16/BF16).
- Evaluation: track task metrics on a validation set and watch for overfitting and forgetting of general knowledge.
2. LoRA (Low-Rank Adaptation)
LoRA freezes the pre-trained weights and adds small trainable low-rank matrices to selected layers. The key insight from the original LoRA paper (Hu et al., 2021) is that the weight update needed for a new task can be approximated with far fewer numbers. On GPT-3 175B, the authors report around 10,000 times fewer trainable parameters than full fine-tuning.
Technical workflow
- Freeze the base model so its general knowledge is preserved.
- Insert LoRA modules: for a weight matrix W of size d × k, add two small matrices B (d × r) and A (r × k), where the rank r is much smaller than d or k. The effective weight becomes W′ = W + ΔW, with ΔW = BA.
- Train only A and B. The number of trainable parameters drops by orders of magnitude.
- Deploy: merge ΔW into W for zero extra inference cost, or keep adapters separate and swap them per task.
3. QLoRA (Quantized LoRA)
QLoRA combines LoRA with quantisation: the frozen base model is stored in 4-bit precision, and only lightweight LoRA modules are trained. Dettmers et al. (2023) showed this is enough to fine-tune a 65B-parameter model on a single 48 GB GPU while keeping strong performance, which is why QLoRA is the usual choice for very large models on limited hardware.
Technical workflow
- Quantise the base model weights to 4-bit precision to cut the memory footprint.
- Insert LoRA adapters into selected layers; these are the only parameters that are updated.
- Train the adapters while the quantised base stays frozen.
- Deploy by swapping adapters per task on a reusable quantised base.
4. Prefix-tuning
Prefix-tuning learns a small set of continuous “prefix” vectors that are prepended at every transformer layer, while the model itself stays frozen. These vectors act as soft context that steers the model’s attention. In the original paper (Li and Liang, 2021), prefix-tuning trained about 0.1% of the parameters and performed well on text generation tasks.
Technical workflow
- Freeze the pre-trained LLM.
- Add trainable prefix embeddings at each transformer layer.
- Train only the prefixes on your task data.
- Deploy by storing one small prefix per task.
5. Prompt-tuning
Prompt-tuning is even lighter: it learns a few “soft prompt” embeddings that are prepended to the input only, without changing the model’s weights. Lester et al. (2021) found that the gap between prompt-tuning and full fine-tuning closes as models reach billions of parameters, so it works best on large models.
Technical workflow
- Freeze the pre-trained LLM.
- Prepend trainable vectors to the input sequence; these are the only trainable parameters.
- Optimise the embeddings on task-specific data.
- Deploy by swapping prompts per task.
6. Adapters
Adapters are small trainable layers inserted between the frozen layers of a pre-trained model. In Houlsby et al. (2019), adapters added about 3.6% extra parameters per task and came within 0.4% of full fine-tuning on the GLUE benchmark. Because each task gets its own adapter, they suit modular, multi-task deployments.
Technical workflow
- Freeze the original weights.
- Insert small adapter modules between transformer blocks.
- Train only the adapters on task data.
- Deploy by loading the adapter for each task or domain.
7. P-Tuning v2
P-Tuning v2 applies trainable prompt vectors at every transformer layer, not just the input, which makes it more expressive and stable than early prompt-tuning on smaller models. Liu et al. (2021) report that it matches full fine-tuning across a wide range of model scales and tasks while tuning only 0.1% to 3% of the parameters.
Technical workflow
- Freeze the original LLM.
- Inject trainable prompt vectors at multiple transformer layers.
- Train only the prompt embeddings.
- Deploy by swapping prompts per task.
8. Instruction fine-tuning (supervised fine-tuning)
Instruction fine-tuning, also called supervised fine-tuning (SFT), trains a model on curated instruction-response pairs so it follows human-style prompts reliably. Done as full fine-tuning, it updates a large share of the weights; in practice it is often run with LoRA or QLoRA instead (the QLoRA paper’s own experiments are instruction tuning). Either way, the result depends on high-quality labelled data.
Technical workflow
- Dataset preparation: collect diverse, high-quality instruction-response pairs (for example, “Explain Newton’s laws” → “Newton’s laws state…”).
- Training: start from a pre-trained checkpoint and fine-tune with supervised learning at a small learning rate.
- Evaluation: test on held-out instructions for helpfulness, accuracy and hallucinations.
9. RLHF (reinforcement learning from human feedback)
RLHF aligns a model with human preferences. The pipeline described in Ouyang et al. (2022) has three stages: supervised fine-tuning, training a reward model from human rankings of outputs, and reinforcement learning (often PPO) to maximise that reward. It improves helpfulness and safety, but needs a lot of human annotation and careful reward design.
Technical workflow
- Supervised fine-tuning: train on curated instruction-response pairs for a baseline.
- Reward model: annotators rank several outputs per prompt, and a reward model learns to predict those rankings.
- Reinforcement learning: the LLM generates answers, the reward model scores them, and PPO updates the LLM.
- Evaluation: human evaluation of helpfulness, harmlessness and instruction-following.
Comparison table
| Technique | Parameters trained | Hardware needs | Best use case |
|---|---|---|---|
| Full fine-tuning | 100% | Very high | Maximum accuracy, smaller models |
| LoRA | Typically well under 1% (depends on rank and target layers) | Moderate | Efficient task adaptation |
| QLoRA | Same as LoRA, on a 4-bit base | Low | Large models on a single GPU |
| Prefix-tuning | About 0.1% | Very low | Text generation tasks |
| Prompt-tuning | Tiny (input embeddings only) | Very low | Very large models, lightweight tasks |
| Adapters | A few percent per task | Moderate | Modular multi-task set-ups |
| P-Tuning v2 | 0.1% to 3% | Low | Prompt-style tuning that works on smaller models too |
| Instruction FT | Varies (full or PEFT) | Moderate | Following instructions |
| RLHF | Varies | High | Human-aligned responses |
How to choose a fine-tuning method
- One consumer GPU and a 7B to 70B model: start with QLoRA.
- Enough GPU memory for the model in 16-bit: use LoRA, which is simpler and slightly faster than QLoRA.
- Many tasks on one shared base model: LoRA or adapters, so each task is a small file you can swap in.
- A small model, a large dataset and maximum accuracy: full fine-tuning, if you have the compute.
- The model knows the domain but ignores instructions: instruction fine-tuning, usually done with LoRA or QLoRA.
- You need answers that people prefer, not just correct ones: add RLHF or a preference-based method after SFT.
The free Hugging Face PEFT library implements LoRA, prefix-tuning, prompt-tuning and more, and works with 4-bit quantisation for QLoRA-style training.
Trade-offs
- Efficiency vs. accuracy: the lightest methods (prompt-tuning, prefix-tuning) save the most compute but can underperform LoRA or QLoRA, especially on smaller models.
- Scalability: QLoRA is currently the most practical way to fine-tune very large models on limited hardware.
- Alignment: instruction fine-tuning and RLHF are essential for safety and usability, but need curated datasets and human feedback.
Frequently asked questions
What is the difference between LoRA and QLoRA?
Both freeze the base model and train small low-rank matrices. QLoRA also stores the frozen base model in 4-bit precision, which cuts GPU memory sharply. The QLoRA paper fine-tuned a 65-billion-parameter model on a single 48 GB GPU.
Do I need RLHF for my project?
Usually not. RLHF needs human preference data, a reward model and a reinforcement learning stage, so it is expensive. For most domain or task-specific projects, instruction fine-tuning (often done with LoRA or QLoRA) is enough.
Which library should I use?
The free Hugging Face PEFT library implements LoRA, prefix-tuning, prompt-tuning and other parameter-efficient methods, and works with 4-bit models for QLoRA-style training.
How do I choose between methods?
Try two methods on a small version of your task and compare accuracy, cost and training speed. Choose your base model first, because it limits every option that follows.
Choosing the right wrench
Fine-tuning isn’t a one-size-fits-all game. It’s a toolbox, and each method has its place: heavyweight like full fine-tuning, efficient like LoRA and QLoRA, or featherweight like prompt-tuning and prefix-tuning. Adapters and P-Tuning v2 give you modularity, while instruction fine-tuning and RLHF push models toward human-aligned behaviour.
Now it’s your turn: experiment with two methods on a small task, benchmark them on accuracy, cost and speed, and decide strategically. Don’t just ask “Can I fine-tune this?”; ask “Which fine-tuning method makes the most sense for my context?” So, which wrench will you pick up first? Tell me in the comments.
Sources
- Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
- Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs.
- Li, X. L. and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation.
- Lester, B., Al-Rfou, R. and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning.
- Houlsby, N. et al. (2019). Parameter-Efficient Transfer Learning for NLP.
- Liu, X. et al. (2021). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks.
- Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback.
- Hugging Face. PEFT documentation.
Want a step-by-step plan for learning machine learning?
The 90-Day Machine Learning Roadmap takes you from Python basics to a deployed ML portfolio in 13 weeks, with a plan for every day and a progress tracker. Going for your first ML or data science job? The ML Career Bundle adds an interview kit, Python practice and resume templates.
Prefer video? Watch the tutorials on YouTube · Browse all toolkits




4 responses to “Complete Guide to Fine-Tuning LLMs: From Prompt-Tuning to RLHF for Smarter AI”
[…] Complete Guide to Fine-Tuning LLMs: From Prompt-Tuning to RLHF […]
[…] Complete Guide to Fine-Tuning LLMs: From Prompt-Tuning to RLHF […]
[…] data, teaching it your domain, and turning a capable giant into a focused teammate. That’s where Part Two of this series comes in: a hands‑on workflow that takes you from data prep → padding → QLoRA setup → […]
[…] Where it fails: An LLM (large language model, a system trained to predict likely text) does not look things up in a standards book. It can invent a formula, mix units, quote a safety factor from the wrong standard, or quietly drop an assumption. It is also weak at spatial reasoning: ask it to judge whether a bracket geometry will clear a bolt and you are gambling. If you want to know how these models are adapted for specialised work, see our complete guide to fine-tuning LLMs. […]