A four-way empirical comparison of TF-IDF + SVM, fine-tuned DistilBERT, prompted Claude Haiku 4.5 and QLoRA on Qwen2.5-1.5B, measured on macro-F1, p95 latency and dollar cost per 1,000 predictions.
On a label-rich short-text emotion classification task built from the CARER corpus (416,809 samples, six classes), I benchmarked four approaches on the same data and the same evaluation harness: a TF-IDF and Linear SVM baseline, a fine-tuned DistilBERT encoder, a prompted Claude Haiku 4.5 via constrained structured output, and a parameter-efficient QLoRA adapter on Qwen2.5-1.5B. The headline result is counter to common practitioner instinct: the 67M-parameter fine-tuned encoder reaches 0.9128 macro-F1 and beats the frontier LLM by roughly 40 points, while even the classical SVM beats the LLM by 32 points. The LLM is the least accurate and the most expensive option here.
Introduction
Generative LLMs have become the default reach for many text-understanding problems. This paper asks a sharper question. When you already have a well-scoped, label-rich classification task, is prompting a frontier LLM actually better than fine-tuning a small model, and what does each choice really cost? I treat the comparison as an engineering decision rather than a leaderboard contest, measuring not only macro-F1 but also p95 inference latency and dollar cost per 1,000 predictions, with the whole pipeline built as tested software so the numbers are auditable rather than anecdotal.
Key Takeaways
- Fine-tuned DistilBERT reaches 0.9128 macro-F1 with a 67M-parameter encoder and runs locally in 45.9 ms p95.
- Prompted Claude Haiku 4.5 lands at 0.51 macro-F1, roughly 40 points behind the encoder, at about $1.27 per 1,000 predictions.
- Even the classical TF-IDF and Linear SVM baseline (0.83 macro-F1) beats the frontier LLM by 32 macro-F1 points.
- Few-shot prompting did not help: 0.5147 zero-shot moved to 0.5127 with k=6 examples. The deficit is one of label calibration, not capability.
- Structured output enums constrain but do not guarantee. Roughly 1% of LLM calls returned an off-vocabulary label and were coerced to the nearest valid class.
- Practical takeaway: reach for a generative LLM when you lack labels. Reach for a fine-tuned model when you have them.
Conclusion
The takeaway is not that LLMs are bad. It is that an off-the-shelf LLM is the wrong default when you already have labels. Once you have a few thousand labelled examples, a fine-tuned encoder dominates on accuracy, latency and cost simultaneously. The contribution of this work is as much methodological as empirical: production model selection needs measured cost and latency, structured output validation, response caching, rate limiting, deterministic seeds, regression tests and CI. The full paper and a live three-way classifier on Hugging Face Spaces are available below.