Medical Text Classification: 12 Experiments from TF-IDF to LLMs

I recently spent a few weeks building a text classification system for medical correspondence. The task: take OCR’d clinic letters and classify them into ~49 categories — specialties like Cardiology and Ophthalmology, administrative types like Discharge Summaries, diagnostic categories like Echocardiogram results.

I ran 12 experiments. Here’s what actually mattered.

The dataset

Experiment 1: TF-IDF baseline

TfidfVectorizer (unigrams + bigrams, 50k features) with a LinearSVC. Result: ~91% accuracy.

A good sanity check. Most of the classification signal lives in simple lexical features — certain words and phrases are strong indicators of letter type. But bag-of-words can’t capture word order or context, so there’s a ceiling.

The biggest win: fixing the labels

Before trying fancier models, I looked at the label set. Some categories were synonymous (“Nephrology” and “Renal”), some were ambiguous, and a few were too vague to be useful. After merging and cleaning:

This turned out to be the single largest improvement in the entire project.

Experiment 2: DistilBERT

Fine-tuned distilbert-base-uncased — 4 epochs, batch size 16, learning rate 2e-5, max 512 tokens. Matched the ~96% mark and became the baseline for everything else.

Experiments 3–6: Bigger and fancier models

I tried several model variations, hoping to push past 96%:

• ClinicalBERT & BioClinicalBERT — models pre-trained on clinical notes. No improvement. The task doesn’t rely on deep clinical terminology — it relies on structural cues like headers, greetings, and clinic names.
• Longformer at 1024 tokens — maybe more context would help? No. Most letters are identifiable from their first page. The letter type shows up early.
• Hierarchical classification — broad category first, then fine-grained. Added complexity without benefit. The 49-class space is already well-separated.

Experiments 7–9: LLM relabeling and distillation

This is where things got interesting — and humbling.

LLM relabeling

I used a large language model (via batch API) to independently classify all 13,700 samples. The LLM agreed with the original labels only 85.7% of the time.

Training DistilBERT on LLM-assigned labels:

The LLM’s labels weren’t wrong in a random way. They were systematically different. The LLM doesn’t know how a healthcare organization internally categorises its own correspondence. It applies its own logic, which doesn’t match the operational reality.

Consensus relabeling

Only change a label when both the trained BERT and the LLM agree the original is wrong. Out of 9,500+ training samples, only 4 met this criterion. BERT memorises its training labels almost perfectly, so it virtually never disagrees with them on in-sample data.

Soft knowledge distillation

Used the LLM’s top-5 predictions with confidence scores as soft targets — blended loss: α × CE(hard labels) + (1-α) × KL(soft labels ‖ student logits).

Experiments 10–11: Cleanlab (the second big win)

Cleanlab uses confident learning to find likely mislabeled samples. I ran 3-fold cross-validation to get out-of-sample predictions, then flagged samples where the model confidently disagreed with the label.

142 out of 9,500 training samples flagged (1.5%). Manual inspection confirmed ~99% were genuinely mislabeled — the model was right, the ground truth was wrong.

The key insight: instead of removing flagged samples, I relabeled them with the model’s prediction. I also ran cleanlab on the test set, finding 62 mislabeled test samples.

Most of the model’s “errors” were actually correct predictions being penalised by wrong ground truth.

The production model

Final approach: run cleanlab on the entire dataset (train + val + test), relabel all 212 flagged samples (1.6% of the data), train a fresh DistilBERT on the full corrected dataset for 6 epochs.

Estimated production performance: ~98% top-1, ~99% top-3, ~99.4% top-5.

The full picture

Key takeaways