Question Classifier — a tiny model is all you need for binary classification

Built as part of a team portfolio project during my 2022 data-science training; the classifier itself — concept, labeling, implementation — is my own work. Code on GitHub: SinaRampe/question_classifier.

What the classifier does

A question comes in — “What is the treatment for high-grade astrocytomas?” — and the output is a binary decision: health topic or not. Use case: a question-answering pipeline that needs to separate health-related queries from everything else.

The method: freeze the transformer, train the head

The idea comes from chapter 2 (“Classification”) of the O’Reilly book Natural Language Processing with Transformers: use a pretrained DistilBERT (a distilled, smaller BERT variant) as a frozen feature extractor. The transformer weights stay untouched throughout the entire training run. Instead:

1. Each question is tokenized and encoded by DistilBERT, producing hidden states — 768-dimensional vectors from the last hidden layer.
2. The [CLS] hidden states — the compressed representation of the whole question — serve as features.
3. On top sits a plain logistic regression — and that is what gets trained.

The effect: a transformer’s language understanding without a transformer’s training. Runs entirely on CPU, ~98% accuracy on the held-out split. Right-sized for the task.

And that is the key point for binary classification: it is a full training run — but only 769 parameters are trained (the 768 weights of the logistic regression plus its bias), a mere 0.001% of the model. The encoder above, DistilBERT with 66 million parameters, stays completely frozen. For scale against today’s LLMs: the smallest current Llama (3.2, 1 billion parameters) is 15 times the size of DistilBERT, an 8B model 121 times, a 70B model over 1,000 times. Training or fine-tuning a model of that scale for a yes/no decision wastes compute and energy: tiny is enough, as long as the representation is right.

The principle behind it: meaning as position in space

What actually happens when a question enters the model? DistilBERT does not compute with words but with numbers. For each question, the encoder produces a vector of 768 numbers — and these numbers are, at heart, a mathematical representation of semantics (meaning): questions with a similar meaning land on neighboring points in this 768-dimensional space, questions with a different meaning somewhere else. The model did not invent this “map of meaning” — it is baked into the weights from training on huge amounts of text. The classifier only uses the map; it never changes it.

The figure shows a slice of it: with UMAP (a nonlinear method that squeezes high-dimensional data onto two dimensions for visualization — 768 dimensions do not fit on a screen), the training questions are projected onto the plane. Left: the point density of non-health questions (blue), right: health questions (red).

You can see: the two groups sit in different parts of the space — not perfectly separated, they overlap at the edges, but their density centers lie clearly elsewhere. That is exactly why a simple linear boundary suffices — a straight line in the figure, a plane in the full space. It has 769 parameters. The roughly 2% errors of the classifier come mostly from this overlap zone.

What is in it, and what is not

The dataset: 2,226 questions, every one attributed to a source — 1,355 from SQuAD (Stanford Question Answering Dataset; mostly the non-health questions, plus a few health ones) and 871 from AskDocs (Reddit r/AskDocs, titles). The labels (health yes/no) are my own work.

Important for context: SQuAD is CC BY-SA 4.0, AskDocs is GPL-3.0 — both licenses with redistribution conditions. That is why the repo ships no ready-made dataset. Instead it contains labels.csv (the labeling decisions: source, id, label, split) and a build script that pulls the question texts reproducibly from the original sources. Part of the questions in the first version could not be traced to a licensable source — those were removed. README and NOTICE document sources and licenses in full.

Why this project matters to me

It was the first time I took a method from the literature and implemented it on my own.