Chapter 4 — Reference-based Annotation
Where this chapter sits. Companion to Lecture 04 and Tutorial 04. Prerequisites: Chapter 3 — Manual Annotation (the complement to this).
4.1 The two paradigms
Cell-type labeling has two methodological camps:
Manual annotation (Chapter 3) — inspect canonical markers per cluster, assign names by hand. Interpretable, biased by the annotator’s experience, requires a literature.
Reference-based annotation (this chapter) — map your cells against a labeled reference dataset using a statistical method. Reproducible, biased by the reference’s coverage, requires a reference of the same tissue.
Best practice is to run both and reconcile disagreements — they catch different kinds of errors. Manual annotation catches references missing your cell type; reference-based annotation catches missed subdivisions and reduces annotator bias. Automated annotation is now a crowded field — large comparisons evaluate 20–30+ classifiers and find that the best correlation- and classifier-based methods are accurate when the reference matches the query, but degrade sharply on cell types absent from the reference1,2.
Every reference-based method — Azimuth, SingleR, or CellTypist — frames annotation as a nearest-neighbor problem in a shared gene-expression space: given a query cell and a set of reference cells with known labels, the query is assigned the label of the reference cells it most resembles, weighted by some confidence metric. What differs between methods is how resemblance is measured (correlation, Euclidean distance in PCA space, or learned logistic-regression weights) and how the reference was prepared (bulk RNA-seq averages, single-cell atlas, or a trained model). Crucially, all methods share the same blind spot: they can only assign labels that exist in the reference. A cell type absent from the reference will be assigned the closest available label, possibly with low confidence — it will never spontaneously return “novel.” This is why reading the confidence metric and reconciling against manual markers is not optional.
The workshop tutorial uses Azimuth as its reference-based method: it ships curated, atlas-quality references (including a PBMC reference that is a perfect fit for ifnb), downloads them on demand, and needs no separate Bioconductor reference-data packages. Table 1 summarizes the three paradigms side by side; §4.4 walks SingleR and CellTypist further.
| Method | Approach | Reference needed | Confidence metric | Best for |
|---|---|---|---|---|
| Azimuth | Anchor-based mapping that projects the query onto the reference’s precomputed (s)PCA structure (pcaproject/spca), not a joint CCA |
Pre-built atlas references (single-cell) | mapping.score per cell |
PBMC, brain, lung, fetal: tissues with curated Seurat refs |
| SingleR | Per-cell Spearman correlation against per-label reference profiles | Bulk or single-cell; celldex library |
delta (top score − median); prune with pruneScores() |
When only a bulk reference exists; Bioconductor-native workflows |
| CellTypist | Logistic-regression classifier trained on atlas data | Pre-trained models (Python package) | Per-label probability (multi-label aware) | Python / Scanpy workflows; multi-label / ambiguous cells |
§4.4 sketches SingleR and CellTypist further.
4.2 Azimuth — anchor-based reference mapping
Azimuth wraps Seurat’s anchor-based mapping3 (the same method as Seurat integration, applied with a query → reference orientation). The key design advantage is that the reference UMAP and cell labels are precomputed once from a large atlas dataset; at query time, your cells are projected into that manifold rather than re-computing everything jointly. This is fundamentally different from methods that require both the query and reference to be processed together: because Azimuth’s reference is frozen at publication time and its UMAP is precomputed, mapping a new query to it is fast (minutes on a laptop), consistent across queries, and does not require re-running the reference-side integration every time. The shared feature space is determined by finding mutual nearest neighbors across the two datasets after projecting the query onto the reference’s precomputed (supervised) PCA (pcaproject/spca) — note this differs from the de novo CCA used for multi-sample integration in Chapter 5: here the reference structure is fixed and the query is mapped onto it, not learned jointly. Pre-built atlas-quality references are available:
mapping.score and high error rates.
| Reference | Tissue | Source |
|---|---|---|
pbmcref |
PBMCs | Hao et al. 2021 (Seurat v4 paper) |
motorcortexref |
mouse cortex | Allen Brain Atlas |
fetusref |
human fetal | HCA Fetal Atlas |
lungref |
adult lung | HCA Lung Cell Atlas |
| (more) | various | atlas consortia |
For each query cell, Azimuth:
- Projects it into the reference’s PC space using shared HVGs
- Finds its anchored partner cells in the reference (mutual nearest neighbors after L2 normalization)
- Transfers the anchor partners’ labels (with confidence weights)
- Reports a mapping score per cell — how well the cell projected into the reference manifold
library(Azimuth)
seu <- RunAzimuth(seu, reference = "pbmcref")
# new metadata: predicted.celltype.l1 / l2 / l3 (+ .score) and mapping.scoreStrengths:
- Very fast (the reference is precomputed) and needs no Bioconductor reference-data download
- Hierarchical labels (
predicted.celltype.l1,l2,l3) — coarse to fine resolution - Atlas-trained references are huge and well-annotated
Weaknesses:
- Reference must match the tissue — a PBMC reference will mis-label tumor cells
- Cells outside the reference manifold get low mapping scores but still get labeled — must filter on
mapping.score
4.3 Reading Azimuth’s mapping score
mapping.score is the primary quality filter for Azimuth output; it reflects how well the cell’s local neighborhood projects into the reference’s manifold. Table 3 gives practical interpretation ranges. Bimodal distributions are common: one peak represents in-reference cells (mapping score 0.7–1.0), the other out-of-reference cells (0.0–0.4). A good first diagnostic after running Azimuth is to plot FeaturePlot(seu, features = "mapping.score") on the UMAP — a clean matched-reference run produces a uniformly high-scoring cloud with occasional low-scoring outliers; a bimodal distribution with a substantial low-scoring population tells you the reference doesn’t cover a large fraction of your data and the labels on those cells should not be trusted. Plotting tapply(seu$mapping.score, seu$stim, mean) for a multi-condition dataset quantifies any systematic asymmetry between conditions.
mapping.score. On a well-matched reference (e.g. pbmcref applied to a healthy-donor PBMC dataset), the vast majority of cells should score above 0.7; a large low-scoring fraction indicates a reference mismatch.
| Score range | Interpretation | Recommended action |
|---|---|---|
| 0.7 – 1.0 | Cell maps confidently into the reference manifold | Trust the label |
| 0.4 – 0.7 | Borderline — cell is near but not firmly inside the reference | Inspect manually; cross-check against manual marker annotation |
| 0.0 – 0.4 | Cell does not project into the reference | Label is unreliable; likely a cell type absent from the reference |
For cells with low mapping score:
- The label assignment is unreliable — don’t trust it
- These are usually cells whose biology isn’t in the reference (e.g. tumor cells when the reference is PBMC)
In an ifnb PBMC analysis with the pbmcref reference, mapping scores should be uniformly high (>0.7) because the data is the reference’s intended target. In a tumor analysis using pbmcref, the immune cells score high and the tumor/stromal cells score low — and you should not trust the labels on the low-scoring cells.
The ifnb STIM/CTRL asymmetry. Even on a matched reference, you may notice that STIM cells receive systematically lower mapping scores than CTRL cells of the same type. This is expected: the pbmcref reference was built from resting PBMCs, and IFN-β stimulation shifts the STIM cells away from the resting manifold even though their cell-type identity is unchanged. The right interpretation is that the label is still correct (the cells are genuinely CD14 Monocytes or CD4 T cells) but the confidence is lower because the activated state is under-represented in the reference. Use tapply(seu$mapping.score, seu$stim, mean) to quantify the asymmetry; if it is large and systematic, consider comparing Azimuth labels against your manual annotations more carefully for the STIM condition.
Always report the share of low-mapping-score cells in your methods, the same way you would report a prune fraction. Healthy on a matched reference: very few. A large low-scoring fraction means the reference doesn’t cover your data — pick a different one.
Azimuth reports mapping.score per cell in the range [0, 1]. Recommended threshold: > 0.7 to trust a label. There is no single hard cutoff baked into the method — the Azimuth documentation uses 0.7 as a practical rule of thumb, and the workshop adopts it. Cells below 0.7 are “borderline” (0.4–0.7) or clearly off-manifold (< 0.4) and should be compared against manual marker annotation before their label is accepted. In practice, on a well-matched reference (e.g. pbmcref applied to healthy-donor PBMCs) virtually all cells score > 0.7; on a mismatched reference (e.g. pbmcref applied to tumor tissue) a large fraction will score < 0.4. Always report the proportion of cells below threshold in your methods section.
4.4 Other reference-based methods you’ll meet
The workshop runs Azimuth, but two other paradigms are common in the literature. Understanding them lets you read methods sections critically and choose the right tool when Azimuth’s pre-built references don’t match your tissue.
SingleR — correlation against a reference4. SingleR compares each query cell’s expression vector against per-label reference profiles by Spearman correlation, then assigns the best-matching label, optionally with a fine-tuning step that iteratively removes low-scoring labels until only one remains. Its classic pairing is with the celldex package, which provides curated bulk RNA-seq references (e.g. HumanPrimaryCellAtlasData(), covering 37 main human cell types, and ImmGenData() for mouse immune). The key diagnostic is the delta statistic — the top-label score minus the median score over all labels — which pruneScores() uses to NA the label when the margin is too small to trust. SingleR shines when you only have a bulk reference (no matching single-cell atlas exists for your tissue) and in Bioconductor-native workflows. Its main weakness is that, like all reference methods, it always returns a label — including for cells whose type is genuinely absent from the reference. The pruneScores() step is not optional; it is what separates reliable from unreliable calls.
CellTypist — pre-trained logistic regression5. CellTypist trains a sparse logistic-regression classifier per cell type from atlas-scale references (primarily the Human Cell Atlas and large CELLxGENE census datasets), then applies the trained model to your query cells. It is fast, returns per-label probability scores (making it multi-label aware — a cell can score high for both “CD4 T activated” and “CD4 T naive”), and ships over 40 pre-trained models covering many human and mouse tissues. The Python implementation is the mature one; for Scanpy workflows it is the most natural choice. Like all classifiers, its accuracy depends on whether its training labels match the granularity and tissue context of your query.
All three methods share the same fundamental failure mode: when the reference doesn’t cover a cell type present in the query, the method still assigns the closest available label. The benchmarks in §4.1 make this point repeatedly1,2. The practical lesson is identical regardless of method: always read the confidence diagnostic (mapping.score for Azimuth, delta for SingleR, probability for CellTypist), filter low-confidence cells, and reconcile against manual markers before publishing any annotation.
SingleR: pruneScores(results, min.diff.med = 0.05) removes labels where the top-label Spearman score minus the median score over all labels (delta) is too small to be trustworthy. The default min.diff.med = 0.05 is a conservative threshold; reducing it accepts more marginal calls. Cells pruned to NA by pruneScores are the SingleR equivalent of low Azimuth mapping scores — they should not be used without further investigation. CellTypist: CellTypist.annotate(majority_voting = True) consolidates per-cell probability scores into a cluster-level majority vote before assigning a final label — reducing noise on a per-cell basis at the cost of assigning the same label to all cells in a cluster even when a few clearly belong elsewhere. majority_voting = False (the default) assigns each cell independently based on its highest per-label probability, which can be inconsistent across cells in the same cluster but exposes genuine heterogeneity.
4.5 The reconciliation step
After running an automatic method, you have:
- Manual labels (
celltype_manualfrom Chapter 3, or the author-curatedseurat_annotationsground truth) - Azimuth labels (
predicted.celltype.l2)
A workable reconciliation rule: keep the manual label where the automatic call agrees, flag the rest — but first, a critical gotcha:
Different methods speak different vocabularies. Manual/ground-truth labels and Azimuth labels rarely use identical strings, and they sit at different granularities — CD14+ Monocytes (a muscData ground-truth label) vs CD14 Mono (Azimuth level 2) vs Mono (Azimuth level 1); B cells vs B naive. A raw == comparison would mark every cell as a disagreement. Harmonize to a common coarse vocabulary first, then compare at the granularity the two sides share — usually Azimuth level 1 (predicted.celltype.l1).
# Map both sides to a common coarse vocabulary, then compare at level 1.
to_coarse <- function(x) dplyr::recode(as.character(x),
"B cells" = "B", "CD4 T cells" = "CD4 T", "CD8 T cells" = "CD8 T",
"NK cells" = "NK", "Dendritic cells" = "DC",
"CD14+ Monocytes" = "Mono", "FCGR3A+ Monocytes" = "Mono",
"Megakaryocytes" = "Mk", .default = NA_character_)
manual_coarse <- to_coarse(seu$seurat_annotations)
agree <- !is.na(manual_coarse) & manual_coarse == as.character(seu$predicted.celltype.l1)
seu$celltype_final <- ifelse(agree, as.character(seu$seurat_annotations),
paste0("REVIEW:", as.character(seu$seurat_annotations)))Cells where the calls agree at level 1 keep the ground-truth label; the rest get a REVIEW: prefix and you investigate manually. (If you also ran a second method, add a clause so a cell that agrees with either is accepted.)
For the ifnb workflow this typically agrees on 92%+ of cells — the disagreements concentrate in:
- T-cell subsets (CD4 Naive vs Memory)
- Rare populations (pDC, Mk, Eryth)
- Activated states (T activated, B activated) where lineage and state coexist
These are exactly the cells that benefit from human review.
4.6 The Cell Ontology connection
For final labels, store the ontology term ID alongside the human label:
seu$cell_type_ontology_term_id <- case_when(
seu$celltype_final == "B" ~ "CL:0000236",
seu$celltype_final == "CD14 Mono" ~ "CL:0001054",
...
)This is required for CELLxGENE submission6 (see Tutorial 17) and makes your dataset interoperable with atlas resources built on cell-type ontologies7. Look up IDs at OLS.
4.7 Common errors
Table 4 collects the most frequent problems encountered with reference-based annotation tools, their root causes, and the fixes.
mapping.score before using any labels — automated methods always produce a label, regardless of whether the cell type is represented in the reference.
| You see | What’s wrong | What to do |
|---|---|---|
RunAzimuth errors during reference download |
Partial cache or no internet | Delete ~/.cache/Azimuth/, confirm connectivity, retry |
| Mapping scores are uniformly low | Wrong reference for your tissue | Match tissue; consider an atlas like Tabula Sapiens8 |
| All cells get a coarse label | Using predicted.celltype.l1 |
Use predicted.celltype.l2 / l3 for finer resolution |
| Manual and Azimuth disagree on T-cell subsets | Normal — T-cell subdivision is genuinely hard | Inspect markers; document; sometimes both are partly right |
| Every cell is labeled even with low mapping score | No confidence filter applied | Filter with mapping.score > 0.7 before accepting labels |
4.8 Where this material is also discussed
- Lecture: Lecture 04
- Tutorial: Tutorial 04
- Manual annotation (the complement): Chapter 3
- Cell Ontology lookup: Appendix E § 1.2, Tutorial 17
4.9 Going further
Go to the method papers — Azimuth / Seurat v43, and for the other paradigms SingleR4 and CellTypist5 — and to the two surveys that put them in context: the benchmark of 22 automated classifiers1 and the review of 32 annotation methods2. For hands-on guidance, the Azimuth web tool is the practical reference; large atlases such as Tabula Sapiens8 make good general-purpose references when a tissue-specific one is unavailable. The full curated list is on Key Papers, Reviews & Benchmarks.