RF-DETR for Industrial Anomaly
Detection and Segmentation

Abstract. We evaluate RF-DETR¹, a real-time detection transformer, as an anomaly detector and segmenter on the VisA industrial inspection dataset, fine-tuning one model per product class. While it is comparable to the supervised baseline when annotated samples are sufficient, we find that in the few-shot regime, with only 5–10 labelled anomalous images, it outperforms the supervised baselines by a significant margin, verifying its ability to transfer to specialized industrial products. Furthermore, we break down performance by defect size and demonstrate that the pixel-level scores are mainly attributed to large defects, which are easier to detect. Spotting and localizing small-area defects remains an open challenge.

1Anomaly Detection & Segmentation

Anomaly detection is the task of identifying image samples that deviate from the norm, deciding whether an image contains any defect at all. Anomaly segmentation goes one step further and localizes the defective regions within each sample. In real-world problems such as the quality control of industrial products, abnormal samples are generally difficult to collect. Therefore, existing solutions focus on training models that identify data that deviate from the learned distribution of normality as anomalies. However, some deviations — such as lighting or orientation — are caused by the imaging pipeline rather than the product itself, and labelled defects are the only signal that tells the two apart. In this report, we focus on supervised anomaly detection and segmentation.

This problem poses the following two main challenges:

• Defective examples are scarce. An industrial pipeline rarely produces defects, so usable defect images accumulate slowly — typically fewer than 100 annotated samples per product. The MVTec AD dataset contains roughly 80 defective images per category.² The Visual Anomaly (VisA) dataset has 100 anomalous images per product, of which only 60 are used for training.³
• Defects are diverse and product-specific. Industrial images are highly diverse and rarely resemble common natural images: each product line has its own appearance, materials, and imaging setup, so the defects that matter on one line rarely appear on another.

RF-DETR¹ — a lightweight object-detection and instance-segmentation model built for fine-tuning on small domain-specific datasets — fits both constraints. In this report, we evaluate the strengths of RF-DETR and the remaining challenges.

2Dataset and Evaluation Metrics

VisA (Visual Anomaly) is one of the standard industrial anomaly-detection benchmarks: 10,821 high-resolution images across 12 product subsets — 9,621 normal and 1,200 anomalous, as depicted in Figure 1. The subsets span four printed circuit boards, capsules, and candles photographed in multi-object layouts, and six food products — cashews, chewing gum, two kinds of fryum, and two kinds of macaroni. The defects are physical damage on real products: scratches, dents, contamination spots, missing parts, bent leads. A detailed illustration of the defects can be found in Figure 2.

In this report we adopt (1) the supervised 2-class “high-shot” split (§4.2), which moves 60 anomalous images per subset into the training set, and (2) the supervised 2-class “few-shot” split (§4.3), which includes only 5 or 10 anomalous images per subset in the training set.

3Method

3.1  RF-DETR-Seg Model

RF-DETR is a family of real-time detection transformers built on a DINOv2-pretrained backbone⁴; RF-DETR-Seg is its instance-segmentation variant, which adds a mask head on top of the detector. It predicts each defect as a mask, along with the corresponding bounding box and confidence score. The neural architecture search (NAS) pretraining makes RF-DETR generalize better to diverse target datasets, as verified by the RF100-VL⁵ experiment in the original paper. This ability makes RF-DETR a suitable choice for the anomaly detection and segmentation task, as annotated defective samples are usually scarce and specific to each industrial product.

For image-level anomaly detection, we use the maximum confidence score across all predicted defects as the anomaly score. If no defects are predicted, the anomaly score is set to 0. For pixel-level anomaly segmentation, we use the masks produced by the segmentation head.

3.2  Training objective

Training follows the standard DETR recipe. Each ground-truth defect is first matched one-to-one with the predicted instance that fits it best (Hungarian matching); the loss is then computed over the matched pairs, as a weighted sum of three parts:

• Classification — an IoU-aware binary cross-entropy: a prediction's confidence is pushed towards how well its box actually overlaps the defect.
• Box regression — an L1 loss on box coordinates plus a GIoU loss on box overlap.
• Mask — per-pixel binary cross-entropy plus a Dice loss. Dice scores the overlap of the whole mask rather than each pixel independently.

4Experiments

Due to limited computational resources, we use the COCO-pretrained⁶ RF-DETR-Seg-Small model with 384×384 input resolution and 100 queries. We fine-tune the model separately for each of the 12 VisA product classes and evaluate on its official test split. In the high-shot setting, the model trains on the entire training split. In the few-shot setting, it instead sees only N anomalous and N normal training images (N = 5 or 10), drawn with 5 different sampling seeds, and every few-shot number reported below is averaged over those 5 seeds. Every run trains for 30 epochs with identical hyperparameters (AdamW, lr 5×10⁻⁵, effective batch size 16), and all reported numbers come from the final-epoch EMA weights.

4.1  Evaluation metrics

• AU-PR (area under the precision–recall curve). It sweeps the anomaly-score threshold and integrates the resulting curve, summarizing the trade-off between precision and recall. At the image level, each test image is one sample, scored by the single anomaly score; at the pixel level, every pixel is a sample, scored by the anomaly map against the ground-truth mask.
• AU-ROC (area under the ROC curve). It plots the true-positive rate against the false-positive rate under the same threshold sweep and integrates the resulting curve. A random scorer ranks defective and normal samples equally often, resulting in a diagonal curve whose area is 0.5. The image/pixel split is the same as above: ranking whole images by their anomaly score, or all pixels by their anomaly-map value.

4.2  Supervised high-shot

In this report, we compare to the baselines reported in the original VisA paper: three self-supervised pre-training methods (SimSiam, MoCo, SimCLR) and a supervised pre-training baseline. They propose the SPot-the-Difference (SPD) regularization loss to increase model invariance to slight global changes by contrastive learning. Their supervised baseline fine-tunes an ImageNet-pretrained ResNet-50: a standard binary classifier for image-level detection, and a U-Net with a ResNet-50 encoder for segmentation. In the VisA paper, their recommended resolution is 800×800. Although the RF-DETR-Seg-Small model in this report runs at only 384×384, it achieves results comparable to the supervised baseline, as shown in Tables 1 and 2.

Table 1 — Supervised high-shot anomaly detection (image-level), using the full official training split. The table reports the mean across 12 products.

Table 2 — Supervised high-shot anomaly segmentation (pixel-level), using the full official training split. The table reports the mean across 12 products.

4.3  Supervised few-shot

We report the results of image-level anomaly detection in Table 3 and pixel-level anomaly segmentation in Table 4. We found that RF-DETR outperforms the supervised baselines on both metrics. In the 5-shot setting, where training samples are scarcest, the margin is significant: RF-DETR reaches 70.6 image-level AU-PR against 59.8 for Supervised ResNet+SPD, and 89.5 versus 85.9 AU-ROC. Similar results are observed in anomaly segmentation — 29.7 versus 18.7 AU-PR and 84.8 versus 75.9 AU-ROC over Supervised Unet+SPD. The results suggest that RF-DETR can adapt to a specialized industrial product with only a few samples.

We hypothesize that this can be attributed to the architecture augmentation via weight-sharing NAS in the training process of RF-DETR. Because the model has learned across various image resolutions, patch sizes, and query counts, it can be fine-tuned to a target dataset with only a few samples. This is consistent with the RF100-VL⁵ experiments in the RF-DETR paper, which showed that the model transfers well across diverse target domains. The detailed results for each product class are shown in the collapsible Tables 5 and 6 (click to expand).

Table 3 — Anomaly classification (image-level). The table reports the mean across 12 products, averaged over 5 sampling seeds (± std across seeds).

Table 4 — Anomaly segmentation (pixel-level). The table reports the mean across 12 products, averaged over 5 sampling seeds (± std across seeds).

Table 5 — Per-subset anomaly detection results

Table 5 — Per-subset anomaly detection (image-level) results; each cell is the mean over the 5 sampling seeds (final-epoch EMA weights, 384×384 eval grid, official AUPR convention).

Subset	5-shot		10-shot
	AU-PR	AU-ROC	AU-PR	AU-ROC
candle	53.2	86.6	65.8	92.7
capsules	72.7	88.1	89.9	96.5
cashew	80.8	90.9	86.4	94.4
chewinggum	92.4	97.5	96.6	98.7
fryum	72.1	86.5	77.0	90.5
macaroni1	78.3	91.7	85.5	95.6
macaroni2	59.0	85.6	66.2	89.9
pcb1	60.0	88.7	76.6	94.1
pcb2	62.1	85.9	65.6	88.7
pcb3	55.8	85.9	61.3	84.7
pcb4	72.4	91.4	70.5	92.0
pipe_fryum	88.4	95.5	93.6	98.0
macro mean	70.6	89.5	77.9	93.0

Table 6 — Per-subset anomaly segmentation results

Table 6 — Per-subset anomaly segmentation (pixel-level) results; each cell is the mean over the 5 sampling seeds (final-epoch EMA weights, 384×384 eval grid, official AUPR convention).

Subset	5-shot		10-shot
	AU-PR	AU-ROC	AU-PR	AU-ROC
candle	17.2	80.6	18.8	78.7
capsules	39.0	86.5	70.0	96.2
cashew	34.8	93.6	50.8	92.1
chewinggum	79.0	97.1	87.4	98.0
fryum	19.0	71.5	75.6	98.7
macaroni1	26.6	80.1	34.2	78.0
macaroni2	17.4	74.3	21.6	87.1
pcb1	12.4	84.3	55.5	83.3
pcb2	22.6	84.7	31.7	88.1
pcb3	23.8	84.8	30.1	89.2
pcb4	19.9	87.2	23.6	91.2
pipe_fryum	44.7	92.5	74.8	94.5
macro mean	29.7	84.8	47.8	89.6

4.4  Defect sizes study for anomaly segmentation

While the anomaly segmentation performance seems satisfactory in the above experiments, the model could still focus on a few large and easy defects but miss many small defective instances. To evaluate the performance on each defect scale individually, we divide the ground-truth defects by pixel area, using the model input size of 384×384 as the reference resolution: tiny under 16 px², small under 100 px², medium under 441 px², and large above 441 px².

Table 7 — High-shot pixel metrics per mask-area tier (the §2.1 tiers). One model per product class trained on the full official training split; each cell is the mean over the 12 classes. Scoring one tier ignores the pixels of every other tier.

Table 8 — Few-shot pixel metrics per mask-area tier (the §2.1 tiers). Each cell is the mean over the 12 product classes, averaged over the 5 sampling seeds (± std across seeds). Scoring one tier ignores the pixels of every other tier.

From Tables 7 and 8, we can observe that most of the contribution to the total scores comes from the large defects, which is expected. This is likely because both the training objective and the evaluation metrics are designed to reward predicting as many defective pixels as possible. However, performance drops rapidly as defect size decreases. If each defect is instead seen as a distinct instance, the model cannot find small defects as well as the total AU-PR or AU-ROC would suggest.

We can expect that a model with a higher input resolution may alleviate this issue — e.g., RF-DETR-Seg Max at 890×890, the largest variant in the family. However, treating every defect instance equally would still require redesigning the training objective.

References

1. Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, and Neehar Peri. RF-DETR: Real-time detection transformer. In International Conference on Learning Representations (ICLR), 2026.
2. Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. MVTec AD — a comprehensive real-world dataset for unsupervised anomaly detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
3. Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. SPot-the-Difference self-supervised pre-training for anomaly detection and segmentation. In European Conference on Computer Vision (ECCV), 2022.
4. Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024.
5. Peter Robicheaux, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow100-VL: A multi-domain object detection benchmark for vision-language models. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025.
6. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.

Appendix

Training schedule, per-epoch curves & COCO sanity check

• Schedule: a fixed 30 epochs per subset, no early stopping, learning rates at half the official RF-DETR fine-tune recipe (5e-5 / 7.5e-5 encoder); everything else — effective batch 16, EMA, augmentation — follows the official recipe unchanged. Both come from a 50-epoch pilot at the stock recipe, whose curves converged by epoch ~20–25 and mildly degraded after. The deployed checkpoint is the best-on-validation segm mAP (EMA-aware) snapshot; the test set is never touched during training.
• Per-epoch curves: the full metric set is recorded at every eval point (every 5 epochs + final), so the learning curves show the same quantities the final table reports. No test-split number ever feeds model selection.
• Sanity check: before any VisA training, we ran the pretrained RF-DETR checkpoints on COCO val2017 through our exact evaluation path. Box and segmentation AP both land within 0.6 points of the published numbers (Table 9), so the data loading, model construction and evaluation code can be trusted. This step also caught a real bug: the framework’s default loader remaps COCO category ids, silently misaligning the pretrained classification head.

  Table 9 — Pretrained RF-DETR checkpoints re-evaluated on COCO val2017, against the published numbers.

  Sanity anchor	AP50:95	AP50
  RF-DETR-Small, box AP (ours)	52.47	71.32
  RF-DETR-Small, box AP (published)	53.0	72.1
  RF-DETR-Seg-Small, segm AP (ours)	42.50	65.92
  RF-DETR-Seg-Small, segm AP (published)	43.1	66.2