Illusions of Progress?: Why hyperspectral deep learning needs modern benchmarks

Benchmark datasets have been the backbone of hyperspectral image classification research, with classics such as Indian Pines (IP), Salinas (SA), Pavia (PA), Pavia University (PU), Pavia Center (PC), Kennedy Space Center (KSC), and Botswana (BO) providing the foundation for decades of progress. These datasets have supported numerous algorithmic advances, yet the rapid evolution of deep learning now exposes their shortcomings: they are limited in scale, lack diversity, and have been reused to the point of saturation. As a result, new methods are often judged on benchmarks that no longer capture the complexity of real-world scenarios, leading to inconsistent evaluations and the illusion of steady progress. This article offers a critical review of hyperspectral benchmarking, tracing the field’s reliance on classical datasets and revealing how their constraints shape, and sometimes distort, methodological outcomes. Through experimental analyses with state-of-the-art deep learning models, we illustrate how existing practices can bias comparisons and obscure meaningful conclusions. Our aim is not to rank methods but to demonstrate the urgent need for larger, more representative, and rigorously designed datasets. We close with a forward-looking perspective: to sustain meaningful progress, the hyperspectral remote sensing community must embrace modern benchmarks that reflect real-world diversity and align with emerging learning paradigms. By linking the historical trajectory of datasets to current methodological challenges, including issues of generalization, information leakage, and benchmark reliability, this review article provides timely guidance for building the next generation of benchmarks and ensuring that future research delivers reliable advances.