Hyperspectral image classification (HSIC) has been considerably improved by many lightweight and efficient networks developed to meet real-time application needs and computing resource limitations. However, theoretical floating-point operations alone are not enough to evaluate real-time quality, especially in scenarios where inference latency is highly influenced by memory access cost and hardware characteristics. To address these challenges, we create a low-latency-oriented network architecture for HSIC, which is adaptable to any dataset without requiring architectural adjustments. First, starting from a pretrained backbone network, we deploy a latency-oriented network architecture search, with search flexibility spanning multiple levels of the model, and add inference latency as a model evaluator to identify low-latency subnetwork architectures adapted to hyperspectral data. Moreover, we develop a computational efficiency model that can anticipate and evaluate the peak performance of operators that use hyperspectral input. Based on this, we introduce a split convolution approach that replaces depthwise convolution, resulting in enhanced arithmetic intensity without significant increase in latency. The networks created by implementing our strategies are both compact in structure and hardware-friendly. After testing on three different datasets, the proposed networks achieve significantly better inference speed and energy-saving ability over advanced classification networks and lightweight models, while maintaining an equivalent or even better classification performance.