The automatic detection of dangerous goods in X-ray baggage images is a crucial task in the field of industrial security. This paper proposes a novel dangerous goods detection network (DGDN) designed to address the challenges of detecting dangerous goods in complex X-ray baggage images. In the network, a multilevel refinement (MR) module is designed to improve multi-level feature representation, enabling the network to capture varied goods sizes and overlapping structures effectively. Specifically, the selective spatial-channel adaptive module (SSCAM) introduces dynamically adaptive weights across spatial and channel dimensions, allowing the network to focus on critical features more effectively and thereby improving detection accuracy. In the experiments, the performance of the proposed DGDN is evaluated by using the mean average precision (mAP) metric on the SIXray and PIDray datasets. The experimental results demonstrate that the proposed DGDN achieves mAP scores of 91.9% and 69.4% on the SIXray and PIDray datasets, respectively. Compared with some state-of-the-art methods, our network improves performance by 5–10% while reducing parameters and increasing computational efficiency. These indicate that DGDN has good potential application capabilities.