The computational complexity of existing multi-symbol detection (MSD) algorithms grows exponentially as the observation intervals increase, resulting in difficulties in algorithm implementation for detecting pulse code modulation/frequency modulation (PCM/FM) signals, especially for multi-channel signals. To address the challenges, we proposed a low-complexity MSD algorithm based on the averaged matched filtering. The proposed algorithm groups the local reference signals based on the different importance levels of the middle and edge bits in the correlation operations and averages the edge bits, leading to a considerable decrease in matching branches. Furthermore, it leverages the phase symmetry, and the proposed algorithm retains half of the averaged local reference signals for the matching operation, thus further reducing the matching branches. The proposed algorithm reduces the storage of the local signals and correlation operations to one-eighth compared to the traditional MSD algorithm under different observation lengths. Additionally, based on the structure of multiplexed FIR filters, the proposed algorithm optimizes single-channel single-coefficient FIR filters into four-channel double-coefficient FIR filters, further reducing the hardware resource consumption by approximately 25%. The simulation results showed that the proposed algorithm achieved demodulation performance comparable to the traditional MSD algorithms while reducing the computational complexity by 87.5%. Compared to the decision-feedback MSD algorithm, it achieves higher demodulation gain with a 75% complexity reduction. The Field Programmable Gate Array (FPGA) platform implementation results showed that the proposed algorithm reduces hardware resource consumption by nearly 90% compared with the traditional algorithm, and the hardware demodulation performance loss is less than 1 dB compared with the simulation results.