Knee joint vibration signals are very useful for computer-aided analysis of the pathological conditions in the knee. In a vibration arthrometry test, the legs of patients with knee joint disorders may tremble due to the reaction of pain, which causes the baseline wander that may affect the diagnostic decision making in medical study. This paper presents a new type of cascade moving average filter with hierarchical layers to remove the baseline wander in the raw knee joint vibration signals. The first layer of the cascade filter contains two moving averaging operators with the same order. The five tail inputs of the first moving averaging operator are overlapping with the beginning inputs of the successive operator. The piecewise linear trends estimated by the moving average operators in the first layer were smoothed in the final cascade filter output. The simulation results showed that the cascade filter can effectively remove the baseline wander in the raw knee joint vibration signals.