In order to solve the detection problem of bad real-time performance and robustness in complex scene, a new method for soft cascade classifier based on SVM was built. The image features can be extracted by the algorithm of using ORBP feature descriptor. Then, based on efficiently combining manifold features and cascaded threshold, a multistage classifier frame is introduced in detail. To ensure the rationality of sample selection, positive and negative samples are randomly selected by fuzzy evaluating for training classifier. Furthermore, the shi-tomasi corner detection and Median-Flow tracking are applied to improve detection performance. The proposed algorithm is evaluated on the object detection tasks of actual scene and PASCAL VOC 2010, and the excellent performance is achieved. In this paper, a detection algorithm is proposed to construct the multi-level classification threshold by adaptive feature selection. Experiments demonstrate that the proposed approach can greatly reduce eventually produce image detection window, and have good robustness and real-time performance.