We consider the problem of automatically separating coal and coal gangue based on computer vision and design a coal and coal gangue separation system framework based on video. Grayscale histogram, fractal dimension and energy value are extracted as ore features. Then we design a 4-layer Levenberg Marquart BP Neural Network to implement multi-feature fusion. Test results demonstrate that the system has well performance on separation accuracy and its processing speed achieves real-time. It can be used in automatic statistics for open-pit coal output. Moreover, the extended feature vector can be used in coal separation on conveyor belt combined with other automation technology.