Facial micro-expression refers to split-second muscle changes in the face, indicating that a person is either consciously or unconsciously suppressing their true emotions. Although these expressions are constantly occurring on people faces, they were easily ignored by people with the eye blinking. That is to say, most people don't notice them and it is the true representation of people emotions and mental health. Accordingly, both of psychologists and computer scientists (in the fields of computer vision and machine learning in particular) pay attention to it owing to their promising applications in various fields (e.g. Mental clinical diagnosis and therapy, affective computing). However, detecting micro-expression is still difficult task. Here, we proposed a novel approach based on deep multi-task learning method with the HOOF(Histograms of oriented optical flow) feature for micro-expression detection. We investigated a deep multi-task learning method for facial landmark localization and split the facial area into regions of interest(ROIS). Faical micro-expression are generated by the movement of facial muscles, so we combined robust optical flow approach with the HOOF feature for evaluating the direction of movement of facial muscles. Through some experiments on CASME spontaneous micro-expression database, we can demonstrate our proposal method can achieve good performance for detecting micro-expression.