For supplying the reference to the spaced-based optical observation system design and performance analysis, the space-based observation mode for space debris is established. Considering the geometry size, the material characteristics, and the distribution region of the debris, the visual magnitude of space debris is calculated in the condition of different detection range and different phase angle. Based on the typical instance, the simulation analysis of the space debris detection ability of space-based optical observation system, which is in LEO orbit and Sub-GEO orbit respectively, is carried out. The results show that the LEO spaced-based optical observation system having an aperture of 25cm can detect the LEO 3cm space debris, which is 100km far away and has the relative velocity of 1km/s, and can also detect the GEO 1m space object, which is 37000km far away and has the relative velocity of 4km/s. The Sub-GEO spaced-based optical observation system having an aperture of 25cm can detect the GEO 1cm space debris, which is 800km far away and has the relative velocity of 15m/s.