We study the leptogenesis scenario in models with multi-Higgs doublets. It is pointed out that the washing-out process through the effective dimension-five interactions, which has not been taken into account seriously in the conventional scenario, can be effective, and the resultant baryon asymmetry can be exponentially suppressed. This fact implies new possible scenario where the observed baryon asymmetry is the remnant of the washed out lepton asymmetry which was originally much larger than the one in the conventional scenario. Our new scenario is applicable to some neutrino mass matrix models which predict too large CP-violating parameter and makes them viable through the washing-out process.