With the emergence of information technology, mobile-based public participation (MPP) has been introduced into urban planning practices. How to evaluate this new approach of public participation is important for supporting decision-making process and achieving effective public participation. Integrating traditional evaluation criteria and Internet product operation strategies, this study proposes an evaluation framework for MPP, which includes seven indicators from three dimensions: preparation, process and outcome. Taking PinStreet, a widely-used MPP platform in China, as a case, this study evaluates the effectiveness of seven pilot projects with a variety of themes and scales. The evaluation results show that factors lying in the aspects of environment, participant and organizer have a significant impact on the effectiveness of MPP. Finally, policy suggestions are proposed to promote the development of MPP.