This paper proposes a novel continuous-time algorithm for solving optimization problems by leveraging the concept of so-called constraint-based control to guarantee a specific convergence speed. Conventional distributed optimization algorithms evaluate the convergence speed by the order of the convergence, which does not allow direct specification of the convergence speed. Moreover, the continuous-time version of the distributed optimization does not provide any convergence speed guarantee, with a few exceptions under a strong assumption. To address the issue, we incorporate constraint-based control into a typical continuous-time distributed optimization algorithm, namely primal-dual dynamics. Specifically, we introduce a constraint that ensures the desired increase rate of the dual function in the optimization problem. Our proposed method determines the update rules for the dual variables of the optimization problem by leveraging the present constraint. The proposed method is also shown to preserve the inherent distributed nature of the primal-dual dynamics. A numerical example demonstrates the effectiveness of our approach, highlighting not only the operability of the convergence speed but also the improvement of the convergence performance compared to the original primal-dual dynamics.