Motivated by many social phenomena such as bird flocking and fish schooling, in this paper three types of constrained hybrid multiagent swarm optimization (HMSO) algorithms are presented to address the constrained optimization problem by incorporating the fly-back mechanism into the update formula for the particle's position. The original HMSO algorithm is proposed for solving unconstrained, continuous optimization problems by using a simple logic switching structure to achieve superior performance. However, this method cannot be directly used to solve discrete optimization problems like the binary programming. Since the application of the mixed-binary nonlinear programming (MBNLP) problem is widespread in many system engineering problems, it is necessary to develop an HMSO based optimization algorithm to address the mixed-binary optimization so that one can achieve the better performance for MBNLP with a simple algorithm structure. In this context, first a binary version of the constrained type of the HMSO algorithm is provided by introducing communication topologies for the particles to exchange their position information, which is well studied under multiagent coordination problems in control theory. By taking advantage of the fly-back mechanism dealing with constraints in optimization, a new architecture for HMSO is introduced to form a constrained HMSO algorithm for constrained optimization. Finally, we combine the proposed binary HMSO and constrained HMSO to create a modified HMSO algorithm to address the MBNLP problem. Several benchmark functions are used for the evaluation of the binary HMSO, constrained HMSO, and modified HMSO algorithm and compared with the standard particle swarm optimization algorithm.