Actuators for physical human-robot interaction (pHRI) such as rehabilitation or assistive systems should generate the desired torque precisely. However, the resistive and inertia loads inherent in the actuators (e.g., friction, damping, and inertia) set challenges in the control of actuators in a force/torque mode. The resistive factors include nonlinear effects and should be considered in the controller design to generate the desired force accurately. Moreover, the uncertainties in the plant dynamics make the precise torque control difficult. In this paper, nonlinear control algorithms are exploited for a rotary series elastic actuator to generate the desired torque precisely in the presence of nonlinear resistive factors and modeling uncertainty. The sliding mode control smoothed by a boundary layer is applied to enhance the robustness for the modeling uncertainty without chattering phenomenon. In this paper, the rotary series elastic actuator (RSEA) is installed on the knee joint of an orthosis, and the thickness of the boundary layer is changed by gait phases in order to minimize the torque error without the chattering phenomenon. The performance of the proposed controller is verified by experiments with actual walking motions.