This paper proposed an optimization method to minimize the building energy consumption and visual discomfort for a passive building in Shanghai, China. A total of 35 design parameters relating to building form, envelope properties, thermostat settings, and green roof configurations were considered. First, the Latin hypercube sampling method (LHSM) was used to generate a set of design samples, and the energy consumption and visual discomfort of the samples were obtained through computer simulation and calculation. Second, four machine learning prediction models, including stepwise linear regression (SLR), back-propagation neural networks (BPNN), support vector machine (SVM), and random forest (RF) models, were developed. It was found that the BPNN model performed the best, with average absolute relative errors of 3.27% and 1.25% for energy consumption and visual comfort, respectively. Third, six optimization algorithms were selected to couple with the BPNN models to find the optimal design solutions. The multi-objective ant lion optimization (MOALO) algorithm was found to be the best algorithm. Finally, optimization with different groups of design variables was conducted by using the MOALO algorithm with the associated outcomes being analyzed. Compared with the reference building, the optimal solutions helped reduce energy consumption up to 34.8% and improved visual discomfort up to 100%.