The design of complex system architectures brings with it a number of challenging issues, among others large combinatorial design spaces. Optimization can be applied to explore the design space, however gradient-based optimization algorithms cannot be applied due to the mixed-discrete nature of the design variables. It is investigated how effective surrogate-based optimization algorithms are for solving the black-box, hierarchical, mixed-discrete, multi-objective system architecture optimization problems. Performance is compared to the NSGA-II multi-objective evolutionary algorithm. An analytical benchmark problem that exhibits most important characteristics of architecture optimization is defined. First, an investigation into algorithm effectiveness is performed by measuring how accurately a known Pareto-front can be approximated for a fixed number of function evaluations. Then, algorithm efficiency is investigated by applying various multi-objective convergence criteria to the algorithms and establishing the possible trade-off between result quality and function evaluations needed. Finally, the impact of hidden constraints on algorithm performance is investigated. The code used for this paper has been published.