As the high carbon emission entities in the city, laying photovoltaic (PV) panels for public buildings is an effective way to reduce building carbon emissions. Meanwhile, public buildings play an important role as charging station access for the explosive growth of electric vehicles. However, the disorderly charging behavior of single-phase charging piles exacerbates the existing three-phase unbalance inside the buildings, which in turn affects operating costs and PV consumption. Energy storage system (ESS) configuration is considered an effective solution. Thus, An ESS configuration strategy is proposed for public buildings aiming at PV local consumption and three-phase unbalance management. To quantify the correlation between multiple loads and PV output, an improved affinity-propagation clustering algorithm based on the spatial weighted matrix distance is developed to obtain operational typical power supply-demand modes. Based on the construction of the three-phase power supply system with both single-phase and three-phase ESSs, a bi-level planning model is formulated for the configuration and operation optimization of ESSs inside the public building. The upper-level problem aims to minimize the life cycle cost of ESS allocation. The lower-level model deals with the coordinated economic scheduling of single-phase ESS and three-phase ESS under the obtained typical operational modes. Numerical results show the effectiveness and rationality of the proposed clustering algorithm and ESS configuration strategy.