Silicon presents itself as a high-capacity anode material for lithium-ion batteries with a promising future. The high ability for lithiation comes along with massive volume changes and a problematic voltage hysteresis, causing reduced efficiency, detrimental heat generation, and a complicated state-of-charge estimation. During slow cycling, amorphous silicon nanoparticles show a larger voltage hysteresis than after relaxation periods. Interestingly, the voltage relaxes for at least several days, which has not been physically explained so far. We apply a chemo-mechanical continuum model in a core-shell geometry interpreted as a silicon particle covered by the solid-electrolyte interphase to account for the hysteresis phenomena. The silicon core (de)lithiates during every cycle while the covering shell is chemically inactive. The visco-elastoplastic behavior of the shell explains the voltage hysteresis during cycling and after relaxation. We identify a logarithmic voltage relaxation, which fits with the established Garofalo law for viscosity. Our chemo-mechanical model describes the observed voltage hysteresis phenomena and outperforms the empirical Plett model. In addition to our full model, we present a reduced model to allow for easy voltage profile estimations. The presented results support the mechanical explanation of the silicon voltage hysteresis with a core-shell model and encourage further efforts into the investigation of the silicon anode mechanics.