Dengue (DENV), a neglected tropical disease, is a globally distributed arboviral (genus \emph{Flavivirus}) pathogen primarily spread by \emph{Aedes} mosquitoes and infecting approximately 390 million individuals annually. A challenge to successful control of DENV is that after primary infection (or vaccination) due to waning, secondarily infected patients (or vaccinated individuals) can have an elevated risk of severe Dengue due to a phenomenon known as antibody-dependent enhancement (ADE), that is: preexisting cross-reactive IgG antibody concentrations can increase dengue severity. In this study, we first robustly parameterize a unified within-host viral and immune kinetics model to viral kinetics data for serotypes DENV1, 2, and 3 collected at the Hospital for Tropical Diseases (Ho Chi Minh City, Vietnam) while allowing independent variation in infection start time among hosts. Our model recapitulates the data well, including cross-reactive antibody concentration-dependent enhanced severity in secondary infections, and captures empirically observed differences between primary and secondary DENV infections, such as time to peak viral load, duration of viremia, and maximum viral titer. Our parameterization also captures meaningful differences in serotype-specific kinetic parameters that drive these differences. Subsequently, we (i) show that variation in initial IgG antibody concentration is sufficient to mechanistically explain the observed differences between primary and secondary infection in terms of the time course of events across serotypes and (ii) leverage our modeling results paired with long-term NS1-specific IgG antibody decay data from Recife, Northeast Brazil, to estimate the half-life of Dengue IgG antibodies and the time frame of the risk window for escalated disease severity due to ADE.