The architecture and evolution of planetary systems are shaped in part by stellar flybys. Within this context, we look at stellar encounters which are too weak to immediately destabilize a planetary system but are nevertheless strong enough to measurably perturb the system's dynamical state. We estimate the strength of such perturbations on secularly evolving systems using a simple analytic model and confirm those estimates with direct N-body simulations. We then run long-term integrations and show that even small perturbations from stellar flybys can influence the stability of planetary systems over their lifetime. We find that small perturbations to the outer planets' orbits are transferred between planets, increasing the likelihood that the inner planetary system will destabilize. Specifically, our results for the Solar System show that relative perturbations to Neptune's semi-major axis of order 0.1% are strong enough to increase the probability of destabilizing the Solar System within 5 Gyrs by one order of magnitude.