It has been difficult to identify genes affecting drug response to Selective Serotonin Reuptake Inhibitors (SSRIs). We used multiple cross-sectional assessments of depressive symptoms in a population-based study to identify potential genetic interactions with SSRIs as a model to study genetic variants associated with SSRI response. This study, embedded in the prospective Rotterdam Study, included all successfully genotyped participants with data on depressive symptoms (CES-D scores). We used repeated measurement models to test multiplicative interaction between genetic variants and use of SSRIs on repeated CESD scores. Besides a genome-wide analysis, we also performed an analysis which was restricted to genes related to the serotonergic signaling pathway. A total of 273 out of 14,937 assessments of depressive symptoms in 6443 participants, use of an SSRI was recorded. After correction for multiple testing, no plausible loci were identified in the genome-wide analysis. However, among the top 10 independent loci with the lowest p-values, findings within two genes (FSHR and HMGB4) might be of interest. Among 26 genes related to the serotonergic signaling pathway, the rs6108160 polymorphism in the PLCB1 gene reached statistical significance after Bonferroni correction (p-value = 8.1e-5). Also, the widely replicated 102C > T polymorphism in the HTR2A gene showed a statistically significant drug-gene interaction with SSRI use. Therefore, the present study suggests that drug-gene interaction models on (repeated) cross-sectional assessments of depressive symptoms in a population-based study can identify potential loci that may influence SSRI response.