Abstract The specificity of a T-cell receptor (TCR) repertoire determines personalized immune capacity. Existing methods have modeled the qualitative aspects of TCR specificity, while the quantitative aspects remained unaddressed. We developed a package, TCRanno, to quantify the specificity of TCR repertoires. We created deep-learning-based, epitope-aware vector embeddings to infer individual TCR specificity. Then we aggregated clonotype frequencies of TCRs to obtain a quantitative profile of repertoire specificity at epitope, antigen and organism levels. Applying TCRanno to 4195 TCR repertoires revealed quantitative changes in repertoire specificity upon infections, autoimmunity and cancers. Specifically, TCRanno found cytomegalovirus-specific TCRs in seronegative healthy individuals, supporting the possibility of abortive infections. TCRanno discovered age-accumulated fraction of severe acute respiratory syndrome coronavirus 2 specific TCRs in pre-pandemic samples, which may explain the aggressive symptoms and age-related severity of coronavirus disease 2019. TCRanno also identified the encounter of Hepatitis B antigens as a potential trigger of systemic lupus erythematosus. TCRanno annotations showed capability in distinguishing TCR repertoires of healthy and cancers including melanoma, lung and breast cancers. TCRanno also demonstrated usefulness to single-cell TCRseq+gene expression data analyses by isolating T-cells with the specificity of interest.