Collecting the relevant list of patient phenotypes,known as deep phenotyping, can significantly improve the finaldiagnosis. As textual clinical reports are the richest source ofphenotypes information, their automatic extraction is a criticaltask. The main challenges of this Information Extraction (IE) taskare to identify precisely the text spans related to a phenotype andto link them unequivocally to referenced entities from a sourcesuch as the Human Phenotype Ontology (HPO).Recently, Language Models (LMs) have been the most suc-cessful approach for extracting phenotypes from clinical reports.Solutions such as PhenoBERT, relying on BERT or GPT, haveshown promising results when applied to datasets built on thehypothesis that most phenotypes are explicitly mentioned in thetext. However, this assumption is not always true in medicalgenetics. Hence, although the LMs carry powerful semanticabilities, their contributions are not clear compared to syntacticstring-matching steps that are used within the current pipelines.The goal of this study is to improve phenotype extraction fromclinical notes related to genetic diseases. Our contributions arethreefold: First, we provide a clear definition of the phenotypeextraction task from free text, along with a high-level overview ofthe involved functions. Second, we conduct an in-depth analysisof PhenoBERT, one of the best existing solutions, to evaluate theproportion of phenotypes predicted with simple string-matching.Third, we demonstrate how utilizing and incorporating largelanguage models (LLMs) for span detection step can improveperformance especially with implicit phenotypes. In addition, thisexperiment revealed that the annotations of existing dataset arenot exhaustive, and that LLM can identify relevant spans missedby human labelers.