We report on an evaluation of the effectiveness of considering a user's familiarity with a topic in improving information retrieval performance. This approach to personalization is based on previous results indicating differences in user search behavior and judgments according to his/her familiarity to the topic explored, and to research on using implicit sources of evidence to determine the user's context and preferences. Our attempt was to relate a topic-dependent concept and measure, familiarity with the topic, with topic-independent measures of documents such as readability, concreteness/abstractness, and specificity/generality. Contrary to our expectations, a user’s familiarity with a topic has no effect on the utility of readability or concrete/abstract scoring. We are encouraged, however, to find that high readability had a positive effect on search results, regardless of a user’s familiarity with a topic.