Abstract The main purpose of this study was to examine the reliability of S, a lexical richness measure using word frequency lists, by comparing it with three other lexical richness measures: Beyond 2000 ( Laufer, 1994 , Laufer, 1995 ), P_Lex ( Meara & Bell, 2001 ), and Advanced Guiraud ( Daller, van Hout, & Treffers-Daller, 2003 ). These measures were developed in order to assess the vocabulary use of learners on the basis of word frequency lists. We investigated the reliability of these four lexical richness measures in 38 essays written by Japanese learners of English. All the essays were segmented into five lengths. The results of our assessment suggest that S is the most robust of the four measures for short texts. S and P_Lex were found to yield the constant values independent of the text length studied, whereas Beyond 2000 and Advanced Guiraud values tended to increase as the text length did. Further, only S attained adequate reliability for short texts of around 200 words; the other measures proved to need much longer texts to obtain the same level of reliability. These results suggest that S is a more suitable measure for research and diagnostic purposes with short L2 texts, as compared to the major alternatives.