
Words and phrases acquire meaning from the way they are used in society, from their relative semantics to other words and phrases. For computers the equivalent of `society' is `database,' and the equivalent of `use' is `way to search the database.' We present a new theory of similarity between words and phrases based on information distance and Kolmogorov complexity. To fix thoughts we use the world-wide-web as database, and Google as search engine. The method is also applicable to other search engines and databases. This theory is then applied to construct a method to automatically extract similarity, the Google similarity distance, of words and phrases from the world-wide-web using Google page counts. The world-wide-web is the largest database on earth, and the context information entered by millions of independent users averages out to provide automatic semantics of useful quality. We give applications in hierarchical clustering, classification, and language translation. We give examples to distinguish between colors and numbers, cluster names of paintings by 17th century Dutch masters and names of books by English novelists, the ability to understand emergencies, and primes, and we demonstrate the ability to do a simple automatic English-Spanish translation. Finally, we use the WordNet database as an objective baseline against which to judge the performance of our method. We conduct a massive randomized trial in binary classification using support vector machines to learn categories based on our Google distance, resulting in an a mean agreement of 87% with the expert crafted WordNet categories.
15 pages, 10 figures; changed some text/figures/notation/part of theorem. Incorporated referees comments. This is the final published version up to some minor changes in the galley proofs
FOS: Computer and information sciences, Computer Science - Machine Learning, Computer Science - Computation and Language, I.2.4, Computer Science - Artificial Intelligence, I.2.7, Databases (cs.DB), Computer Science - Information Retrieval, Machine Learning (cs.LG), Artificial Intelligence (cs.AI), Computer Science - Databases, Computation and Language (cs.CL), Information Retrieval (cs.IR), I.2.4; I.2.7
FOS: Computer and information sciences, Computer Science - Machine Learning, Computer Science - Computation and Language, I.2.4, Computer Science - Artificial Intelligence, I.2.7, Databases (cs.DB), Computer Science - Information Retrieval, Machine Learning (cs.LG), Artificial Intelligence (cs.AI), Computer Science - Databases, Computation and Language (cs.CL), Information Retrieval (cs.IR), I.2.4; I.2.7
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 1K | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Top 1% | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Top 0.1% | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Top 0.1% |
