Tongue diagnosis is one of the most important diagnostic methods in Oriental Medical Science (OMS). This diagnosis is painless and non-invasive method. However, it is not easy to cultivate skillful doctors. As one of the reasons, definition of tongue color is rather subjective and sensuous measure and color isn't related to quantitative physical value. It is, therefore, necessary to associate tongue color with physical numerical value. There are two problems to overcome the issue. 1) It is necessary for diagnosis to extract a region for diagnosis from entire picture because a tongue picture consists of two regions, a tongue and a background. 2) Associate tongue color with physical numerical value. For extracting tongue region, we used Progressive LiveWire method that is an Active Contour Model. And, for associating tongue color with physical measurement, we propose a hierarchical method. We use static rule and support vector machine for clustering colors. The performance of developed system is improved compared with an early developed one. In addition, the developed system did not make a critical incorrect discernment that causes incorrect choice about inspection in the layer of rule base. In this research average color appraisal is done from the region of 37 points. But, color judgment in the literature with the judgment by the eye of the human, has always done average judgment with not to limit, there is also a possibility some weight attaching being done. Therefore, from either one enabling the mass data and the comparison with the group of specialists is necessary as an appraisal.