Open source projects are most often evaluated by potential contributors and consumers using metrics that describe a level of activity within the project because those measurements are available. The principle question in the minds of most evaluators, however, is ”How healthy and sustainable is this project in the context of its competitors or dependent projects”? Limitations of current analysis methods focused on trace data alone are discussed, and reviewed in depth. Next, our methods for conducting engaged field research, developing metrics standards as part of a corporate communal partnership, and molding tools that evolve through a reflexive discourse with practitioners using standard metrics is framed as an approach to consider for examining open source software health and sustainability. Researchers, in particular, need tools for increasing the feasibility of comprehensive, multi-project health and sustainability studies, and connecting trace data with human experience. From a practice perspective, these same conditions are increasing the difficulty organizations and individuals engaged in open source face when trying to understand the status, condition, and health of a particular project, the project’s ecosystem or ecosystems emerging around their specific project context. This study exam- ines the work of a Linux Foundation working group, CHAOSS (Community Health Analytics Open Source Software) during the first four years of the formation. The paper concludes with ex- amples of CHAOSS metrics operationalized in partnership with corporate collaborators in a manner that emphasizes comparison, transparency, trajectory and visualization as components for discursive, evolutionary understanding of open source software health.