The Truth, the Whole Truth, and Nothing but the Truth: A Pragmatic Guide to Assessing Empirical Evaluations

Article English OPEN
Blackburn, Stephen M; Diwan, Amer; Hauswirth, Mattias; Sweeney, Peter F; Amaral, Jose Nelson; Brecht, Tim; Bulej, Lubomr; Click, Cliff; Eeckhout, Lieven; Fischmeister, Sebastian; Frampton, Daniel; Hendren, Laurie J; Hind, Michael; Hosking, Antony L; Jones, Richard E.; Kalibera, Tomas; Keynes, Nathan; Nystrom, Nathaniel; Zeller, Andreas;

An unsound claim can misdirect a field, encouraging the pursuit of unworthy ideas and the abandonment of promising ideas. An inadequate description of a claim can make it difficult to reason about the claim, for example to determine whether the claim is sound. Many prac... View more
  • References (33)
    33 references, page 1 of 4

    2013. Unreliable research. Trouble at the lab. The Economist (19 October 2013). news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

    Phillip G. Armour. 2000. The Five Orders of Ignorance. Commun. ACM 43, 10 (Oct. 2000), 17-20. DOI:

    David H. Bailey. 2009. Misleading performance claims in parallel computation. In 46th Annual Design Automation Conference. ACM, ACM Press, New York, NY, 528-33. 1630049

    David H. Bailey, Jonathan M. Borwein, and Victoria Stodden. 2014. Facilitating reproducibility in scientific computing: Principles and practice. (30 June 2014).

    M. Baker. 2012. Independent labs to verify high-profile papers: Reproducibility Initiative aims to speed up preclinical research. Nature : News (14 August 2012). doi:10.1038/nature.2012.11176

    Sharon Begley. 2012. More trial, less error - An effort to improve scientific studies. Reuters (14 August 2012).

    Stephen M. Blackburn, Kathryn S. McKinley, Robin Garner, Chris Hoffmann, Asjad M. Khan, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z. Guyer, Martin Hirzel, Antony Hosking, Maria Jump, Han Lee, J. Eliot B. Moss, Aashish Phansalkar, Darko Stefanovik, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2008. Wake Up and Smell the Coffee: Evaluation Methodology for the 21st Century. Commun. ACM 51, 8 (Aug. 2008), 83-89. DOI:

    Philippe Bonnet, Stefan Manegold, Matias Bjørling, Wei Cao, Javier Gonzalez, Joel Granados, Nancy Hall, Stratos Idreos, Milena Ivanova, Ryan Johnson, David Koop, Tim Kraska, Rene´ M u¨ller, Dan Olteanu, Paolo Papotti, Christine Reilly, Dimitris Tsirogiannis, Cong Yu, Juliana Freire, and Dennis Shasha. 2011. Repeatability and Workability Evaluation of SIGMOD 2011. SIGMOD Rec. 40, 2 (Sept. 2011), 45-48. DOI:

    Frederick P. Brooks, Jr. 1996. The Computer Scientist As Toolsmith II. Commun. ACM 39, 3 (March 1996), 61-68. DOI:

    D. Buytaert, A. Georges, M. Hind, M. Arnold, L. Eeckhout, and K. De Bosschere. 2007. Using hpm-sampling to drive dynamic compilation. In Proceedings of the 22nd annual ACM SIGPLAN conference on Objectoriented programming systems and applications. ACM, ACM Press, New York, NY, 553-568. http://dx.

  • Metrics
Share - Bookmark