publication . Preprint . Article . 2016

Crowdsourcing quality control for Dark Energy Survey images

Peter Melchior; Erin S. Sheldon; Alex Drlica-Wagner; Eli S. Rykoff; T. M. C. Abbott; F. B. Abdalla; Sahar S. Allam; A. Benoit-Lévy; David Brooks; E. Buckley-Geer; ...
Open Access English
  • Published: 11 May 2016
  • Publisher: Elsevier
Abstract
We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the "DES exposure checker", it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net
Comment: 12 pages, 7 figures, accepted by Astronomy & Computing
Persistent Identifiers
Subjects
free text keywords: QB, Human-centered computing: Collaborative filtering, Information systems: Crowdsourcing, Surveys, [PHYS.ASTR]Physics [physics]/Astrophysics [astro-ph], surveys, information systems, crowdsourcing, humna-centered computing, collaborative filtering, RCUK, STFC, AST-1138766, /dk/atira/pure/subjectarea/asjc/3100/3103, Astronomy and Astrophysics, /dk/atira/pure/subjectarea/asjc/1700/1706, Computer Science Applications, Computer Science Applications, Astronomy and Astrophysics, Space and Planetary Science, [ PHYS.ASTR ] Physics [physics]/Astrophysics [astro-ph], Astrophysics - Instrumentation and Methods for Astrophysics, Data management, business.industry, business, World Wide Web, Computer science, Set (psychology), Collaborative filtering, Web application, Crowdsourcing, Host (network), Quality (business), media_common.quotation_subject, media_common, Information system
Funded by
NSF| Programs on Critical Problems in Physics, Astrophysics and Biophysics at the Aspen Center for Physics
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1066293
  • Funding stream: Directorate for Mathematical & Physical Sciences | Division of Physics
,
EC| COSMICDAWN
Project
COSMICDAWN
Understanding the Origin of Cosmic Structure
  • Funder: European Commission (EC)
  • Project Code: 306478
  • Funding stream: FP7 | SP2 | ERC
,
NSF| Collaborative Research: The Dark Energy Survey Data Management Operations
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1138766
  • Funding stream: Directorate for Mathematical & Physical Sciences | Division of Astronomical Sciences
,
EC| COGS
Project
COGS
Capitalizing on Gravitational Shear
  • Funder: European Commission (EC)
  • Project Code: 240672
  • Funding stream: FP7 | SP2 | ERC
,
EC| TESTDE
Project
TESTDE
Testing the Dark Energy Paradigm and Measuring Neutrino Mass with the Dark Energy Survey
  • Funder: European Commission (EC)
  • Project Code: 291329
  • Funding stream: FP7 | SP2 | ERC
22 references, page 1 of 2

Balbinot E., Santiago B., Girardi L., da Costa L. N., Maia M. A. G., Pellegrini P. S. S., Makler M., 2012, in Astronomical Society of the Pacific Conference Series Vol. 461, p. 287

Bertin E., Arnouts S., 1996, Astronomy and Astrophysics Supplement Series, 117, 393

Blackburn L. et al., 2008, Classical and Quantum Gravity, 25, 184004

Dark Energy Survey Collaboration 2005, arXiv:astro-ph/0510346

Desai S. et al., 2012, ApJ, 757, 83

Diehl H. T. et al., 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 9149, p. 91490V

Duda R. O., Hart P. E., 1972, Commun. ACM, 15, 11

Flaugher B. et al., 2015, AJ, 150, 150

Greenhill A., Holmes K., Lintott C., Simmons B., Masters K., Cox J., Graham G., 2014, in GAME-ON 2014 15th International Conference on Intelligent Games and Simulation, pp 15-24

Honscheid K. et al., 2012, in Software and Cyberinfrastructure for Astronomy II, pp 845112-845112-15

Hough P. V. C., 1959, in Proceedings, 2nd International Conference on HighEnergy Accelerators and Instrumentation, HEACC 1959 Vol. C590914, pp 554-558

Jones I., 1989, The Last Crusade

Kapadia A., Smith A., 2013, in Astronomical Society of the Pacific Conference Series Vol. 475, p. 299

Lintott C. J. et al., 2008, Monthly Notices of the Royal Astronomical Society, 389, 1179

Littmann M., Suomela T., 2014, Endeavour, 38, 130

22 references, page 1 of 2
2 research outcomes, page 1 of 1
Abstract
We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the "DES exposure checker", it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net
Comment: 12 pages, 7 figures, accepted by Astronomy & Computing
Persistent Identifiers
Subjects
free text keywords: QB, Human-centered computing: Collaborative filtering, Information systems: Crowdsourcing, Surveys, [PHYS.ASTR]Physics [physics]/Astrophysics [astro-ph], surveys, information systems, crowdsourcing, humna-centered computing, collaborative filtering, RCUK, STFC, AST-1138766, /dk/atira/pure/subjectarea/asjc/3100/3103, Astronomy and Astrophysics, /dk/atira/pure/subjectarea/asjc/1700/1706, Computer Science Applications, Computer Science Applications, Astronomy and Astrophysics, Space and Planetary Science, [ PHYS.ASTR ] Physics [physics]/Astrophysics [astro-ph], Astrophysics - Instrumentation and Methods for Astrophysics, Data management, business.industry, business, World Wide Web, Computer science, Set (psychology), Collaborative filtering, Web application, Crowdsourcing, Host (network), Quality (business), media_common.quotation_subject, media_common, Information system
Funded by
NSF| Programs on Critical Problems in Physics, Astrophysics and Biophysics at the Aspen Center for Physics
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1066293
  • Funding stream: Directorate for Mathematical & Physical Sciences | Division of Physics
,
EC| COSMICDAWN
Project
COSMICDAWN
Understanding the Origin of Cosmic Structure
  • Funder: European Commission (EC)
  • Project Code: 306478
  • Funding stream: FP7 | SP2 | ERC
,
NSF| Collaborative Research: The Dark Energy Survey Data Management Operations
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1138766
  • Funding stream: Directorate for Mathematical & Physical Sciences | Division of Astronomical Sciences
,
EC| COGS
Project
COGS
Capitalizing on Gravitational Shear
  • Funder: European Commission (EC)
  • Project Code: 240672
  • Funding stream: FP7 | SP2 | ERC
,
EC| TESTDE
Project
TESTDE
Testing the Dark Energy Paradigm and Measuring Neutrino Mass with the Dark Energy Survey
  • Funder: European Commission (EC)
  • Project Code: 291329
  • Funding stream: FP7 | SP2 | ERC
22 references, page 1 of 2

Balbinot E., Santiago B., Girardi L., da Costa L. N., Maia M. A. G., Pellegrini P. S. S., Makler M., 2012, in Astronomical Society of the Pacific Conference Series Vol. 461, p. 287

Bertin E., Arnouts S., 1996, Astronomy and Astrophysics Supplement Series, 117, 393

Blackburn L. et al., 2008, Classical and Quantum Gravity, 25, 184004

Dark Energy Survey Collaboration 2005, arXiv:astro-ph/0510346

Desai S. et al., 2012, ApJ, 757, 83

Diehl H. T. et al., 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 9149, p. 91490V

Duda R. O., Hart P. E., 1972, Commun. ACM, 15, 11

Flaugher B. et al., 2015, AJ, 150, 150

Greenhill A., Holmes K., Lintott C., Simmons B., Masters K., Cox J., Graham G., 2014, in GAME-ON 2014 15th International Conference on Intelligent Games and Simulation, pp 15-24

Honscheid K. et al., 2012, in Software and Cyberinfrastructure for Astronomy II, pp 845112-845112-15

Hough P. V. C., 1959, in Proceedings, 2nd International Conference on HighEnergy Accelerators and Instrumentation, HEACC 1959 Vol. C590914, pp 554-558

Jones I., 1989, The Last Crusade

Kapadia A., Smith A., 2013, in Astronomical Society of the Pacific Conference Series Vol. 475, p. 299

Lintott C. J. et al., 2008, Monthly Notices of the Royal Astronomical Society, 389, 1179

Littmann M., Suomela T., 2014, Endeavour, 38, 130

22 references, page 1 of 2
2 research outcomes, page 1 of 1
Any information missing or wrong?Report an Issue