Design and Analysis of the NIPS 2016 Review Process
Design and Analysis of the NIPS 2016 Review Process
Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year. The massive scale as well as rapid growth of the conference calls for a thorough quality assessment of the peer-review process and novel means of improvement. In this paper, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers. Our goal is to check the soundness of the review process, and provide insights that may be useful in the design of the review process of subsequent conferences.
[INFO.INFO-AI] Computer Science [cs]/Artificial Intelligence [cs.AI], Social and Information Networks (cs.SI), FOS: Computer and information sciences, Applications of statistics to social sciences, Computer Science - Machine Learning, consistency, Learning and adaptive systems in artificial intelligence, ordinal, Computer Science - Digital Libraries, Computer Science - Social and Information Networks, Machine Learning (stat.ML), Informational aspects of data analysis and big data, Machine Learning (cs.LG), Statistics - Machine Learning, peer review, Digital Libraries (cs.DL), post hoc analysis, Neural Information Processing System (NIPS)
[INFO.INFO-AI] Computer Science [cs]/Artificial Intelligence [cs.AI], Social and Information Networks (cs.SI), FOS: Computer and information sciences, Applications of statistics to social sciences, Computer Science - Machine Learning, consistency, Learning and adaptive systems in artificial intelligence, ordinal, Computer Science - Digital Libraries, Computer Science - Social and Information Networks, Machine Learning (stat.ML), Informational aspects of data analysis and big data, Machine Learning (cs.LG), Statistics - Machine Learning, peer review, Digital Libraries (cs.DL), post hoc analysis, Neural Information Processing System (NIPS)
selected citations These citations are derived from selected sources.This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).0 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average
Citations provided by BIP!Popularity provided by BIP!