Since its creation 350 years ago, the scientific peer-reviewed journal has become the central and most important form of scholarly communication in the natural sciences and medicine. Although the digital revolution has facilitated and accelerated the publishing process by moving from print to online, it has not changed the scientific journal and scholarly communication as such. Today publications and citations in peer-reviewed journals are considered as indicators of scientific productivity and impact and used and misused in research evaluation. As scholarly communication is becoming more open and diverse and manuscripts, data, presentations and code are shared online, the altmetrics and open science movement demand the adaption of evaluation practices. Parallels are drawn between the early days of bibliometrics and current altmetrics research highlighting possibilities and limitations of various metrics and warning against adverse effects.