search
Include:
3 Research products, page 1 of 1

Relevance
arrow_drop_down
  • Open Access
    Authors: 
    Anneke Zuiderwijk; Keith G. Jeffery; Marijn Janssen;
    Publisher: JEDEM Journal of e-Democracy and Open Government
    Country: Netherlands

    Public and private organizations increasingly release their data to gain benefits such as transparency and economic growth. The use of these open data can be supported and stimulated by providing considerable metadata (data about the data), including discovery, contextual and detailed metadata. In this paper we argue that metadata are key enablers for the effective use of Linked Open Data (LOD). We illustrate the potential of metadata by 1) presenting an overview of advantages and disadvantages of metadata derived from literature, 2) presenting metadata requirements for LOD architectures derived from literature, workshops and a questionnaire, 3) describing a LOD metadata architecture that meets the requirements and 4) showing examples of the application of this architecture in the ENGAGE project. The paper shows that using metadata with the appropriate metadata architecture can yield considerable benefits for LOD publication and use, including improving find ability, accessibility, storing, preservation, analysing, comparing, reproducing, finding inconsistencies, correct interpretation, visualizing, linking data, assessing and ranking the quality of data and avoiding unnecessary duplication of data. The Common European Research Information Format (CERIF) can be used to build the metadata architecture and achieve the advantages.

  • Research software . 2020
    Open Access
    Authors: 
    Clausen, Alexander; Weber, Dieter; Ruzaeva, Karina; Müller-Caspary, Knut; Dunin-Borkowski, Rafal E.;
    Publisher: Zenodo
    Project: EC | VIDEO (780487), EC | CritCat (686053), EC | ESTEEM3 (823717)

    {"references": ["[Weber2018] Weber, Dieter (2018): Development of {IT} system and {TEM} camera performance. Zenodo. 10.5281/zenodo.2450624", "[Bon2012] Pierre Bon and Serge Monneret and Benoit Wattellier (2012): Noniterative boundary-artifact-free wavefront reconstruction from its derivatives. The Optical Society. 10.1364/ao.51.005698", "[K2IS2018] {{Gatan Inc.}} (2018): {K2} {IS} camera. https://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera", "[Clausen2018] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1477847", "[Sauter2013] Nicholas K. Sauter and Johan Hattne and Ralf W. Grosse-Kunstleve and Nathaniel Echols (2013): New Python-based methods for data processing. International Union of Crystallography ({IUCr}). 10.1107/s0907444913000863", "[Koennecke2015] Mark K\u00f6nnecke and Frederick A. Akeroyd and Herbert J. Bernstein and Aaron S. Brewster and Stuart I. Campbell and Bj\u00f6rn Clausen and Stephen Cottrell and Jens Uwe Hoffmann and Pete R. Jemian and David M\u00e4nnicke and Raymond Osborn and Peter F. Peterson and Tobias Richter and Jiro Suzuki and Benjamin Watts and Eugen Wintersberger and Joachim Wuttke (2015): The {NeXus} data format. International Union of Crystallography ({IUCr}). 10.1107/s1600576714027575", "[Fangohr2018] Fangohr, Hans and Aplin, Steve and Barty, Anton and Beg, Marijan and Bondar, Valerii and Boukhelef, Djelloul and Brockhauser, Sandor and Danilevski, Cyril and Ehsan, Wajid and Esenov, Sergey and Flucke, Gero and Giovanetti, Gabriele and Goeries, Dennis and Hauf, Steffen and Heisen, Burkhard and Hickin, David and Khakhulin, Dmitry and Klimovskaia, Anna and Kluyver, Thomas and Kuhn, Manuela and Kuster, Markus and Lang, Philipp-Michael and Maia, Luis and Mariani, Valerio and Mekinda, Leonce and Michelat, Thomas and Parenti, Andrea and Previtali, Gianpietro and Santos, Hugo and Silenzi, Alessandro and Sztuk-Dambietz, Jolanta and Szuba, Janusz and Teichmann, Martin and Weger, Kerstin and Wiggins, John and Wrona, Krzysztof and Xu, Chen (2018): Data Analysis Support in Karabo at European XFEL. JACoW, Geneva, Switzerland. 10.18429/jacow-icalepcs2017-tucpa01", "[Jesse2016] S. Jesse and M. Chi and A. Belianinov and C. Beekman and S. V. Kalinin and A. Y. Borisevich and A. R. Lupini (2016): Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography. Springer Nature. 10.1038/srep26348", "[Belianinov2015] Alex Belianinov and Rama Vasudevan and Evgheni Strelcov and Chad Steed and Sang Mo Yang and Alexander Tselev and Stephen Jesse and Michael Biegalski and Galen Shipman and Christopher Symons and Albina Borisevich and Rick Archibald and Sergei Kalinin (2015): Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets. Springer Nature. 10.1186/s40679-015-0006-6", "[Krajnak2016] Matus Krajnak and Damien McGrouther and Dzmitry Maneuski and Val O{\\textquotesingle} Shea and Stephen McVitie (2016): Pixelated detectors and improved efficiency for magnetic imaging in {STEM} differential phase contrast. Elsevier {BV}. 10.1016/j.ultramic.2016.03.006", "[Yang2015] H Yang and L Jones and H Ryll and M Simson and H Soltau and Y Kondo and R Sagawa and H Banba and I MacLaren and P D Nellist (2015): 4D {STEM}: High efficiency phase contrast imaging using a fast pixelated detector. {IOP} Publishing. 10.1088/1742-6596/644/1/012032", "[Sagawa2017] Ryusuke Sagawa and Hao Yang and Lewys Jones and Martin Simson and Martin Huth and Heike Soltau and Peter D. Nellist and Yukihito Kondo (2017): Development of Fast Pixelated {STEM} Detector and its Applications using 4-Dimensional Dataset. Cambridge University Press ({CUP}). 10.1017/s1431927617000940", "[doi:10.1002/9783527808465.EMC2016.6284] MacLaren, Ian and Nord, Magnus and Ross, Andrew and Krajnak, Matus and Hart, Martin and Doye, Alastair and McGrouther, Damien and Bali, Rantej and Banerjee, Archan and Hadfield, Robert (2016): Pixelated STEM detectors: opportunities and challenges. American Cancer Society. 10.1002/9783527808465.EMC2016.6284", "[Nguyen2016] Kayla X. Nguyen and Prafull Purohit and Robert Hovden and Emrah Turgut and Mark W. Tate and Lena F. Kourkoutis and Gregory D. Fuchs and Sol M. Gruner and David A. Muller (2016): 4D-{STEM} for Quantitative Imaging of Magnetic Materials with Enhanced Contrast and Resolution. Cambridge University Press ({CUP}). 10.1017/s1431927616009430", "[Cowley1979] J.M. Cowley (1979): Coherent interference in convergent-beam electron diffraction and shadow imaging. Elsevier {BV}. 10.1016/s0304-3991(79)80021-2", "[Hoppe1969] W. Hoppe (1969): Beugung im inhomogenen Prim\u00e4rstrahlwellenfeld. I. Prinzip einer Phasenmessung von Elektronenbeungungsinterferenzen. International Union of Crystallography ({IUCr}). 10.1107/s0567739469001045", "[Hegerl1970] R. Hegerl and W. Hoppe (1970): Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogenen Prim\u00e4rstrahlwellenfeld. Wiley. 10.1002/bbpc.19700741112", "[Nellist1995] P. D. Nellist and B. C. McCallum and J. M. Rodenburg (1995): Resolution beyond the {\\textquotesingle}information limit{\\textquotesingle} in transmission electron microscopy. Springer Nature. 10.1038/374630a0", "[McCallum1993] B. C. McCallum and J. M. Rodenburg (1993): Simultaneous reconstruction of object and aperture functions from multiple far-field intensity measurements. The Optical Society. 10.1364/josaa.10.000231", "[Friedman1992] S L Friedman and J M Rodenburg (1992): Optical demonstration of a new principle of far-field microscopy. {IOP} Publishing. 10.1088/0022-3727/25/2/003", "[Humphreys1988] C.J. Humphreys and D.J. Eaglesham and D.M. Maher and H.L. Fraser (1988): {CBED} and {CBIM} from semiconductors and superconductors. Elsevier {BV}. 10.1016/0304-3991(88)90371-3", "[Steeds1979] J. W. Steeds (1979): Convergent Beam Electron Diffraction. Springer {US}. 10.1007/978-1-4757-5581-7_15", "[Tate2016] Mark W. Tate and Prafull Purohit and Darol Chamberlain and Kayla X. Nguyen and Robert Hovden and Celesta S. Chang and Pratiti Deb and Emrah Turgut and John T. Heron and Darrell G. Schlom and Daniel C. Ralph and Gregory D. Fuchs and Katherine S. Shanks and Hugh T. Philipp and David A. Muller and Sol M. Gruner (2016): High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy. Cambridge University Press ({CUP}). 10.1017/s1431927615015664", "[Jiang2018] Yi Jiang and Zhen Chen and Yimo Han and Pratiti Deb and Hui Gao and Saien Xie and Prafull Purohit and Mark W. Tate and Jiwoong Park and Sol M. Gruner and Veit Elser and David A. Muller (2018): Electron ptychography of 2D materials to deep sub-{\\aa}ngstr\u00f6m resolution. Springer Nature. 10.1038/s41586-018-0298-5", "[Ophus2014] Colin Ophus and Peter Ercius and Michael Sarahan and Cory Czarnik and Jim Ciston (2014): Recording and Using 4D-{STEM} Datasets in Materials Science. Cambridge University Press ({CUP}). 10.1017/s1431927614002037", "[Simson2015] M. Simson and H. Ryll and H. Banba and R. Hartmann and M. Huth and S. Ihle and L. Jones and Y. Kondo and K. Muller and P.D. Nellist and R. Sagawa and J. Schmidt and H. Soltau and L. Striider and H. Yang (2015): 4D-{STEM} Imaging With the {pnCCD} (S){TEM}-Camera. Cambridge University Press ({CUP}). 10.1017/s1431927615011836", "[Pennycook2015] Timothy J. Pennycook and Andrew R. Lupini and Hao Yang and Matthew F. Murfitt and Lewys Jones and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part 1: Experimental demonstration at atomic resolution. Elsevier {BV}. 10.1016/j.ultramic.2014.09.013", "[Yang2015a] Hao Yang and Timothy J. Pennycook and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part {II}: Optimisation of imaging conditions. Elsevier {BV}. 10.1016/j.ultramic.2014.10.013", "[QuantumMerlin2017] {Quantum Detectors} (2017): Merlin for {EM} technical datasheet. http://quantumdetectors.com/wp-content/uploads/2017/01/1532-Merlin-for-EM-Technical-Datasheet-v2.pdf", "[Lawrence2017] Ethan L. Lawrence and Shery L. Y. Chang and Peter A. Crozier (2017): In situ {TEM} observations of Oxygen Surface Dynamics in {CeO}2 Cubes. Cambridge University Press ({CUP}). 10.1017/s1431927617010637", "[Li2013] Xueming Li and Paul Mooney and Shawn Zheng and Christopher R Booth and Michael B Braunfeld and Sander Gubbens and David A Agard and Yifan Cheng (2013): Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-{EM}. Springer Nature. 10.1038/nmeth.2472", "[XSpectrum-LambdaM2] X-Spectrum: {Lambda2M} large area {Medipix3} based detector array. http://www.x-spectrum.de/index_htm_files/X-Spectrum_datasheet_2M.pdf", "[Li:2018ngp] Li, Xin and Dyck, Ondrej and Kalinin, Sergei V. and Jesse, Stephen (2018): Compressed Sensing of Scanning Transmission Electron Microscopy {(STEM)} on Non-Rectangular Scans. 1805.04957", "[Nielsen1994] Nielsen, Jakob (1994): Usability Engineering. Elsevier Science. https://www.ebook.de/de/product/21119819/jakob_nielsen_usability_engineering.html", "[Lee2011] Craig A. Lee and Samuel D. Gasster and Antonio Plaza and Chein-I Chang and Bormin Huang (2011): Recent Developments in High Performance Computing for Remote Sensing: A Review. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/jstars.2011.2162643", "[Tezduyar1996] T. Tezduyar and S. Aliabadi and M. Behr and A. Johnson and V. Kalro and M. Litke (1996): Flow simulation and high performance computing. Springer Nature America, Inc. 10.1007/bf00350249", "[Allcock2002] Bill Allcock and Joe Bester and John Bresnahan and Ann L. Chervenak and Ian Foster and Carl Kesselman and Sam Meder and Veronika Nefedova and Darcy Quesnel and Steven Tuecke (2002): Data management and transfer in high-performance computational grid environments. Elsevier {BV}. 10.1016/s0167-8191(02)00094-7", "[Pan2016] Ming Pan and Cory Czarnik (2016): Image Detectors for Environmental Transmission Electron Microscopy ({ETEM}). Springer International Publishing. 10.1007/978-3-319-22988-1_5", "[Knuth1974] Donald E. Knuth (1974): Structured Programming with go to Statements. Association for Computing Machinery ({ACM}). 10.1145/356635.356640", "[Busch2016] Busch, Kat (2016): The Rules of Optimization: Why So Many Performance Efforts Fail. https://hackernoon.com/the-rules-of-optimization-why-so-many-performance-efforts-fail-cf06aad89099", "[Delvecchio2011] Delvecchio, Paul (2011): De-Mystifying Software Performance Optimization. https://software.intel.com/en-us/articles/de-mystifying-software-performance-optimization", "[Mavridis2017] Ilias Mavridis and Helen Karatza (2017): Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. Elsevier {BV}. 10.1016/j.jss.2016.11.037", "[Dean2008] Jeffrey Dean and Sanjay Ghemawat (2008): {MapReduce}. Association for Computing Machinery ({ACM}). 10.1145/1327452.1327492", "[Stegmaier2017] Stegmaier, Johannes (2017): New Methods to Improve Large-Scale Microscopy Image Analysis with Prior Knowledge and Uncertainty. KIT Scientific Publishing. 10.5445/ksp/1000060221", "[Goscinski2014] Wojtek J. Goscinski and Paul McIntosh and Ulrich Felzmann and Anton Maksimenko and Christopher J. Hall and Timur Gureyev and Darren Thompson and Andrew Janke and Graham Galloway and Neil E. B. Killeen and Parnesh Raniga and Owen Kaluza and Amanda Ng and Govinda Poudel and David G. Barnes and Toan Nguyen and Paul Bonnington and Gary F. Egan (2014): The multi-modal Australian {ScienceS} Imaging and Visualization Environment ({MASSIVE}) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers Media {SA}. 10.3389/fninf.2014.00030", "[Foust2015] Gabriel Foust and Jaakko J\u00e4rvi and Sean Parent (2015): Generating reactive programs for graphical user interfaces from multi-way dataflow constraint systems. {ACM} Press. 10.1145/2814204.2814207", "[Shneiderman2009] Ben Shneiderman and Catherine Plaisant and Maxine Cohen and Steven Jacobs (2009): Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th Edition). Pearson. https://www.amazon.com/Designing-User-Interface-Human-Computer-Interaction/dp/0321537351?SubscriptionId=AKIAIOBINVZYXZQZ2U3A&tag=chimbori05-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=0321537351", "[Tsukamoto2014] Takafumi Tsukamoto and Takuo Yasunaga (2014): Development of a user-friendly system for image processing of electron microscopy by integrating a web browser and {PIONE} with Eos. Oxford University Press ({OUP}). 10.1093/jmicro/dfu070", "[Rogic] Ivan Rogic: {React}, {Redux} and {Immutable.js}: Ingredients for Efficient Web Applications. https://www.toptal.com/react/react-redux-and-immutablejs", "[Lachlan] Lachlan Miller (2017): Redux-driven {UI} design. https://medium.com/@lachlanmiller_52885/redux-driven-ui-design-8586a84d808f", "[Taylor1996] R.N. Taylor and N. Medvidovic and K.M. Anderson and E.J. Whitehead and J.E. Robbins and K.A. Nies and P. Oreizy and D.L. Dubrow (1996): A component- and message-based architectural style for {GUI} software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/32.508313", "[pyFAI] {Data analysis unit, European Synchrotron Radiation Facility, Grenoble} (2018): General introduction to {PyFAI}. https://pyfai.readthedocs.io/en/latest/pyFAI.html", "[Alted2010] Francesc Alted (2010): Why Modern {CPUs} Are Starving and What Can Be Done about It. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2010.51", "[Clausen2018a] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1478763", "[IntelLustre] {Intel High Performance Data Division} (2014): Architecting a High Performance Storage System. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/architecting-lustre-storage-white-paper.pdf", "[Patel2012] Aditya B. Patel and Manashvi Birla and Ushma Nair (2012): Addressing big data problem using {Hadoop} and {Map Reduce}. {IEEE}. 10.1109/nuicone.2012.6493198", "[Larsen2000] Samuel Larsen and Saman Amarasinghe (2000): Exploiting superword level parallelism with multimedia instruction sets. Association for Computing Machinery ({ACM}). 10.1145/358438.349320", "[Walt2011] St{\\'{e}}fan van der Walt and S Chris Colbert and Ga\u00ebl Varoquaux (2011): The {NumPy} Array: A Structure for Efficient Numerical Computation. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2011.37", "[Thadani19951AE] Moti Thadani and Yousef Y. A. Khalidi (1995): An Efficient Zero-Copy {I / O} Framework for {UNIX} \u00ae.", "[Stancevic2003] Dragan Stancevic (2003): Zero Copy I: User-Mode Perspective. https://www.linuxjournal.com/article/6345?page=0,0", "[Kelly2003] Paul H J Kelly (2003): Advanced Computer Architecture: Caches and Memory Systems. https://www.doc.ic.ac.uk/~phjk/AdvancedCompArchitecture/2003-04/Lectures/Ch03/ACA-CH03-CurrentVersion.pdf", "[Shvachko2010] Konstantin Shvachko and Hairong Kuang and Sanjay Radia and Robert Chansler (2010): The Hadoop Distributed File System. {IEEE}. 10.1109/msst.2010.5496972", "[HDFS-Short] {Apache Software Foundation} (2018): {HDFS} Short-Circuit Local Reads. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html", "[Cooley1965] James W. Cooley and John W. Tukey (1965): An algorithm for the machine calculation of complex Fourier series. American Mathematical Society ({AMS}). 10.1090/s0025-5718-1965-0178586-1", "[Zaharia2016] Matei Zaharia and Michael J. Franklin and Ali Ghodsi and Joseph Gonzalez and Scott Shenker and Ion Stoica and Reynold S. Xin and Patrick Wendell and Tathagata Das and Michael Armbrust and Ankur Dave and Xiangrui Meng and Josh Rosen and Shivaram Venkataraman (2016): Apache Spark. Association for Computing Machinery ({ACM}). 10.1145/2934664", "[Rocklin2015] Matthew Rocklin (2015): Dask: Parallel Computation with Blocked algorithms and Task Scheduling. {SciPy}. 10.25080/majora-7b98e3ed-013", "[BLAS] (2017): {BLAS} ({Basic Linear Algebra Subprograms}). http://www.netlib.org/blas/", "[PyTorch] (2018): PyTorch. https://pytorch.org/", "[MKL] Intel (2018): Intel\u00ae Math Kernel Library. https://software.intel.com/en-us/mkl", "[Godbolt] Matt Godbolt: Compiler Explorer. https://godbolt.org/", "[Numba] (2018): Numba: A high-performance {Python} compiler. http://numba.pydata.org/", "[ReactJS] {Facebook Inc.} (2018): React: A {JavaScript} library for building user interfaces. https://reactjs.org/", "[ReduxJS] Dan Abramov (2018): Redux: A predictable state container for {JavaScript} apps.. https://redux.js.org/", "[MerkelDocker] Dirk Merkel (2014): Docker: Lightweight Linux Containers for Consistent Development and Deployment. https://www.linuxjournal.com/content/docker-lightweight-linux-containers-consistent-development-and-deployment", "[Bernstein2014] David Bernstein (2014): Containers and Cloud: From {LXC} to Docker to Kubernetes. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcc.2014.51", "[Shaw1990] Mary Shaw (1990): Prospects for an engineering discipline of software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/52.60586", "[Schindelin2012] Johannes Schindelin and Ignacio Arganda-Carreras and Erwin Frise and Verena Kaynig and Mark Longair and Tobias Pietzsch and Stephan Preibisch and Curtis Rueden and Stephan Saalfeld and Benjamin Schmid and Jean-Yves Tinevez and Daniel James White and Volker Hartenstein and Kevin Eliceiri and Pavel Tomancak and Albert Cardona (2012): Fiji: an open-source platform for biological-image analysis. Springer Nature. 10.1038/nmeth.2019", "[Willinsky2005] John Willinsky (2005): The unacknowledged convergence of open source, open access, and open science. University of Illinois Libraries. 10.5210/fm.v10i8.1265", "[Zuiderwijk2012] Anneke Zuiderwijk and Keith Jeffery and Marijn Janssen (2012): The Potential of Metadata for Linked Open Data and its Value for Users and Publishers. {JEDEM} Journal of e-Democracy and Open Government. 10.29379/jedem.v4i2.138", "[HDF5] (2016): High Level Introduction to {HDF5}. https://support.hdfgroup.org/HDF5/Tutor/HDF5Intro.pdf", "[Okoli2015] Chitu Okoli and Johannes Nguyen (2015): Business Models for Free and Open Source Software. Elsevier {BV}. 10.2139/ssrn.2568185", "[ApacheFoundation] {{The APACHE Software Foundation}} (2018): The {APACHE} Software Foundation. https://www.apache.org/", "[Linuxfoundation] {The Linux Foundation} (2018): The Linux Foundation. https://www.linuxfoundation.org/", "[Williams1996] David B. Williams and C. Barry Carter (1996): The Transmission Electron Microscope. Springer {US}. 10.1007/978-1-4757-2519-3_1", "[Goto2008] Kazushige Goto and Robert A. van de Geijn (2008): Anatomy of high-performance matrix multiplication. Association for Computing Machinery ({ACM}). 10.1145/1356052.1356053", "[DaskSpark] {Anaconda Inc.} (2018): Comparison to Spark. http://docs.dask.org/en/latest/spark.html", "[Lemire2012] Daniel Lemire and Leonid Boytsov: Decoding billions of integers per second through vectorization. 10.1002/spe.2203", "[grosser2011polly] Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simb{\\\"u}rger, Andreas and Gr{\\\"o}{\\ss}linger, Armin and Pouchet, Louis-No{\\\"e}l: Polly-Polyhedral optimization in {LLVM}. http://perso.ens-lyon.fr/christophe.alias/impact2011/impact-07.pdf", "[Song2018] Dongsheng Song and Zi-An Li and Jan Caron and Andr{\\'{a}}s Kov{\\'{a}}cs and Huanfang Tian and Chiming Jin and Haifeng Du and Mingliang Tian and Jianqi Li and Jing Zhu and Rafal E. Dunin- Homepage: https://libertem.github.io/LiberTEM-blobfinderGitHub repository: https://github.com/LiberTEM/LiberTEM-blobfinderPyPI: https://pypi.org/project/libertem-blobfinder/ Usage See Usage documentation! InstallationThe quick version from PyPI: $ pip install libertem-blobfinder[pyfftw] See full installation instructions for more details! Reference See blobfinder API reference for details! Examples See example applications! LicenseLiberTEM-blobfinder is licensed under GPLv3. We gratefully acknowledge funding from the Initiative and Network Fund of the Helmholtz Association within the Helmholtz Young Investigator Group moreSTEM under Contract No. VH-NG-1317 at Forschungszentrum Jülich in Germany. We kindly acknowledge funding from Google Summer of Code 2019 under the umbrella of the Python software foundation. Forschungszentrum Jülich is supporting LiberTEM with funding for personnel, access to its infrastructure and administrative support.

  • Research software . 2022
    Open Access
    Authors: 
    Clausen, Alexander; Weber, Dieter; Bryan, Matthew; Ruzaeva, Karina; Migunov, Vadim; Baburajan, Anand; Bahuleyan, Abijith; Caron, Jan; Chandra, Rahul; Dey, Shankhadeep; +12 more
    Publisher: Zenodo
    Project: EC | CritCat (686053), EC | 3D MAGiC (856538), EC | VIDEO (780487), EC | ESTEEM3 (823717)

    {"references": ["[Weber2018] Weber, Dieter (2018): Development of {IT} system and {TEM} camera performance. Zenodo. 10.5281/zenodo.2450624", "[Bon2012] Pierre Bon and Serge Monneret and Benoit Wattellier (2012): Noniterative boundary-artifact-free wavefront reconstruction from its derivatives. The Optical Society. 10.1364/ao.51.005698", "[K2IS2018] {{Gatan Inc.}} (2018): {K2} {IS} camera. https://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera", "[Clausen2018] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1477847", "[Sauter2013] Nicholas K. Sauter and Johan Hattne and Ralf W. Grosse-Kunstleve and Nathaniel Echols (2013): New Python-based methods for data processing. International Union of Crystallography ({IUCr}). 10.1107/s0907444913000863", "[Koennecke2015] Mark K\u00f6nnecke and Frederick A. Akeroyd and Herbert J. Bernstein and Aaron S. Brewster and Stuart I. Campbell and Bj\u00f6rn Clausen and Stephen Cottrell and Jens Uwe Hoffmann and Pete R. Jemian and David M\u00e4nnicke and Raymond Osborn and Peter F. Peterson and Tobias Richter and Jiro Suzuki and Benjamin Watts and Eugen Wintersberger and Joachim Wuttke (2015): The {NeXus} data format. International Union of Crystallography ({IUCr}). 10.1107/s1600576714027575", "[Fangohr2018] Fangohr, Hans and Aplin, Steve and Barty, Anton and Beg, Marijan and Bondar, Valerii and Boukhelef, Djelloul and Brockhauser, Sandor and Danilevski, Cyril and Ehsan, Wajid and Esenov, Sergey and Flucke, Gero and Giovanetti, Gabriele and Goeries, Dennis and Hauf, Steffen and Heisen, Burkhard and Hickin, David and Khakhulin, Dmitry and Klimovskaia, Anna and Kluyver, Thomas and Kuhn, Manuela and Kuster, Markus and Lang, Philipp-Michael and Maia, Luis and Mariani, Valerio and Mekinda, Leonce and Michelat, Thomas and Parenti, Andrea and Previtali, Gianpietro and Santos, Hugo and Silenzi, Alessandro and Sztuk-Dambietz, Jolanta and Szuba, Janusz and Teichmann, Martin and Weger, Kerstin and Wiggins, John and Wrona, Krzysztof and Xu, Chen (2018): Data Analysis Support in Karabo at European XFEL. JACoW, Geneva, Switzerland. 10.18429/jacow-icalepcs2017-tucpa01", "[Jesse2016] S. Jesse and M. Chi and A. Belianinov and C. Beekman and S. V. Kalinin and A. Y. Borisevich and A. R. Lupini (2016): Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography. Springer Nature. 10.1038/srep26348", "[Belianinov2015] Alex Belianinov and Rama Vasudevan and Evgheni Strelcov and Chad Steed and Sang Mo Yang and Alexander Tselev and Stephen Jesse and Michael Biegalski and Galen Shipman and Christopher Symons and Albina Borisevich and Rick Archibald and Sergei Kalinin (2015): Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets. Springer Nature. 10.1186/s40679-015-0006-6", "[Krajnak2016] Matus Krajnak and Damien McGrouther and Dzmitry Maneuski and Val O{\\textquotesingle} Shea and Stephen McVitie (2016): Pixelated detectors and improved efficiency for magnetic imaging in {STEM} differential phase contrast. Elsevier {BV}. 10.1016/j.ultramic.2016.03.006", "[Yang2015] H Yang and L Jones and H Ryll and M Simson and H Soltau and Y Kondo and R Sagawa and H Banba and I MacLaren and P D Nellist (2015): 4D {STEM}: High efficiency phase contrast imaging using a fast pixelated detector. {IOP} Publishing. 10.1088/1742-6596/644/1/012032", "[Sagawa2017] Ryusuke Sagawa and Hao Yang and Lewys Jones and Martin Simson and Martin Huth and Heike Soltau and Peter D. Nellist and Yukihito Kondo (2017): Development of Fast Pixelated {STEM} Detector and its Applications using 4-Dimensional Dataset. Cambridge University Press ({CUP}). 10.1017/s1431927617000940", "[doi:10.1002/9783527808465.EMC2016.6284] MacLaren, Ian and Nord, Magnus and Ross, Andrew and Krajnak, Matus and Hart, Martin and Doye, Alastair and McGrouther, Damien and Bali, Rantej and Banerjee, Archan and Hadfield, Robert (2016): Pixelated STEM detectors: opportunities and challenges. American Cancer Society. 10.1002/9783527808465.EMC2016.6284", "[Nguyen2016] Kayla X. Nguyen and Prafull Purohit and Robert Hovden and Emrah Turgut and Mark W. Tate and Lena F. Kourkoutis and Gregory D. Fuchs and Sol M. Gruner and David A. Muller (2016): 4D-{STEM} for Quantitative Imaging of Magnetic Materials with Enhanced Contrast and Resolution. Cambridge University Press ({CUP}). 10.1017/s1431927616009430", "[Cowley1979] J.M. Cowley (1979): Coherent interference in convergent-beam electron diffraction and shadow imaging. Elsevier {BV}. 10.1016/s0304-3991(79)80021-2", "[Hoppe1969] W. Hoppe (1969): Beugung im inhomogenen Prim\u00e4rstrahlwellenfeld. I. Prinzip einer Phasenmessung von Elektronenbeungungsinterferenzen. International Union of Crystallography ({IUCr}). 10.1107/s0567739469001045", "[Hegerl1970] R. Hegerl and W. Hoppe (1970): Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogenen Prim\u00e4rstrahlwellenfeld. Wiley. 10.1002/bbpc.19700741112", "[Nellist1995] P. D. Nellist and B. C. McCallum and J. M. Rodenburg (1995): Resolution beyond the {\\textquotesingle}information limit{\\textquotesingle} in transmission electron microscopy. Springer Nature. 10.1038/374630a0", "[McCallum1993] B. C. McCallum and J. M. Rodenburg (1993): Simultaneous reconstruction of object and aperture functions from multiple far-field intensity measurements. The Optical Society. 10.1364/josaa.10.000231", "[Friedman1992] S L Friedman and J M Rodenburg (1992): Optical demonstration of a new principle of far-field microscopy. {IOP} Publishing. 10.1088/0022-3727/25/2/003", "[Humphreys1988] C.J. Humphreys and D.J. Eaglesham and D.M. Maher and H.L. Fraser (1988): {CBED} and {CBIM} from semiconductors and superconductors. Elsevier {BV}. 10.1016/0304-3991(88)90371-3", "[Steeds1979] J. W. Steeds (1979): Convergent Beam Electron Diffraction. Springer {US}. 10.1007/978-1-4757-5581-7_15", "[Tate2016] Mark W. Tate and Prafull Purohit and Darol Chamberlain and Kayla X. Nguyen and Robert Hovden and Celesta S. Chang and Pratiti Deb and Emrah Turgut and John T. Heron and Darrell G. Schlom and Daniel C. Ralph and Gregory D. Fuchs and Katherine S. Shanks and Hugh T. Philipp and David A. Muller and Sol M. Gruner (2016): High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy. Cambridge University Press ({CUP}). 10.1017/s1431927615015664", "[Jiang2018] Yi Jiang and Zhen Chen and Yimo Han and Pratiti Deb and Hui Gao and Saien Xie and Prafull Purohit and Mark W. Tate and Jiwoong Park and Sol M. Gruner and Veit Elser and David A. Muller (2018): Electron ptychography of 2D materials to deep sub-{\\aa}ngstr\u00f6m resolution. Springer Nature. 10.1038/s41586-018-0298-5", "[Ophus2014] Colin Ophus and Peter Ercius and Michael Sarahan and Cory Czarnik and Jim Ciston (2014): Recording and Using 4D-{STEM} Datasets in Materials Science. Cambridge University Press ({CUP}). 10.1017/s1431927614002037", "[Simson2015] M. Simson and H. Ryll and H. Banba and R. Hartmann and M. Huth and S. Ihle and L. Jones and Y. Kondo and K. Muller and P.D. Nellist and R. Sagawa and J. Schmidt and H. Soltau and L. Striider and H. Yang (2015): 4D-{STEM} Imaging With the {pnCCD} (S){TEM}-Camera. Cambridge University Press ({CUP}). 10.1017/s1431927615011836", "[Pennycook2015] Timothy J. Pennycook and Andrew R. Lupini and Hao Yang and Matthew F. Murfitt and Lewys Jones and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part 1: Experimental demonstration at atomic resolution. Elsevier {BV}. 10.1016/j.ultramic.2014.09.013", "[Yang2015a] Hao Yang and Timothy J. Pennycook and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part {II}: Optimisation of imaging conditions. Elsevier {BV}. 10.1016/j.ultramic.2014.10.013", "[QuantumMerlin2017] {Quantum Detectors} (2017): Merlin for {EM} technical datasheet. http://quantumdetectors.com/wp-content/uploads/2017/01/1532-Merlin-for-EM-Technical-Datasheet-v2.pdf", "[Lawrence2017] Ethan L. Lawrence and Shery L. Y. Chang and Peter A. Crozier (2017): In situ {TEM} observations of Oxygen Surface Dynamics in {CeO}2 Cubes. Cambridge University Press ({CUP}). 10.1017/s1431927617010637", "[Li2013] Xueming Li and Paul Mooney and Shawn Zheng and Christopher R Booth and Michael B Braunfeld and Sander Gubbens and David A Agard and Yifan Cheng (2013): Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-{EM}. Springer Nature. 10.1038/nmeth.2472", "[XSpectrum-LambdaM2] X-Spectrum: {Lambda2M} large area {Medipix3} based detector array. http://www.x-spectrum.de/index_htm_files/X-Spectrum_datasheet_2M.pdf", "[Li:2018ngp] Li, Xin and Dyck, Ondrej and Kalinin, Sergei V. and Jesse, Stephen (2018): Compressed Sensing of Scanning Transmission Electron Microscopy {(STEM)} on Non-Rectangular Scans. 1805.04957", "[Nielsen1994] Nielsen, Jakob (1994): Usability Engineering. Elsevier Science. https://www.ebook.de/de/product/21119819/jakob_nielsen_usability_engineering.html", "[Lee2011] Craig A. Lee and Samuel D. Gasster and Antonio Plaza and Chein-I Chang and Bormin Huang (2011): Recent Developments in High Performance Computing for Remote Sensing: A Review. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/jstars.2011.2162643", "[Tezduyar1996] T. Tezduyar and S. Aliabadi and M. Behr and A. Johnson and V. Kalro and M. Litke (1996): Flow simulation and high performance computing. Springer Nature America, Inc. 10.1007/bf00350249", "[Allcock2002] Bill Allcock and Joe Bester and John Bresnahan and Ann L. Chervenak and Ian Foster and Carl Kesselman and Sam Meder and Veronika Nefedova and Darcy Quesnel and Steven Tuecke (2002): Data management and transfer in high-performance computational grid environments. Elsevier {BV}. 10.1016/s0167-8191(02)00094-7", "[Pan2016] Ming Pan and Cory Czarnik (2016): Image Detectors for Environmental Transmission Electron Microscopy ({ETEM}). Springer International Publishing. 10.1007/978-3-319-22988-1_5", "[Knuth1974] Donald E. Knuth (1974): Structured Programming with go to Statements. Association for Computing Machinery ({ACM}). 10.1145/356635.356640", "[Busch2016] Busch, Kat (2016): The Rules of Optimization: Why So Many Performance Efforts Fail. https://hackernoon.com/the-rules-of-optimization-why-so-many-performance-efforts-fail-cf06aad89099", "[Delvecchio2011] Delvecchio, Paul (2011): De-Mystifying Software Performance Optimization. https://software.intel.com/en-us/articles/de-mystifying-software-performance-optimization", "[Mavridis2017] Ilias Mavridis and Helen Karatza (2017): Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. Elsevier {BV}. 10.1016/j.jss.2016.11.037", "[Dean2008] Jeffrey Dean and Sanjay Ghemawat (2008): {MapReduce}. Association for Computing Machinery ({ACM}). 10.1145/1327452.1327492", "[Stegmaier2017] Stegmaier, Johannes (2017): New Methods to Improve Large-Scale Microscopy Image Analysis with Prior Knowledge and Uncertainty. KIT Scientific Publishing. 10.5445/ksp/1000060221", "[Goscinski2014] Wojtek J. Goscinski and Paul McIntosh and Ulrich Felzmann and Anton Maksimenko and Christopher J. Hall and Timur Gureyev and Darren Thompson and Andrew Janke and Graham Galloway and Neil E. B. Killeen and Parnesh Raniga and Owen Kaluza and Amanda Ng and Govinda Poudel and David G. Barnes and Toan Nguyen and Paul Bonnington and Gary F. Egan (2014): The multi-modal Australian {ScienceS} Imaging and Visualization Environment ({MASSIVE}) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers Media {SA}. 10.3389/fninf.2014.00030", "[Foust2015] Gabriel Foust and Jaakko J\u00e4rvi and Sean Parent (2015): Generating reactive programs for graphical user interfaces from multi-way dataflow constraint systems. {ACM} Press. 10.1145/2814204.2814207", "[Shneiderman2009] Ben Shneiderman and Catherine Plaisant and Maxine Cohen and Steven Jacobs (2009): Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th Edition). Pearson. https://www.amazon.com/Designing-User-Interface-Human-Computer-Interaction/dp/0321537351?SubscriptionId=AKIAIOBINVZYXZQZ2U3A&tag=chimbori05-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=0321537351", "[Tsukamoto2014] Takafumi Tsukamoto and Takuo Yasunaga (2014): Development of a user-friendly system for image processing of electron microscopy by integrating a web browser and {PIONE} with Eos. Oxford University Press ({OUP}). 10.1093/jmicro/dfu070", "[Rogic] Ivan Rogic: {React}, {Redux} and {Immutable.js}: Ingredients for Efficient Web Applications. https://www.toptal.com/react/react-redux-and-immutablejs", "[Lachlan] Lachlan Miller (2017): Redux-driven {UI} design. https://medium.com/@lachlanmiller_52885/redux-driven-ui-design-8586a84d808f", "[Taylor1996] R.N. Taylor and N. Medvidovic and K.M. Anderson and E.J. Whitehead and J.E. Robbins and K.A. Nies and P. Oreizy and D.L. Dubrow (1996): A component- and message-based architectural style for {GUI} software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/32.508313", "[pyFAI] {Data analysis unit, European Synchrotron Radiation Facility, Grenoble} (2018): General introduction to {PyFAI}. https://pyfai.readthedocs.io/en/latest/pyFAI.html", "[Alted2010] Francesc Alted (2010): Why Modern {CPUs} Are Starving and What Can Be Done about It. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2010.51", "[Clausen2018a] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1478763", "[IntelLustre] {Intel High Performance Data Division} (2014): Architecting a High Performance Storage System. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/architecting-lustre-storage-white-paper.pdf", "[Patel2012] Aditya B. Patel and Manashvi Birla and Ushma Nair (2012): Addressing big data problem using {Hadoop} and {Map Reduce}. {IEEE}. 10.1109/nuicone.2012.6493198", "[Larsen2000] Samuel Larsen and Saman Amarasinghe (2000): Exploiting superword level parallelism with multimedia instruction sets. Association for Computing Machinery ({ACM}). 10.1145/358438.349320", "[Walt2011] St{\\'{e}}fan van der Walt and S Chris Colbert and Ga\u00ebl Varoquaux (2011): The {NumPy} Array: A Structure for Efficient Numerical Computation. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2011.37", "[Thadani19951AE] Moti Thadani and Yousef Y. A. Khalidi (1995): An Efficient Zero-Copy {I / O} Framework for {UNIX} \u00ae.", "[Stancevic2003] Dragan Stancevic (2003): Zero Copy I: User-Mode Perspective. https://www.linuxjournal.com/article/6345?page=0,0", "[Kelly2003] Paul H J Kelly (2003): Advanced Computer Architecture: Caches and Memory Systems. https://www.doc.ic.ac.uk/~phjk/AdvancedCompArchitecture/2003-04/Lectures/Ch03/ACA-CH03-CurrentVersion.pdf", "[Shvachko2010] Konstantin Shvachko and Hairong Kuang and Sanjay Radia and Robert Chansler (2010): The Hadoop Distributed File System. {IEEE}. 10.1109/msst.2010.5496972", "[HDFS-Short] {Apache Software Foundation} (2018): {HDFS} Short-Circuit Local Reads. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html", "[Cooley1965] James W. Cooley and John W. Tukey (1965): An algorithm for the machine calculation of complex Fourier series. American Mathematical Society ({AMS}). 10.1090/s0025-5718-1965-0178586-1", "[Zaharia2016] Matei Zaharia and Michael J. Franklin and Ali Ghodsi and Joseph Gonzalez and Scott Shenker and Ion Stoica and Reynold S. Xin and Patrick Wendell and Tathagata Das and Michael Armbrust and Ankur Dave and Xiangrui Meng and Josh Rosen and Shivaram Venkataraman (2016): Apache Spark. Association for Computing Machinery ({ACM}). 10.1145/2934664", "[Rocklin2015] Matthew Rocklin (2015): Dask: Parallel Computation with Blocked algorithms and Task Scheduling. {SciPy}. 10.25080/majora-7b98e3ed-013", "[BLAS] (2017): {BLAS} ({Basic Linear Algebra Subprograms}). http://www.netlib.org/blas/", "[PyTorch] (2018): PyTorch. https://pytorch.org/", "[MKL] Intel (2018): Intel\u00ae Math Kernel Library. https://software.intel.com/en-us/mkl", "[Godbolt] Matt Godbolt: Compiler Explorer. https://godbolt.org/", "[Numba] (2018): Numba: A high-performance {Python} compiler. http://numba.pydata.org/", "[ReactJS] {Facebook Inc.} (2018): React: A {JavaScript} library for building user interfaces. https://reactjs.org/", "[ReduxJS] Dan Abramov (2018): Redux: A predictable state container for {JavaScript} apps.. https://redux.js.org/", "[MerkelDocker] Dirk Merkel (2014): Docker: Lightweight Linux Containers for Consistent Development and Deployment. https://www.linuxjournal.com/content/docker-lightweight-linux-containers-consistent-development-and-deployment", "[Bernstein2014] David Bernstein (2014): Containers and Cloud: From {LXC} to Docker to Kubernetes. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcc.2014.51", "[Shaw1990] Mary Shaw (1990): Prospects for an engineering discipline of software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/52.60586", "[Schindelin2012] Johannes Schindelin and Ignacio Arganda-Carreras and Erwin Frise and Verena Kaynig and Mark Longair and Tobias Pietzsch and Stephan Preibisch and Curtis Rueden and Stephan Saalfeld and Benjamin Schmid and Jean-Yves Tinevez and Daniel James White and Volker Hartenstein and Kevin Eliceiri and Pavel Tomancak and Albert Cardona (2012): Fiji: an open-source platform for biological-image analysis. Springer Nature. 10.1038/nmeth.2019", "[Willinsky2005] John Willinsky (2005): The unacknowledged convergence of open source, open access, and open science. University of Illinois Libraries. 10.5210/fm.v10i8.1265", "[Zuiderwijk2012] Anneke Zuiderwijk and Keith Jeffery and Marijn Janssen (2012): The Potential of Metadata for Linked Open Data and its Value for Users and Publishers. {JEDEM} Journal of e-Democracy and Open Government. 10.29379/jedem.v4i2.138", "[HDF5] (2016): High Level Introduction to {HDF5}. https://support.hdfgroup.org/HDF5/Tutor/HDF5Intro.pdf", "[Okoli2015] Chitu Okoli and Johannes Nguyen (2015): Business Models for Free and Open Source Software. Elsevier {BV}. 10.2139/ssrn.2568185", "[ApacheFoundation] {{The APACHE Software Foundation}} (2018): The {APACHE} Software Foundation. https://www.apache.org/", "[Linuxfoundation] {The Linux Foundation} (2018): The Linux Foundation. https://www.linuxfoundation.org/", "[Williams1996] David B. Williams and C. Barry Carter (1996): The Transmission Electron Microscope. Springer {US}. 10.1007/978-1-4757-2519-3_1", "[Goto2008] Kazushige Goto and Robert A. van de Geijn (2008): Anatomy of high-performance matrix multiplication. Association for Computing Machinery ({ACM}). 10.1145/1356052.1356053", "[DaskSpark] {Anaconda Inc.} (2018): Comparison to Spark. http://docs.dask.org/en/latest/spark.html", "[Lemire2012] Daniel Lemire and Leonid Boytsov: Decoding billions of integers per second through vectorization. 10.1002/spe.2203", "[grosser2011polly] Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simb{\\\"u}rger, Andreas and Gr{\\\"o}{\\ss}linger, Armin and Pouchet, Louis-No{\\\"e}l: Polly-Polyhedral optimization in {LLVM}. http://perso.ens-lyon.fr/christophe.alias/impact2011/impact-07.pdf", "[Song2018] Dongsheng Song and Zi-An Li and Jan Caron and Andr{\\'{a}}s Kov{\\'{a}}cs and Huanfang Tian and Chiming Jin and Haifeng Du and Mingliang Tian and Jianqi Li and Jing Zhu and Rafal E. Dunin- Homepage: https://libertem.github.io/LiberTEM/GitHub repository: https://github.com/LiberTEM/LiberTEM/PyPI: https://pypi.org/project/libertem/LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a modified MapReduce programming model. The current focus is pixelated scanning transmission electron microscopy (STEM) and scanning electron beam diffraction data. MapReduce-like processing allows to specify an algorithm through two functions: One function that is mapped on portions of the input data, and another function that merges (reduces) a partial result from this mapping step into the complete result. A wide range of TEM and 4D STEM processing tasks can be expressed in this fashion, see Applications. The UDF interface of LiberTEM offers a standardized, versatile API to decouple the mathematical core of an algorithm from details of data source, parallelism, and use of results. Mapping and merging can be performed in any order and with different subdivisions of the input data, including running parts of the calculation concurrently. That means the same implementation can be used in a wide range of modalities, including massive scaling on clusters. Since each merge step produces an intermediate result, this style of processing is suitable for displaying live results from a running calculation in a GUI application and for processing live data streams. A closed-loop feedback between processing and instrument control can be realized as well. See User-defined functions for more details on the LiberTEM UDF interface. The LiberTEM back-end offers high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves very high aggregate IO performance on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel. LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.6+ and the required packages will likely work as well. The GUI is running in a web browser. InstallationThe short version: $ virtualenv -p python3 ~/libertem-venv/ $ source ~/libertem-venv/bin/activate (libertem-venv) $ python -m pip install "libertem[torch]" # optional for GPU support # See also https://docs.cupy.dev/en/stable/install.html (libertem-venv) $ python -m pip install cupy Please see our documentation for details! Alternatively, to run the LiberTEM Docker image: $ docker run -p localhost:9000:9000 --mount type=bind,source=/path/to/your/data/,dst=/data/,ro libertem/libertem or $ singularity exec docker://libertem/libertem /venv/bin/libertem-server Deployment for offline data processing on a single-node system for a local user is thoroughly tested and can be considered stable. Deployment on a cluster is experimental and still requires some additional work, see Issue #105. Back-end support for live data processing is still experimental as well, see https://github.com/LiberTEM/LiberTEM-live. ApplicationsSince LiberTEM is programmable through user-defined functions (UDFs), it can be used for a wide range of processing tasks on array-like data and data streams. The following applications have been implemented already: Virtual detectors (virtual bright field, virtual HAADF, center of mass , custom shapes via masks) Analysis of amorphous materials Strain mapping Off-axis electron holography reconstruction Single Side Band ptychography Some of these applications are available through an interactive web GUI. Please see the applications section of our documentation for details! The Python API and user-defined functions (UDFs) can be used for complex operations such as arbitrary linear operations and other features like data export. Example Jupyter notebooks are available in the examples directory. If you are having trouble running the examples, please let us know by filing an issue or by joining our Gitter chat. LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. Contact us if you are interested! LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. Currently we are working on live data processing, integration with Dask arrays and HyperSpy, support for sparse data, and implementing analysis methods for various applications of pixelated STEM and other large-scale detector data. If you like to influence the direction this project is taking, or if you'd like to contribute, please join our gitter chat and our general mailing list. File formatsLiberTEM currently opens most file formats used for pixelated STEM. See our general information on loading data and format-specific documentation for more information! Raw binary files NumPy .npy binary files Thermo Fisher EMPAD detector files Quantum Detectors MIB format Nanomegas .blo block files Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for DE-Series detectors Gatan K2 IS raw format Stacks of Gatan DM3 and DM4 files (via openNCEM) FRMS6 from PNDetector pnCCD cameras (currently alpha, gain correction still needs UI changes) FEI SER files (via openNCEM) MRC (via openNCEM) HDF5-based formats such as HyperSpy files, NeXus and EMD TVIPS binary files Please contact us if you are interested in support for an additional format! Detectors (experimental)Currently the Quantum Detectors Merlin camera is supported for live processing. Support for DECTRIS cameras and the Gatan K2 IS camera are in a prototype state. Please contact us if you are interested in this feature! See https://github.com/LiberTEM/LiberTEM-live for more details on live processing. LicenseLiberTEM is licensed under GPLv3. The I/O parts are also available under the MIT license, please see LICENSE files in the subdirectories for details. We gratefully acknowledge funding from the Initiative and Network Fund of the Helmholtz Association within the Helmholtz Young Investigator Group moreSTEM under Contract No. VH-NG-1317 at Forschungszentrum Jülich in Germany. We gratefully acknowledge funding from the Information & Data Science Pilot Project 'Ptychography 4.0' of the Helmholtz Association. We kindly acknowledge funding from Google Summer of Code 2019 under the umbrella of the Python software foundation. STEMx equipment and software for 4D STEM data acquisition with K2 IS camera courtesy of Gatan Inc. LiberTEM development is supported by AIDAS Forschungszentrum Jülich is supporting LiberTEM with funding for personnel, access to its infrastructure and administrative support.

Include:
3 Research products, page 1 of 1
  • Open Access
    Authors: 
    Anneke Zuiderwijk; Keith G. Jeffery; Marijn Janssen;
    Publisher: JEDEM Journal of e-Democracy and Open Government
    Country: Netherlands

    Public and private organizations increasingly release their data to gain benefits such as transparency and economic growth. The use of these open data can be supported and stimulated by providing considerable metadata (data about the data), including discovery, contextual and detailed metadata. In this paper we argue that metadata are key enablers for the effective use of Linked Open Data (LOD). We illustrate the potential of metadata by 1) presenting an overview of advantages and disadvantages of metadata derived from literature, 2) presenting metadata requirements for LOD architectures derived from literature, workshops and a questionnaire, 3) describing a LOD metadata architecture that meets the requirements and 4) showing examples of the application of this architecture in the ENGAGE project. The paper shows that using metadata with the appropriate metadata architecture can yield considerable benefits for LOD publication and use, including improving find ability, accessibility, storing, preservation, analysing, comparing, reproducing, finding inconsistencies, correct interpretation, visualizing, linking data, assessing and ranking the quality of data and avoiding unnecessary duplication of data. The Common European Research Information Format (CERIF) can be used to build the metadata architecture and achieve the advantages.

  • Research software . 2020
    Open Access
    Authors: 
    Clausen, Alexander; Weber, Dieter; Ruzaeva, Karina; Müller-Caspary, Knut; Dunin-Borkowski, Rafal E.;
    Publisher: Zenodo
    Project: EC | VIDEO (780487), EC | CritCat (686053), EC | ESTEEM3 (823717)

    {"references": ["[Weber2018] Weber, Dieter (2018): Development of {IT} system and {TEM} camera performance. Zenodo. 10.5281/zenodo.2450624", "[Bon2012] Pierre Bon and Serge Monneret and Benoit Wattellier (2012): Noniterative boundary-artifact-free wavefront reconstruction from its derivatives. The Optical Society. 10.1364/ao.51.005698", "[K2IS2018] {{Gatan Inc.}} (2018): {K2} {IS} camera. https://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera", "[Clausen2018] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1477847", "[Sauter2013] Nicholas K. Sauter and Johan Hattne and Ralf W. Grosse-Kunstleve and Nathaniel Echols (2013): New Python-based methods for data processing. International Union of Crystallography ({IUCr}). 10.1107/s0907444913000863", "[Koennecke2015] Mark K\u00f6nnecke and Frederick A. Akeroyd and Herbert J. Bernstein and Aaron S. Brewster and Stuart I. Campbell and Bj\u00f6rn Clausen and Stephen Cottrell and Jens Uwe Hoffmann and Pete R. Jemian and David M\u00e4nnicke and Raymond Osborn and Peter F. Peterson and Tobias Richter and Jiro Suzuki and Benjamin Watts and Eugen Wintersberger and Joachim Wuttke (2015): The {NeXus} data format. International Union of Crystallography ({IUCr}). 10.1107/s1600576714027575", "[Fangohr2018] Fangohr, Hans and Aplin, Steve and Barty, Anton and Beg, Marijan and Bondar, Valerii and Boukhelef, Djelloul and Brockhauser, Sandor and Danilevski, Cyril and Ehsan, Wajid and Esenov, Sergey and Flucke, Gero and Giovanetti, Gabriele and Goeries, Dennis and Hauf, Steffen and Heisen, Burkhard and Hickin, David and Khakhulin, Dmitry and Klimovskaia, Anna and Kluyver, Thomas and Kuhn, Manuela and Kuster, Markus and Lang, Philipp-Michael and Maia, Luis and Mariani, Valerio and Mekinda, Leonce and Michelat, Thomas and Parenti, Andrea and Previtali, Gianpietro and Santos, Hugo and Silenzi, Alessandro and Sztuk-Dambietz, Jolanta and Szuba, Janusz and Teichmann, Martin and Weger, Kerstin and Wiggins, John and Wrona, Krzysztof and Xu, Chen (2018): Data Analysis Support in Karabo at European XFEL. JACoW, Geneva, Switzerland. 10.18429/jacow-icalepcs2017-tucpa01", "[Jesse2016] S. Jesse and M. Chi and A. Belianinov and C. Beekman and S. V. Kalinin and A. Y. Borisevich and A. R. Lupini (2016): Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography. Springer Nature. 10.1038/srep26348", "[Belianinov2015] Alex Belianinov and Rama Vasudevan and Evgheni Strelcov and Chad Steed and Sang Mo Yang and Alexander Tselev and Stephen Jesse and Michael Biegalski and Galen Shipman and Christopher Symons and Albina Borisevich and Rick Archibald and Sergei Kalinin (2015): Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets. Springer Nature. 10.1186/s40679-015-0006-6", "[Krajnak2016] Matus Krajnak and Damien McGrouther and Dzmitry Maneuski and Val O{\\textquotesingle} Shea and Stephen McVitie (2016): Pixelated detectors and improved efficiency for magnetic imaging in {STEM} differential phase contrast. Elsevier {BV}. 10.1016/j.ultramic.2016.03.006", "[Yang2015] H Yang and L Jones and H Ryll and M Simson and H Soltau and Y Kondo and R Sagawa and H Banba and I MacLaren and P D Nellist (2015): 4D {STEM}: High efficiency phase contrast imaging using a fast pixelated detector. {IOP} Publishing. 10.1088/1742-6596/644/1/012032", "[Sagawa2017] Ryusuke Sagawa and Hao Yang and Lewys Jones and Martin Simson and Martin Huth and Heike Soltau and Peter D. Nellist and Yukihito Kondo (2017): Development of Fast Pixelated {STEM} Detector and its Applications using 4-Dimensional Dataset. Cambridge University Press ({CUP}). 10.1017/s1431927617000940", "[doi:10.1002/9783527808465.EMC2016.6284] MacLaren, Ian and Nord, Magnus and Ross, Andrew and Krajnak, Matus and Hart, Martin and Doye, Alastair and McGrouther, Damien and Bali, Rantej and Banerjee, Archan and Hadfield, Robert (2016): Pixelated STEM detectors: opportunities and challenges. American Cancer Society. 10.1002/9783527808465.EMC2016.6284", "[Nguyen2016] Kayla X. Nguyen and Prafull Purohit and Robert Hovden and Emrah Turgut and Mark W. Tate and Lena F. Kourkoutis and Gregory D. Fuchs and Sol M. Gruner and David A. Muller (2016): 4D-{STEM} for Quantitative Imaging of Magnetic Materials with Enhanced Contrast and Resolution. Cambridge University Press ({CUP}). 10.1017/s1431927616009430", "[Cowley1979] J.M. Cowley (1979): Coherent interference in convergent-beam electron diffraction and shadow imaging. Elsevier {BV}. 10.1016/s0304-3991(79)80021-2", "[Hoppe1969] W. Hoppe (1969): Beugung im inhomogenen Prim\u00e4rstrahlwellenfeld. I. Prinzip einer Phasenmessung von Elektronenbeungungsinterferenzen. International Union of Crystallography ({IUCr}). 10.1107/s0567739469001045", "[Hegerl1970] R. Hegerl and W. Hoppe (1970): Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogenen Prim\u00e4rstrahlwellenfeld. Wiley. 10.1002/bbpc.19700741112", "[Nellist1995] P. D. Nellist and B. C. McCallum and J. M. Rodenburg (1995): Resolution beyond the {\\textquotesingle}information limit{\\textquotesingle} in transmission electron microscopy. Springer Nature. 10.1038/374630a0", "[McCallum1993] B. C. McCallum and J. M. Rodenburg (1993): Simultaneous reconstruction of object and aperture functions from multiple far-field intensity measurements. The Optical Society. 10.1364/josaa.10.000231", "[Friedman1992] S L Friedman and J M Rodenburg (1992): Optical demonstration of a new principle of far-field microscopy. {IOP} Publishing. 10.1088/0022-3727/25/2/003", "[Humphreys1988] C.J. Humphreys and D.J. Eaglesham and D.M. Maher and H.L. Fraser (1988): {CBED} and {CBIM} from semiconductors and superconductors. Elsevier {BV}. 10.1016/0304-3991(88)90371-3", "[Steeds1979] J. W. Steeds (1979): Convergent Beam Electron Diffraction. Springer {US}. 10.1007/978-1-4757-5581-7_15", "[Tate2016] Mark W. Tate and Prafull Purohit and Darol Chamberlain and Kayla X. Nguyen and Robert Hovden and Celesta S. Chang and Pratiti Deb and Emrah Turgut and John T. Heron and Darrell G. Schlom and Daniel C. Ralph and Gregory D. Fuchs and Katherine S. Shanks and Hugh T. Philipp and David A. Muller and Sol M. Gruner (2016): High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy. Cambridge University Press ({CUP}). 10.1017/s1431927615015664", "[Jiang2018] Yi Jiang and Zhen Chen and Yimo Han and Pratiti Deb and Hui Gao and Saien Xie and Prafull Purohit and Mark W. Tate and Jiwoong Park and Sol M. Gruner and Veit Elser and David A. Muller (2018): Electron ptychography of 2D materials to deep sub-{\\aa}ngstr\u00f6m resolution. Springer Nature. 10.1038/s41586-018-0298-5", "[Ophus2014] Colin Ophus and Peter Ercius and Michael Sarahan and Cory Czarnik and Jim Ciston (2014): Recording and Using 4D-{STEM} Datasets in Materials Science. Cambridge University Press ({CUP}). 10.1017/s1431927614002037", "[Simson2015] M. Simson and H. Ryll and H. Banba and R. Hartmann and M. Huth and S. Ihle and L. Jones and Y. Kondo and K. Muller and P.D. Nellist and R. Sagawa and J. Schmidt and H. Soltau and L. Striider and H. Yang (2015): 4D-{STEM} Imaging With the {pnCCD} (S){TEM}-Camera. Cambridge University Press ({CUP}). 10.1017/s1431927615011836", "[Pennycook2015] Timothy J. Pennycook and Andrew R. Lupini and Hao Yang and Matthew F. Murfitt and Lewys Jones and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part 1: Experimental demonstration at atomic resolution. Elsevier {BV}. 10.1016/j.ultramic.2014.09.013", "[Yang2015a] Hao Yang and Timothy J. Pennycook and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part {II}: Optimisation of imaging conditions. Elsevier {BV}. 10.1016/j.ultramic.2014.10.013", "[QuantumMerlin2017] {Quantum Detectors} (2017): Merlin for {EM} technical datasheet. http://quantumdetectors.com/wp-content/uploads/2017/01/1532-Merlin-for-EM-Technical-Datasheet-v2.pdf", "[Lawrence2017] Ethan L. Lawrence and Shery L. Y. Chang and Peter A. Crozier (2017): In situ {TEM} observations of Oxygen Surface Dynamics in {CeO}2 Cubes. Cambridge University Press ({CUP}). 10.1017/s1431927617010637", "[Li2013] Xueming Li and Paul Mooney and Shawn Zheng and Christopher R Booth and Michael B Braunfeld and Sander Gubbens and David A Agard and Yifan Cheng (2013): Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-{EM}. Springer Nature. 10.1038/nmeth.2472", "[XSpectrum-LambdaM2] X-Spectrum: {Lambda2M} large area {Medipix3} based detector array. http://www.x-spectrum.de/index_htm_files/X-Spectrum_datasheet_2M.pdf", "[Li:2018ngp] Li, Xin and Dyck, Ondrej and Kalinin, Sergei V. and Jesse, Stephen (2018): Compressed Sensing of Scanning Transmission Electron Microscopy {(STEM)} on Non-Rectangular Scans. 1805.04957", "[Nielsen1994] Nielsen, Jakob (1994): Usability Engineering. Elsevier Science. https://www.ebook.de/de/product/21119819/jakob_nielsen_usability_engineering.html", "[Lee2011] Craig A. Lee and Samuel D. Gasster and Antonio Plaza and Chein-I Chang and Bormin Huang (2011): Recent Developments in High Performance Computing for Remote Sensing: A Review. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/jstars.2011.2162643", "[Tezduyar1996] T. Tezduyar and S. Aliabadi and M. Behr and A. Johnson and V. Kalro and M. Litke (1996): Flow simulation and high performance computing. Springer Nature America, Inc. 10.1007/bf00350249", "[Allcock2002] Bill Allcock and Joe Bester and John Bresnahan and Ann L. Chervenak and Ian Foster and Carl Kesselman and Sam Meder and Veronika Nefedova and Darcy Quesnel and Steven Tuecke (2002): Data management and transfer in high-performance computational grid environments. Elsevier {BV}. 10.1016/s0167-8191(02)00094-7", "[Pan2016] Ming Pan and Cory Czarnik (2016): Image Detectors for Environmental Transmission Electron Microscopy ({ETEM}). Springer International Publishing. 10.1007/978-3-319-22988-1_5", "[Knuth1974] Donald E. Knuth (1974): Structured Programming with go to Statements. Association for Computing Machinery ({ACM}). 10.1145/356635.356640", "[Busch2016] Busch, Kat (2016): The Rules of Optimization: Why So Many Performance Efforts Fail. https://hackernoon.com/the-rules-of-optimization-why-so-many-performance-efforts-fail-cf06aad89099", "[Delvecchio2011] Delvecchio, Paul (2011): De-Mystifying Software Performance Optimization. https://software.intel.com/en-us/articles/de-mystifying-software-performance-optimization", "[Mavridis2017] Ilias Mavridis and Helen Karatza (2017): Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. Elsevier {BV}. 10.1016/j.jss.2016.11.037", "[Dean2008] Jeffrey Dean and Sanjay Ghemawat (2008): {MapReduce}. Association for Computing Machinery ({ACM}). 10.1145/1327452.1327492", "[Stegmaier2017] Stegmaier, Johannes (2017): New Methods to Improve Large-Scale Microscopy Image Analysis with Prior Knowledge and Uncertainty. KIT Scientific Publishing. 10.5445/ksp/1000060221", "[Goscinski2014] Wojtek J. Goscinski and Paul McIntosh and Ulrich Felzmann and Anton Maksimenko and Christopher J. Hall and Timur Gureyev and Darren Thompson and Andrew Janke and Graham Galloway and Neil E. B. Killeen and Parnesh Raniga and Owen Kaluza and Amanda Ng and Govinda Poudel and David G. Barnes and Toan Nguyen and Paul Bonnington and Gary F. Egan (2014): The multi-modal Australian {ScienceS} Imaging and Visualization Environment ({MASSIVE}) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers Media {SA}. 10.3389/fninf.2014.00030", "[Foust2015] Gabriel Foust and Jaakko J\u00e4rvi and Sean Parent (2015): Generating reactive programs for graphical user interfaces from multi-way dataflow constraint systems. {ACM} Press. 10.1145/2814204.2814207", "[Shneiderman2009] Ben Shneiderman and Catherine Plaisant and Maxine Cohen and Steven Jacobs (2009): Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th Edition). Pearson. https://www.amazon.com/Designing-User-Interface-Human-Computer-Interaction/dp/0321537351?SubscriptionId=AKIAIOBINVZYXZQZ2U3A&tag=chimbori05-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=0321537351", "[Tsukamoto2014] Takafumi Tsukamoto and Takuo Yasunaga (2014): Development of a user-friendly system for image processing of electron microscopy by integrating a web browser and {PIONE} with Eos. Oxford University Press ({OUP}). 10.1093/jmicro/dfu070", "[Rogic] Ivan Rogic: {React}, {Redux} and {Immutable.js}: Ingredients for Efficient Web Applications. https://www.toptal.com/react/react-redux-and-immutablejs", "[Lachlan] Lachlan Miller (2017): Redux-driven {UI} design. https://medium.com/@lachlanmiller_52885/redux-driven-ui-design-8586a84d808f", "[Taylor1996] R.N. Taylor and N. Medvidovic and K.M. Anderson and E.J. Whitehead and J.E. Robbins and K.A. Nies and P. Oreizy and D.L. Dubrow (1996): A component- and message-based architectural style for {GUI} software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/32.508313", "[pyFAI] {Data analysis unit, European Synchrotron Radiation Facility, Grenoble} (2018): General introduction to {PyFAI}. https://pyfai.readthedocs.io/en/latest/pyFAI.html", "[Alted2010] Francesc Alted (2010): Why Modern {CPUs} Are Starving and What Can Be Done about It. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2010.51", "[Clausen2018a] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1478763", "[IntelLustre] {Intel High Performance Data Division} (2014): Architecting a High Performance Storage System. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/architecting-lustre-storage-white-paper.pdf", "[Patel2012] Aditya B. Patel and Manashvi Birla and Ushma Nair (2012): Addressing big data problem using {Hadoop} and {Map Reduce}. {IEEE}. 10.1109/nuicone.2012.6493198", "[Larsen2000] Samuel Larsen and Saman Amarasinghe (2000): Exploiting superword level parallelism with multimedia instruction sets. Association for Computing Machinery ({ACM}). 10.1145/358438.349320", "[Walt2011] St{\\'{e}}fan van der Walt and S Chris Colbert and Ga\u00ebl Varoquaux (2011): The {NumPy} Array: A Structure for Efficient Numerical Computation. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2011.37", "[Thadani19951AE] Moti Thadani and Yousef Y. A. Khalidi (1995): An Efficient Zero-Copy {I / O} Framework for {UNIX} \u00ae.", "[Stancevic2003] Dragan Stancevic (2003): Zero Copy I: User-Mode Perspective. https://www.linuxjournal.com/article/6345?page=0,0", "[Kelly2003] Paul H J Kelly (2003): Advanced Computer Architecture: Caches and Memory Systems. https://www.doc.ic.ac.uk/~phjk/AdvancedCompArchitecture/2003-04/Lectures/Ch03/ACA-CH03-CurrentVersion.pdf", "[Shvachko2010] Konstantin Shvachko and Hairong Kuang and Sanjay Radia and Robert Chansler (2010): The Hadoop Distributed File System. {IEEE}. 10.1109/msst.2010.5496972", "[HDFS-Short] {Apache Software Foundation} (2018): {HDFS} Short-Circuit Local Reads. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html", "[Cooley1965] James W. Cooley and John W. Tukey (1965): An algorithm for the machine calculation of complex Fourier series. American Mathematical Society ({AMS}). 10.1090/s0025-5718-1965-0178586-1", "[Zaharia2016] Matei Zaharia and Michael J. Franklin and Ali Ghodsi and Joseph Gonzalez and Scott Shenker and Ion Stoica and Reynold S. Xin and Patrick Wendell and Tathagata Das and Michael Armbrust and Ankur Dave and Xiangrui Meng and Josh Rosen and Shivaram Venkataraman (2016): Apache Spark. Association for Computing Machinery ({ACM}). 10.1145/2934664", "[Rocklin2015] Matthew Rocklin (2015): Dask: Parallel Computation with Blocked algorithms and Task Scheduling. {SciPy}. 10.25080/majora-7b98e3ed-013", "[BLAS] (2017): {BLAS} ({Basic Linear Algebra Subprograms}). http://www.netlib.org/blas/", "[PyTorch] (2018): PyTorch. https://pytorch.org/", "[MKL] Intel (2018): Intel\u00ae Math Kernel Library. https://software.intel.com/en-us/mkl", "[Godbolt] Matt Godbolt: Compiler Explorer. https://godbolt.org/", "[Numba] (2018): Numba: A high-performance {Python} compiler. http://numba.pydata.org/", "[ReactJS] {Facebook Inc.} (2018): React: A {JavaScript} library for building user interfaces. https://reactjs.org/", "[ReduxJS] Dan Abramov (2018): Redux: A predictable state container for {JavaScript} apps.. https://redux.js.org/", "[MerkelDocker] Dirk Merkel (2014): Docker: Lightweight Linux Containers for Consistent Development and Deployment. https://www.linuxjournal.com/content/docker-lightweight-linux-containers-consistent-development-and-deployment", "[Bernstein2014] David Bernstein (2014): Containers and Cloud: From {LXC} to Docker to Kubernetes. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcc.2014.51", "[Shaw1990] Mary Shaw (1990): Prospects for an engineering discipline of software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/52.60586", "[Schindelin2012] Johannes Schindelin and Ignacio Arganda-Carreras and Erwin Frise and Verena Kaynig and Mark Longair and Tobias Pietzsch and Stephan Preibisch and Curtis Rueden and Stephan Saalfeld and Benjamin Schmid and Jean-Yves Tinevez and Daniel James White and Volker Hartenstein and Kevin Eliceiri and Pavel Tomancak and Albert Cardona (2012): Fiji: an open-source platform for biological-image analysis. Springer Nature. 10.1038/nmeth.2019", "[Willinsky2005] John Willinsky (2005): The unacknowledged convergence of open source, open access, and open science. University of Illinois Libraries. 10.5210/fm.v10i8.1265", "[Zuiderwijk2012] Anneke Zuiderwijk and Keith Jeffery and Marijn Janssen (2012): The Potential of Metadata for Linked Open Data and its Value for Users and Publishers. {JEDEM} Journal of e-Democracy and Open Government. 10.29379/jedem.v4i2.138", "[HDF5] (2016): High Level Introduction to {HDF5}. https://support.hdfgroup.org/HDF5/Tutor/HDF5Intro.pdf", "[Okoli2015] Chitu Okoli and Johannes Nguyen (2015): Business Models for Free and Open Source Software. Elsevier {BV}. 10.2139/ssrn.2568185", "[ApacheFoundation] {{The APACHE Software Foundation}} (2018): The {APACHE} Software Foundation. https://www.apache.org/", "[Linuxfoundation] {The Linux Foundation} (2018): The Linux Foundation. https://www.linuxfoundation.org/", "[Williams1996] David B. Williams and C. Barry Carter (1996): The Transmission Electron Microscope. Springer {US}. 10.1007/978-1-4757-2519-3_1", "[Goto2008] Kazushige Goto and Robert A. van de Geijn (2008): Anatomy of high-performance matrix multiplication. Association for Computing Machinery ({ACM}). 10.1145/1356052.1356053", "[DaskSpark] {Anaconda Inc.} (2018): Comparison to Spark. http://docs.dask.org/en/latest/spark.html", "[Lemire2012] Daniel Lemire and Leonid Boytsov: Decoding billions of integers per second through vectorization. 10.1002/spe.2203", "[grosser2011polly] Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simb{\\\"u}rger, Andreas and Gr{\\\"o}{\\ss}linger, Armin and Pouchet, Louis-No{\\\"e}l: Polly-Polyhedral optimization in {LLVM}. http://perso.ens-lyon.fr/christophe.alias/impact2011/impact-07.pdf", "[Song2018] Dongsheng Song and Zi-An Li and Jan Caron and Andr{\\'{a}}s Kov{\\'{a}}cs and Huanfang Tian and Chiming Jin and Haifeng Du and Mingliang Tian and Jianqi Li and Jing Zhu and Rafal E. Dunin- Homepage: https://libertem.github.io/LiberTEM-blobfinderGitHub repository: https://github.com/LiberTEM/LiberTEM-blobfinderPyPI: https://pypi.org/project/libertem-blobfinder/ Usage See Usage documentation! InstallationThe quick version from PyPI: $ pip install libertem-blobfinder[pyfftw] See full installation instructions for more details! Reference See blobfinder API reference for details! Examples See example applications! LicenseLiberTEM-blobfinder is licensed under GPLv3. We gratefully acknowledge funding from the Initiative and Network Fund of the Helmholtz Association within the Helmholtz Young Investigator Group moreSTEM under Contract No. VH-NG-1317 at Forschungszentrum Jülich in Germany. We kindly acknowledge funding from Google Summer of Code 2019 under the umbrella of the Python software foundation. Forschungszentrum Jülich is supporting LiberTEM with funding for personnel, access to its infrastructure and administrative support.

  • Research software . 2022
    Open Access
    Authors: 
    Clausen, Alexander; Weber, Dieter; Bryan, Matthew; Ruzaeva, Karina; Migunov, Vadim; Baburajan, Anand; Bahuleyan, Abijith; Caron, Jan; Chandra, Rahul; Dey, Shankhadeep; +12 more
    Publisher: Zenodo
    Project: EC | CritCat (686053), EC | 3D MAGiC (856538), EC | VIDEO (780487), EC | ESTEEM3 (823717)

    {"references": ["[Weber2018] Weber, Dieter (2018): Development of {IT} system and {TEM} camera performance. Zenodo. 10.5281/zenodo.2450624", "[Bon2012] Pierre Bon and Serge Monneret and Benoit Wattellier (2012): Noniterative boundary-artifact-free wavefront reconstruction from its derivatives. The Optical Society. 10.1364/ao.51.005698", "[K2IS2018] {{Gatan Inc.}} (2018): {K2} {IS} camera. https://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera", "[Clausen2018] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1477847", "[Sauter2013] Nicholas K. Sauter and Johan Hattne and Ralf W. Grosse-Kunstleve and Nathaniel Echols (2013): New Python-based methods for data processing. International Union of Crystallography ({IUCr}). 10.1107/s0907444913000863", "[Koennecke2015] Mark K\u00f6nnecke and Frederick A. Akeroyd and Herbert J. Bernstein and Aaron S. Brewster and Stuart I. Campbell and Bj\u00f6rn Clausen and Stephen Cottrell and Jens Uwe Hoffmann and Pete R. Jemian and David M\u00e4nnicke and Raymond Osborn and Peter F. Peterson and Tobias Richter and Jiro Suzuki and Benjamin Watts and Eugen Wintersberger and Joachim Wuttke (2015): The {NeXus} data format. International Union of Crystallography ({IUCr}). 10.1107/s1600576714027575", "[Fangohr2018] Fangohr, Hans and Aplin, Steve and Barty, Anton and Beg, Marijan and Bondar, Valerii and Boukhelef, Djelloul and Brockhauser, Sandor and Danilevski, Cyril and Ehsan, Wajid and Esenov, Sergey and Flucke, Gero and Giovanetti, Gabriele and Goeries, Dennis and Hauf, Steffen and Heisen, Burkhard and Hickin, David and Khakhulin, Dmitry and Klimovskaia, Anna and Kluyver, Thomas and Kuhn, Manuela and Kuster, Markus and Lang, Philipp-Michael and Maia, Luis and Mariani, Valerio and Mekinda, Leonce and Michelat, Thomas and Parenti, Andrea and Previtali, Gianpietro and Santos, Hugo and Silenzi, Alessandro and Sztuk-Dambietz, Jolanta and Szuba, Janusz and Teichmann, Martin and Weger, Kerstin and Wiggins, John and Wrona, Krzysztof and Xu, Chen (2018): Data Analysis Support in Karabo at European XFEL. JACoW, Geneva, Switzerland. 10.18429/jacow-icalepcs2017-tucpa01", "[Jesse2016] S. Jesse and M. Chi and A. Belianinov and C. Beekman and S. V. Kalinin and A. Y. Borisevich and A. R. Lupini (2016): Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography. Springer Nature. 10.1038/srep26348", "[Belianinov2015] Alex Belianinov and Rama Vasudevan and Evgheni Strelcov and Chad Steed and Sang Mo Yang and Alexander Tselev and Stephen Jesse and Michael Biegalski and Galen Shipman and Christopher Symons and Albina Borisevich and Rick Archibald and Sergei Kalinin (2015): Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets. Springer Nature. 10.1186/s40679-015-0006-6", "[Krajnak2016] Matus Krajnak and Damien McGrouther and Dzmitry Maneuski and Val O{\\textquotesingle} Shea and Stephen McVitie (2016): Pixelated detectors and improved efficiency for magnetic imaging in {STEM} differential phase contrast. Elsevier {BV}. 10.1016/j.ultramic.2016.03.006", "[Yang2015] H Yang and L Jones and H Ryll and M Simson and H Soltau and Y Kondo and R Sagawa and H Banba and I MacLaren and P D Nellist (2015): 4D {STEM}: High efficiency phase contrast imaging using a fast pixelated detector. {IOP} Publishing. 10.1088/1742-6596/644/1/012032", "[Sagawa2017] Ryusuke Sagawa and Hao Yang and Lewys Jones and Martin Simson and Martin Huth and Heike Soltau and Peter D. Nellist and Yukihito Kondo (2017): Development of Fast Pixelated {STEM} Detector and its Applications using 4-Dimensional Dataset. Cambridge University Press ({CUP}). 10.1017/s1431927617000940", "[doi:10.1002/9783527808465.EMC2016.6284] MacLaren, Ian and Nord, Magnus and Ross, Andrew and Krajnak, Matus and Hart, Martin and Doye, Alastair and McGrouther, Damien and Bali, Rantej and Banerjee, Archan and Hadfield, Robert (2016): Pixelated STEM detectors: opportunities and challenges. American Cancer Society. 10.1002/9783527808465.EMC2016.6284", "[Nguyen2016] Kayla X. Nguyen and Prafull Purohit and Robert Hovden and Emrah Turgut and Mark W. Tate and Lena F. Kourkoutis and Gregory D. Fuchs and Sol M. Gruner and David A. Muller (2016): 4D-{STEM} for Quantitative Imaging of Magnetic Materials with Enhanced Contrast and Resolution. Cambridge University Press ({CUP}). 10.1017/s1431927616009430", "[Cowley1979] J.M. Cowley (1979): Coherent interference in convergent-beam electron diffraction and shadow imaging. Elsevier {BV}. 10.1016/s0304-3991(79)80021-2", "[Hoppe1969] W. Hoppe (1969): Beugung im inhomogenen Prim\u00e4rstrahlwellenfeld. I. Prinzip einer Phasenmessung von Elektronenbeungungsinterferenzen. International Union of Crystallography ({IUCr}). 10.1107/s0567739469001045", "[Hegerl1970] R. Hegerl and W. Hoppe (1970): Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogenen Prim\u00e4rstrahlwellenfeld. Wiley. 10.1002/bbpc.19700741112", "[Nellist1995] P. D. Nellist and B. C. McCallum and J. M. Rodenburg (1995): Resolution beyond the {\\textquotesingle}information limit{\\textquotesingle} in transmission electron microscopy. Springer Nature. 10.1038/374630a0", "[McCallum1993] B. C. McCallum and J. M. Rodenburg (1993): Simultaneous reconstruction of object and aperture functions from multiple far-field intensity measurements. The Optical Society. 10.1364/josaa.10.000231", "[Friedman1992] S L Friedman and J M Rodenburg (1992): Optical demonstration of a new principle of far-field microscopy. {IOP} Publishing. 10.1088/0022-3727/25/2/003", "[Humphreys1988] C.J. Humphreys and D.J. Eaglesham and D.M. Maher and H.L. Fraser (1988): {CBED} and {CBIM} from semiconductors and superconductors. Elsevier {BV}. 10.1016/0304-3991(88)90371-3", "[Steeds1979] J. W. Steeds (1979): Convergent Beam Electron Diffraction. Springer {US}. 10.1007/978-1-4757-5581-7_15", "[Tate2016] Mark W. Tate and Prafull Purohit and Darol Chamberlain and Kayla X. Nguyen and Robert Hovden and Celesta S. Chang and Pratiti Deb and Emrah Turgut and John T. Heron and Darrell G. Schlom and Daniel C. Ralph and Gregory D. Fuchs and Katherine S. Shanks and Hugh T. Philipp and David A. Muller and Sol M. Gruner (2016): High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy. Cambridge University Press ({CUP}). 10.1017/s1431927615015664", "[Jiang2018] Yi Jiang and Zhen Chen and Yimo Han and Pratiti Deb and Hui Gao and Saien Xie and Prafull Purohit and Mark W. Tate and Jiwoong Park and Sol M. Gruner and Veit Elser and David A. Muller (2018): Electron ptychography of 2D materials to deep sub-{\\aa}ngstr\u00f6m resolution. Springer Nature. 10.1038/s41586-018-0298-5", "[Ophus2014] Colin Ophus and Peter Ercius and Michael Sarahan and Cory Czarnik and Jim Ciston (2014): Recording and Using 4D-{STEM} Datasets in Materials Science. Cambridge University Press ({CUP}). 10.1017/s1431927614002037", "[Simson2015] M. Simson and H. Ryll and H. Banba and R. Hartmann and M. Huth and S. Ihle and L. Jones and Y. Kondo and K. Muller and P.D. Nellist and R. Sagawa and J. Schmidt and H. Soltau and L. Striider and H. Yang (2015): 4D-{STEM} Imaging With the {pnCCD} (S){TEM}-Camera. Cambridge University Press ({CUP}). 10.1017/s1431927615011836", "[Pennycook2015] Timothy J. Pennycook and Andrew R. Lupini and Hao Yang and Matthew F. Murfitt and Lewys Jones and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part 1: Experimental demonstration at atomic resolution. Elsevier {BV}. 10.1016/j.ultramic.2014.09.013", "[Yang2015a] Hao Yang and Timothy J. Pennycook and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part {II}: Optimisation of imaging conditions. Elsevier {BV}. 10.1016/j.ultramic.2014.10.013", "[QuantumMerlin2017] {Quantum Detectors} (2017): Merlin for {EM} technical datasheet. http://quantumdetectors.com/wp-content/uploads/2017/01/1532-Merlin-for-EM-Technical-Datasheet-v2.pdf", "[Lawrence2017] Ethan L. Lawrence and Shery L. Y. Chang and Peter A. Crozier (2017): In situ {TEM} observations of Oxygen Surface Dynamics in {CeO}2 Cubes. Cambridge University Press ({CUP}). 10.1017/s1431927617010637", "[Li2013] Xueming Li and Paul Mooney and Shawn Zheng and Christopher R Booth and Michael B Braunfeld and Sander Gubbens and David A Agard and Yifan Cheng (2013): Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-{EM}. Springer Nature. 10.1038/nmeth.2472", "[XSpectrum-LambdaM2] X-Spectrum: {Lambda2M} large area {Medipix3} based detector array. http://www.x-spectrum.de/index_htm_files/X-Spectrum_datasheet_2M.pdf", "[Li:2018ngp] Li, Xin and Dyck, Ondrej and Kalinin, Sergei V. and Jesse, Stephen (2018): Compressed Sensing of Scanning Transmission Electron Microscopy {(STEM)} on Non-Rectangular Scans. 1805.04957", "[Nielsen1994] Nielsen, Jakob (1994): Usability Engineering. Elsevier Science. https://www.ebook.de/de/product/21119819/jakob_nielsen_usability_engineering.html", "[Lee2011] Craig A. Lee and Samuel D. Gasster and Antonio Plaza and Chein-I Chang and Bormin Huang (2011): Recent Developments in High Performance Computing for Remote Sensing: A Review. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/jstars.2011.2162643", "[Tezduyar1996] T. Tezduyar and S. Aliabadi and M. Behr and A. Johnson and V. Kalro and M. Litke (1996): Flow simulation and high performance computing. Springer Nature America, Inc. 10.1007/bf00350249", "[Allcock2002] Bill Allcock and Joe Bester and John Bresnahan and Ann L. Chervenak and Ian Foster and Carl Kesselman and Sam Meder and Veronika Nefedova and Darcy Quesnel and Steven Tuecke (2002): Data management and transfer in high-performance computational grid environments. Elsevier {BV}. 10.1016/s0167-8191(02)00094-7", "[Pan2016] Ming Pan and Cory Czarnik (2016): Image Detectors for Environmental Transmission Electron Microscopy ({ETEM}). Springer International Publishing. 10.1007/978-3-319-22988-1_5", "[Knuth1974] Donald E. Knuth (1974): Structured Programming with go to Statements. Association for Computing Machinery ({ACM}). 10.1145/356635.356640", "[Busch2016] Busch, Kat (2016): The Rules of Optimization: Why So Many Performance Efforts Fail. https://hackernoon.com/the-rules-of-optimization-why-so-many-performance-efforts-fail-cf06aad89099", "[Delvecchio2011] Delvecchio, Paul (2011): De-Mystifying Software Performance Optimization. https://software.intel.com/en-us/articles/de-mystifying-software-performance-optimization", "[Mavridis2017] Ilias Mavridis and Helen Karatza (2017): Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. Elsevier {BV}. 10.1016/j.jss.2016.11.037", "[Dean2008] Jeffrey Dean and Sanjay Ghemawat (2008): {MapReduce}. Association for Computing Machinery ({ACM}). 10.1145/1327452.1327492", "[Stegmaier2017] Stegmaier, Johannes (2017): New Methods to Improve Large-Scale Microscopy Image Analysis with Prior Knowledge and Uncertainty. KIT Scientific Publishing. 10.5445/ksp/1000060221", "[Goscinski2014] Wojtek J. Goscinski and Paul McIntosh and Ulrich Felzmann and Anton Maksimenko and Christopher J. Hall and Timur Gureyev and Darren Thompson and Andrew Janke and Graham Galloway and Neil E. B. Killeen and Parnesh Raniga and Owen Kaluza and Amanda Ng and Govinda Poudel and David G. Barnes and Toan Nguyen and Paul Bonnington and Gary F. Egan (2014): The multi-modal Australian {ScienceS} Imaging and Visualization Environment ({MASSIVE}) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers Media {SA}. 10.3389/fninf.2014.00030", "[Foust2015] Gabriel Foust and Jaakko J\u00e4rvi and Sean Parent (2015): Generating reactive programs for graphical user interfaces from multi-way dataflow constraint systems. {ACM} Press. 10.1145/2814204.2814207", "[Shneiderman2009] Ben Shneiderman and Catherine Plaisant and Maxine Cohen and Steven Jacobs (2009): Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th Edition). Pearson. https://www.amazon.com/Designing-User-Interface-Human-Computer-Interaction/dp/0321537351?SubscriptionId=AKIAIOBINVZYXZQZ2U3A&tag=chimbori05-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=0321537351", "[Tsukamoto2014] Takafumi Tsukamoto and Takuo Yasunaga (2014): Development of a user-friendly system for image processing of electron microscopy by integrating a web browser and {PIONE} with Eos. Oxford University Press ({OUP}). 10.1093/jmicro/dfu070", "[Rogic] Ivan Rogic: {React}, {Redux} and {Immutable.js}: Ingredients for Efficient Web Applications. https://www.toptal.com/react/react-redux-and-immutablejs", "[Lachlan] Lachlan Miller (2017): Redux-driven {UI} design. https://medium.com/@lachlanmiller_52885/redux-driven-ui-design-8586a84d808f", "[Taylor1996] R.N. Taylor and N. Medvidovic and K.M. Anderson and E.J. Whitehead and J.E. Robbins and K.A. Nies and P. Oreizy and D.L. Dubrow (1996): A component- and message-based architectural style for {GUI} software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/32.508313", "[pyFAI] {Data analysis unit, European Synchrotron Radiation Facility, Grenoble} (2018): General introduction to {PyFAI}. https://pyfai.readthedocs.io/en/latest/pyFAI.html", "[Alted2010] Francesc Alted (2010): Why Modern {CPUs} Are Starving and What Can Be Done about It. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2010.51", "[Clausen2018a] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and M\u00fcller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1478763", "[IntelLustre] {Intel High Performance Data Division} (2014): Architecting a High Performance Storage System. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/architecting-lustre-storage-white-paper.pdf", "[Patel2012] Aditya B. Patel and Manashvi Birla and Ushma Nair (2012): Addressing big data problem using {Hadoop} and {Map Reduce}. {IEEE}. 10.1109/nuicone.2012.6493198", "[Larsen2000] Samuel Larsen and Saman Amarasinghe (2000): Exploiting superword level parallelism with multimedia instruction sets. Association for Computing Machinery ({ACM}). 10.1145/358438.349320", "[Walt2011] St{\\'{e}}fan van der Walt and S Chris Colbert and Ga\u00ebl Varoquaux (2011): The {NumPy} Array: A Structure for Efficient Numerical Computation. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2011.37", "[Thadani19951AE] Moti Thadani and Yousef Y. A. Khalidi (1995): An Efficient Zero-Copy {I / O} Framework for {UNIX} \u00ae.", "[Stancevic2003] Dragan Stancevic (2003): Zero Copy I: User-Mode Perspective. https://www.linuxjournal.com/article/6345?page=0,0", "[Kelly2003] Paul H J Kelly (2003): Advanced Computer Architecture: Caches and Memory Systems. https://www.doc.ic.ac.uk/~phjk/AdvancedCompArchitecture/2003-04/Lectures/Ch03/ACA-CH03-CurrentVersion.pdf", "[Shvachko2010] Konstantin Shvachko and Hairong Kuang and Sanjay Radia and Robert Chansler (2010): The Hadoop Distributed File System. {IEEE}. 10.1109/msst.2010.5496972", "[HDFS-Short] {Apache Software Foundation} (2018): {HDFS} Short-Circuit Local Reads. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html", "[Cooley1965] James W. Cooley and John W. Tukey (1965): An algorithm for the machine calculation of complex Fourier series. American Mathematical Society ({AMS}). 10.1090/s0025-5718-1965-0178586-1", "[Zaharia2016] Matei Zaharia and Michael J. Franklin and Ali Ghodsi and Joseph Gonzalez and Scott Shenker and Ion Stoica and Reynold S. Xin and Patrick Wendell and Tathagata Das and Michael Armbrust and Ankur Dave and Xiangrui Meng and Josh Rosen and Shivaram Venkataraman (2016): Apache Spark. Association for Computing Machinery ({ACM}). 10.1145/2934664", "[Rocklin2015] Matthew Rocklin (2015): Dask: Parallel Computation with Blocked algorithms and Task Scheduling. {SciPy}. 10.25080/majora-7b98e3ed-013", "[BLAS] (2017): {BLAS} ({Basic Linear Algebra Subprograms}). http://www.netlib.org/blas/", "[PyTorch] (2018): PyTorch. https://pytorch.org/", "[MKL] Intel (2018): Intel\u00ae Math Kernel Library. https://software.intel.com/en-us/mkl", "[Godbolt] Matt Godbolt: Compiler Explorer. https://godbolt.org/", "[Numba] (2018): Numba: A high-performance {Python} compiler. http://numba.pydata.org/", "[ReactJS] {Facebook Inc.} (2018): React: A {JavaScript} library for building user interfaces. https://reactjs.org/", "[ReduxJS] Dan Abramov (2018): Redux: A predictable state container for {JavaScript} apps.. https://redux.js.org/", "[MerkelDocker] Dirk Merkel (2014): Docker: Lightweight Linux Containers for Consistent Development and Deployment. https://www.linuxjournal.com/content/docker-lightweight-linux-containers-consistent-development-and-deployment", "[Bernstein2014] David Bernstein (2014): Containers and Cloud: From {LXC} to Docker to Kubernetes. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcc.2014.51", "[Shaw1990] Mary Shaw (1990): Prospects for an engineering discipline of software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/52.60586", "[Schindelin2012] Johannes Schindelin and Ignacio Arganda-Carreras and Erwin Frise and Verena Kaynig and Mark Longair and Tobias Pietzsch and Stephan Preibisch and Curtis Rueden and Stephan Saalfeld and Benjamin Schmid and Jean-Yves Tinevez and Daniel James White and Volker Hartenstein and Kevin Eliceiri and Pavel Tomancak and Albert Cardona (2012): Fiji: an open-source platform for biological-image analysis. Springer Nature. 10.1038/nmeth.2019", "[Willinsky2005] John Willinsky (2005): The unacknowledged convergence of open source, open access, and open science. University of Illinois Libraries. 10.5210/fm.v10i8.1265", "[Zuiderwijk2012] Anneke Zuiderwijk and Keith Jeffery and Marijn Janssen (2012): The Potential of Metadata for Linked Open Data and its Value for Users and Publishers. {JEDEM} Journal of e-Democracy and Open Government. 10.29379/jedem.v4i2.138", "[HDF5] (2016): High Level Introduction to {HDF5}. https://support.hdfgroup.org/HDF5/Tutor/HDF5Intro.pdf", "[Okoli2015] Chitu Okoli and Johannes Nguyen (2015): Business Models for Free and Open Source Software. Elsevier {BV}. 10.2139/ssrn.2568185", "[ApacheFoundation] {{The APACHE Software Foundation}} (2018): The {APACHE} Software Foundation. https://www.apache.org/", "[Linuxfoundation] {The Linux Foundation} (2018): The Linux Foundation. https://www.linuxfoundation.org/", "[Williams1996] David B. Williams and C. Barry Carter (1996): The Transmission Electron Microscope. Springer {US}. 10.1007/978-1-4757-2519-3_1", "[Goto2008] Kazushige Goto and Robert A. van de Geijn (2008): Anatomy of high-performance matrix multiplication. Association for Computing Machinery ({ACM}). 10.1145/1356052.1356053", "[DaskSpark] {Anaconda Inc.} (2018): Comparison to Spark. http://docs.dask.org/en/latest/spark.html", "[Lemire2012] Daniel Lemire and Leonid Boytsov: Decoding billions of integers per second through vectorization. 10.1002/spe.2203", "[grosser2011polly] Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simb{\\\"u}rger, Andreas and Gr{\\\"o}{\\ss}linger, Armin and Pouchet, Louis-No{\\\"e}l: Polly-Polyhedral optimization in {LLVM}. http://perso.ens-lyon.fr/christophe.alias/impact2011/impact-07.pdf", "[Song2018] Dongsheng Song and Zi-An Li and Jan Caron and Andr{\\'{a}}s Kov{\\'{a}}cs and Huanfang Tian and Chiming Jin and Haifeng Du and Mingliang Tian and Jianqi Li and Jing Zhu and Rafal E. Dunin- Homepage: https://libertem.github.io/LiberTEM/GitHub repository: https://github.com/LiberTEM/LiberTEM/PyPI: https://pypi.org/project/libertem/LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a modified MapReduce programming model. The current focus is pixelated scanning transmission electron microscopy (STEM) and scanning electron beam diffraction data. MapReduce-like processing allows to specify an algorithm through two functions: One function that is mapped on portions of the input data, and another function that merges (reduces) a partial result from this mapping step into the complete result. A wide range of TEM and 4D STEM processing tasks can be expressed in this fashion, see Applications. The UDF interface of LiberTEM offers a standardized, versatile API to decouple the mathematical core of an algorithm from details of data source, parallelism, and use of results. Mapping and merging can be performed in any order and with different subdivisions of the input data, including running parts of the calculation concurrently. That means the same implementation can be used in a wide range of modalities, including massive scaling on clusters. Since each merge step produces an intermediate result, this style of processing is suitable for displaying live results from a running calculation in a GUI application and for processing live data streams. A closed-loop feedback between processing and instrument control can be realized as well. See User-defined functions for more details on the LiberTEM UDF interface. The LiberTEM back-end offers high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves very high aggregate IO performance on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel. LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.6+ and the required packages will likely work as well. The GUI is running in a web browser. InstallationThe short version: $ virtualenv -p python3 ~/libertem-venv/ $ source ~/libertem-venv/bin/activate (libertem-venv) $ python -m pip install "libertem[torch]" # optional for GPU support # See also https://docs.cupy.dev/en/stable/install.html (libertem-venv) $ python -m pip install cupy Please see our documentation for details! Alternatively, to run the LiberTEM Docker image: $ docker run -p localhost:9000:9000 --mount type=bind,source=/path/to/your/data/,dst=/data/,ro libertem/libertem or $ singularity exec docker://libertem/libertem /venv/bin/libertem-server Deployment for offline data processing on a single-node system for a local user is thoroughly tested and can be considered stable. Deployment on a cluster is experimental and still requires some additional work, see Issue #105. Back-end support for live data processing is still experimental as well, see https://github.com/LiberTEM/LiberTEM-live. ApplicationsSince LiberTEM is programmable through user-defined functions (UDFs), it can be used for a wide range of processing tasks on array-like data and data streams. The following applications have been implemented already: Virtual detectors (virtual bright field, virtual HAADF, center of mass , custom shapes via masks) Analysis of amorphous materials Strain mapping Off-axis electron holography reconstruction Single Side Band ptychography Some of these applications are available through an interactive web GUI. Please see the applications section of our documentation for details! The Python API and user-defined functions (UDFs) can be used for complex operations such as arbitrary linear operations and other features like data export. Example Jupyter notebooks are available in the examples directory. If you are having trouble running the examples, please let us know by filing an issue or by joining our Gitter chat. LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. Contact us if you are interested! LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. Currently we are working on live data processing, integration with Dask arrays and HyperSpy, support for sparse data, and implementing analysis methods for various applications of pixelated STEM and other large-scale detector data. If you like to influence the direction this project is taking, or if you'd like to contribute, please join our gitter chat and our general mailing list. File formatsLiberTEM currently opens most file formats used for pixelated STEM. See our general information on loading data and format-specific documentation for more information! Raw binary files NumPy .npy binary files Thermo Fisher EMPAD detector files Quantum Detectors MIB format Nanomegas .blo block files Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for DE-Series detectors Gatan K2 IS raw format Stacks of Gatan DM3 and DM4 files (via openNCEM) FRMS6 from PNDetector pnCCD cameras (currently alpha, gain correction still needs UI changes) FEI SER files (via openNCEM) MRC (via openNCEM) HDF5-based formats such as HyperSpy files, NeXus and EMD TVIPS binary files Please contact us if you are interested in support for an additional format! Detectors (experimental)Currently the Quantum Detectors Merlin camera is supported for live processing. Support for DECTRIS cameras and the Gatan K2 IS camera are in a prototype state. Please contact us if you are interested in this feature! See https://github.com/LiberTEM/LiberTEM-live for more details on live processing. LicenseLiberTEM is licensed under GPLv3. The I/O parts are also available under the MIT license, please see LICENSE files in the subdirectories for details. We gratefully acknowledge funding from the Initiative and Network Fund of the Helmholtz Association within the Helmholtz Young Investigator Group moreSTEM under Contract No. VH-NG-1317 at Forschungszentrum Jülich in Germany. We gratefully acknowledge funding from the Information & Data Science Pilot Project 'Ptychography 4.0' of the Helmholtz Association. We kindly acknowledge funding from Google Summer of Code 2019 under the umbrella of the Python software foundation. STEMx equipment and software for 4D STEM data acquisition with K2 IS camera courtesy of Gatan Inc. LiberTEM development is supported by AIDAS Forschungszentrum Jülich is supporting LiberTEM with funding for personnel, access to its infrastructure and administrative support.

Send a message
How can we help?
We usually respond in a few hours.