publication . Other literature type . Report . 2016

Modernizing the monitoring of Mass Storage systems

Gupta, Rosy; Rousseau, Hervé; Ponce, Sebastien;
Open Access English
  • Published: 05 Sep 2016
Abstract
Project Specification The work is a project of IT-ST group, dealing with mass storage systems at CERN and responsible for more that 150PB of data stored on disks and tapes. The task consists of modernizing the logging system of one of of the storage systems namely CASTOR, in particular by integrating it into common monitoring framework provided by the IT department and by moving to new display technologies. The main tools used are Hadoop/HBase, Spark, Python, and web technologies like Django, Graphite and Grafana. Abstract The LHC has entered a new regime by producing more than 25PB of data since the beginning of this year. Thus the storage and monitoring of log...
Subjects
free text keywords: Computing and Computers, CERN openlab summer student
Download fromView all 3 versions
CERN Document Server
Other literature type . 2016
Zenodo
Other literature type . 2016
Provider: Datacite
ZENODO
Report . 2016
Provider: ZENODO

2 Work done 6 2.1 Log producing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.1 Simple log producer to Kafka . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Simple log producer to flume to Kafka . . . . . . . . . . . . . . . . . . . . 7 2.1.3 Flume with interceptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.4 Chosen solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Long-term storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Debug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.4 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue
publication . Other literature type . Report . 2016

Modernizing the monitoring of Mass Storage systems

Gupta, Rosy; Rousseau, Hervé; Ponce, Sebastien;