publication . Article . 2015

Development of site-oriented analytics for grid computing centres

Washbrook, A.; Crooks, D.; Roy, G.; Skipsey, S.; Qin, G.; Stewart, G.P.; Britton, D.;
Open Access English
  • Published: 23 Dec 2015
  • Publisher: IOP Publishing
Abstract
The field of analytics, the process of analysing data to visualise meaningful patterns and trends, has become increasingly important in scientific computing as the volume and variety of data available to process has significantly increased. There is now ongoing work in the High Energy Physics (HEP) community in this area, for example in the augmentation of systems management at WLCG computing sites. We report on work evaluating the feasibility of distributed site-oriented analytics using the Elasticsearch, Logstash and Kibana software stack and demonstrate functionality by the application of two workflows that give greater insight into site operations.
Subjects
free text keywords: QC
Download from

[1] Elastic [online] Available http://www.elastic.co [accessed 15/05/2015]

[2] Logstash-forwarder [online] Available https://github.com/elastic/logstash-forwarder [accessed 15/05/2015]

[3] Computing Resource Execution And Management Service (CREAM) [online] Available https://wiki.italiangrid.it/CREAM [accessed 15/05/2015]

[4] Son of Grid Engine Project [online] Available https://arc.liv.ac.uk/trac/SGE [accessed 15/05/2015]

[5] Elasticsearch API [online] Available https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html [accessed 15/05/2015]

[6] Elasticsearch API [online] Available https://elasticsearch-py.readthedocs.org/en/master/ [accessed 15/05/2015]

[7] Graphite [online] Available http://graphite.readthedocs.org [accessed 15/05/2015]

[8] Grafana [online] Available http://ww.grafana.org [accessed 15/05/2015]

Related research
Abstract
The field of analytics, the process of analysing data to visualise meaningful patterns and trends, has become increasingly important in scientific computing as the volume and variety of data available to process has significantly increased. There is now ongoing work in the High Energy Physics (HEP) community in this area, for example in the augmentation of systems management at WLCG computing sites. We report on work evaluating the feasibility of distributed site-oriented analytics using the Elasticsearch, Logstash and Kibana software stack and demonstrate functionality by the application of two workflows that give greater insight into site operations.
Subjects
free text keywords: QC
Download from

[1] Elastic [online] Available http://www.elastic.co [accessed 15/05/2015]

[2] Logstash-forwarder [online] Available https://github.com/elastic/logstash-forwarder [accessed 15/05/2015]

[3] Computing Resource Execution And Management Service (CREAM) [online] Available https://wiki.italiangrid.it/CREAM [accessed 15/05/2015]

[4] Son of Grid Engine Project [online] Available https://arc.liv.ac.uk/trac/SGE [accessed 15/05/2015]

[5] Elasticsearch API [online] Available https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html [accessed 15/05/2015]

[6] Elasticsearch API [online] Available https://elasticsearch-py.readthedocs.org/en/master/ [accessed 15/05/2015]

[7] Graphite [online] Available http://graphite.readthedocs.org [accessed 15/05/2015]

[8] Grafana [online] Available http://ww.grafana.org [accessed 15/05/2015]

Related research
Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue