• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site

Article

Disk storage management for LHCb based on Data Popularity estimator

Hushchyn M., Charpentier P., Ustyuzhanin A.

This paper presents a system providing recommendations for optimizing the LHCb data storage. The LHCb data storage system is a hybrid system. All datasets are kept as archives on magnetic tapes. The most popular datasets are kept on disks. The recommendation system takes the dataset usage history and metadata (size, type, configuration etc.) to generate a recommendation report. In this article present how we use machine learning algorithms to predict future data popularity. Using these predictions it is possible to estimate which datasets should be removed from disk. We use regression algorithms and time series analysis to find the optimal number of replicas for datasets that are kept on disk. Based on the data popularity and the number of replicas optimization, the recommendation system minimizes a loss function to find the optimal data distribution. The loss function represents all requirements for data distribution in the data storage system. We demonstrate how the recommendation system helps to save disk space and to reduce waiting times for jobs using this data.