Machine Learning Optimizations of the EASY Backfill Scheduler for HPC Systems

Abstract

The goal of this research is to implement machine learning techniques to improve task scheduling algorithms in multi-core architectures. In this study, multi-core architectures are represented by High Performance Computing (HPC) systems, with a particular focus on enhancing the EASY Backfill scheduling algorithm. To achieve this objective, a supervised learning model was developed to estimate the user-provided execution time, followed by a reinforcement learning model designed to refine and stabilize these predictions. The supervised model, based on the TabTransformer architecture, was trained using data in the Standard Workload Format (SWF) such as HPC2N and CEA-Curie. These two datasets contains around 500,000 jobs collected during 5 years. The model was implemented within Batsim, an HPC resource management simulator. After implementation, the supervised model demonstrated improved performance compared to the baseline but tended to be overly aggressive, often underestimating job runtimes. To mitigate this issue, the reinforcement learning model was introduced to predict a dynamic padding factor, which increased the job success rate from 36% to 83% while maintaining stable performance across key metrics such as makespan, mean turnaround time improved by 61%, mean waiting time improved by 57%, and mean slowdown improved by 46%.