The CMS Submission Infrastructure team manages a set of federated HTCondor pools aggregating 450K+ CPU cores across dozens of sites worldwide, with resources acquired dynamically using the GlideinWMS pilot model. The growing computing needs of the CMS experiment make it important not only to grow the pool’s resources, but also to get more out of the resources we already have.

We could already measure efficiency from the payload job and site batch system perspectives, but neither of these gave us the information we needed to evaluate and expand advanced strategies to improve efficiency at the pilot level, such as overloading.

In this contribution we share how we leverage HTCondor activity logs to fill this gap, producing a record of resource usage for each completed pilot and aggregating them via CERN’s monitoring infrastructure. We conclude with how this new perspective has been supporting our efforts to improve pilot efficiency in CMS.

Associated Links

• Presentation Slides (PDF)
• Indico Agenda