HTCondor AMD GPU support is currently broken, as it would not recognize the GPUs present on the SDSC Cosmos system. The problem seems to be in the initial GPU discovery phase.
Moreover, the standard containerization tool used by HTCondor to launch user jobs, namely Apptainer, also does not work correctly with AMD GPUs, and needs to be fixed before the OSPool can effectively use resources providing AMD GPUs.
AI workloads are becoming more important in the scientific research domain, so properly supporting this use-case is becoming increasingly important in the OSPool, too. As with many other compute workloads, the data handling part of AI workloads requires heightened attention on the HTC-oriented OSPool resources. Since the OSPool does not offer a shared file system, the Pelican software stack is the recommended way of handling data. To make access to the Pelican-managed resources easier, the PelicanFS Python bindings have been developed, allowing for transparent access to such data from typical AI workloads, e.g. the PyTorch-based ones.
The existing OSPool pilot setup on Kubernetes in the NRP relies on nested containerization, which is not supported on the majority of Kubernetes deployments and has security drawbacks, too. This document proposes an alternative pilot setup that only uses Kubernetes-native mechanisms to make it usable on most Kubernetes-managed resources. In order to implement this new pilot setup, HTCondor will need to become Kubernetes aware. This document provides pointers to the envisioned Kubernetes mechanisms to be used by various HTCondor processes, as well as a couple simplified pilot-like Kubernetes examples that exercise most of those mechanisms.
The ShadowLog provides a detailed description of a job as it moves through the condor system from when it is submitted until it terminates. For a typical job, the logs are relatively concise and understandable.
The Open Science Data Federation (OSDF) is an OSG service designed to support the sharing of files staged in autonomous “origins” for efficient access to those files from anywhere in the world via a global namespace and network of caches. The OSDF may be used either standalone - allowing data to be downloaded via HTTPS - or with HTCondor managing data transfer for compute jobs running on one of the many resource pools supported by OSG.
One challenge for High-throughput computing (HTC) is integrating with the computational cloud or other clusters to provide more computational power. HTCondor Annex can create working nodes on clusters using schedulers such as HTCondor, Slurm, and others.
While executing scientific applications, they must sometimes stop the process due to hardware problems or even job end-of-life. In that case, some applications can create a set of files to save their current state to be loaded on a restoring process at a later time. However, most applications do not have this feature. CRIU (Checkpoint Restore in Userspace, pronounced kree-oo) is a tool for checkpointing and restoring applications in GNU/Linux environment. With CRIU it is possible to stop an application, save the working memory on disk and restore this state. As the OSPool is built around the notion of pre-emptable resources, this could be very useful on jobs that get pre-empted or exceed allocated runtimes.
HTCondor and Glidein Workflow Management System (GlideinWMS) are two of the tools that provide access to computational resources on the OSG pool. Our goal is to check the information about computational resource provisioning, based only on HTCondor and GlideinWMS.
The OSPool has mostly been relying in GlideinWMS for the provisioning of its execute resources. While that worked reasonably well for Grid-type resources, it currently lacks support for many non-Grid resource types, including Kubernetes and Lancium. Adding support for those kind of resources to GlideinWMS would require a non-trivial amount of work, so an alternative approach was investigated.
OSG currently accounts for GPU resources in “GPU chip hours” [[1][source1]]. The GPUs are however not all the same, some are small and some are big; the science delivered from different models thus varies by an order of magnitude. Moreover, a single GPU could be shared between multiple jobs.
We thus propose that OSG starts treating GPUs similarly to CPUs, i.e., switches GPU-core-hours, like we do for CPUs.
For this proposal, we are concerned only about wall time measures due to the combined gWMS and HTCondor systems. The proposed classification thus may be understood largely as a walk through the “state machine” of how these two systems work together.
The objective of these tests is to check if it is faster to transfer each file separately or compress all the files on a tar.gz file for a job on the Open Science Grid (OSG) using the HTCondor.
This document provides a set of results about the data access in the Open Science Grid (OSG). The objective is to show issues on data access of an OSG job. All the process follows a methodology to understand how a user requests the information and any problem with this process.
This document provides an assessment of the feasibility to use the NVIDIA A100 GPU Multi Instance Graphics (MIG) capabilities inside OSG, with a particular emphasis on IceCube. MIG allows an A100 GPU to be split in several partitions that can be assigned to independent compute jobs. (MIG is also available on the NIVIDA A30 GPUs, but we currently have no HW to test).
GIL conducted an open dicussion with the entire OSG [email protected] mailing list on March 9, 2021. The team reviewed the proposal document and concluded the following.
