Fostering Community Amongst Developers and Researchers: A Reflection on the 2024 European Autumn HTCondor Workshop

Advancing scientific research and HTCondor usage was not the only goal of the European HTCondor Workshop. It was also an occasion to grow a community with colleagues and fellow HTCondor users, and exchange experiences and feedback with experts. This year’s European HTCondor workshop took place in Amsterdam from September 24th to 28th. As the 10th year of the workshop, the week-long experience continued to center on presentations and social events catered to fostering collaboration between the HTCondor team with the individuals, institutions, and organizations that use the HTCondor Softwares Suite (HTCSS) and services. In promoting this sort of partnership, the Autumn Workshop in-person experience fostered the feeling that developers and users work is interconnected and illuminated that both parties are a part of something larger: an interdisciplinary, global collaboration between high throughput computing and research-focused individuals and organizations.

This year, Miron Livny (Director of the UW Center for High Throughput Computing (CHTC), Brian Bockelman (Investigator), Todd Tannenbaum, (HTCondor Technical Lead), Andrew Owen (Research Computing Facilitator), and Cole Bollig (HTCondor Core Developer) all attended the workshop on behalf of the CHTC. Including key individuals in the development of throughput computing and HTCondor, the five, in tandem, presented in a total of 19 workshops and panels over the week. From topics ranging from “What’s New and What’s Upcoming with HTCondor” to the “Philosophy and Architecture” of HTCondor, these participants worked directly with users to discuss the application, and implementation of HTCondor for their scientific projects.

HTCondor is an advanced software suite developed by CHTC for harnessing the power of a network of clustered computers. Rather than overloading an individual computer with a computational task, HTCondor will split a task into simultaneously running self-contained tasks, yielding results that otherwise might have taken weeks or even years to complete on a single system. The application of such software is vast, providing enormous amounts of computing power to researchers in multiple disciplines. HTCondor is not only an open source high-throughput computing software that aids research efforts, but it’s an innovative working research effort itself. As such, feedback, collaboration, and community is critical to furthering both HTCondor and the work it supports; and all three of these features are encouraged and promoted by experiences such as the European HTCondor Workshop.

With approximately 70 attendees, representing 27 organizations and over 13 countries, interaction with users and administrators was the forefront priority of the conference. Much of the work of CHTC promotes connections across disciplines and time zones. Yet, as high throughput computing (HTC) is the execution of computational work as numerous self contained tasks, this work is predicated by division of jobs into smaller groups. Seemingly, the software side mimics the people side: with thousands of people across the globe using and working on HTCondor (a community) but in a socially isolated manner. Because of this, in-person conferences have a significant value. As attendee Cole Bollig notes, “there is nothing like getting in a room together and just talking to people and addressing their specific needs. That sort of collaboration happens best in person…As someone attending this for the first time, a big thing for me was putting faces to email addresses. We lovingly call a lot of people who work on HTCondor our ‘European friends,’ so it was just nice to finally meet them, my peers in England and Germany.”

To promote this sort of collaboration, Chair of the Local Organizing Committee, Mary Hester (Linux DevOps Engineer at Nikhef) explained how cultivating a social, relaxed environment was central to this experience. She notes, “I first attended this workshop in 2023 since we had recently switched over to HTCondor at Nikhef. I wanted to get to know the community and have collaborative support. Now, as an organizer, I wanted to help bring people together since getting to talk to people last year was so helpful for me personally.” Hester, along with other organizers put an emphasis on the social aspect of the conference. To bring together users and administrators with developers to exchange experiences, the schedule for this conference was composed of about two daily workshop sessions, and a town hall and an office hour session. Additionally, the schedule allotted for a daily group lunch, multiple coffee breaks, and two evening social events. The impact of this led to constant mingling. As Cole Bollig states,” Often, I would be walking out of a session and continuing a conversation with someone who asked a question or I would be the one still asking questions. It was easy to keep talking about sessions outside of session; I was hunting down people or being hunted down during the breaks. Plus, coffee was provided! Which is always a plus.”

The coffee was actually a crucial part of the experience. Hester explains that, “it’s very important in Dutch culture to have good coffee in the workplace!” With brand new coffee machines and a newly remodeled building, Nikef, supported by workshop sponsors NetApp, AMD-GPU and Lenovo, ensured attendees felt caffeinated and comfortable while asking questions in a casual way. As Hester elaborates, “It’s extremely valuable to have facetime with people, but more valuable to have a setting where people are relaxed enough to get the most out of the limited facetime.” This included a fresh Stroopwafel cart that Hester organized for the final day of the conference.

Organizational efforts to promote collaboration coupled with the small size of the conference resulted in in-depth discussions. Having presented and led a workshop on DISK enforcement, Bollig reflects, “I was presenting for a 30-minute slot in this really small room, and I was worried about filling up the full 30 minutes. I only had eight slides! But, with all the conversation, we actually ended up going super overtime. I was surprised by how much we all had to talk about.” Compared to larger conferences, the EU Workshop’s smaller scale meant less formality in discussions. Hester and Bollig both commented on how the EU workshop has a shared, unspoken, notion that individuals can ask questions as they arise—even in the middle of a presentation.

These sorts of discussions led to surprising discoveries. Notably, the collective interest in the idea of a Prometheus exporter. Currently, used by CHTC’s Pelican Project, Prometheus is a newer software that helps visualize and store metrics. While CHTC was already working on a first class Prometheus exporter, hearing administrators’ enthusiasm for the project resulted in CHTC attendees prioritizing the project, making it a goal for 2025.

Additionally, scientific endeavors aided by HTCondor were shared at the conference. Zef Wolff, a PhD student in Experimental Physics at Nikhef noted that “cluster systems are a foundational building block” of all the work he does. Wolff’s works on the ATLAS Experiment, an international collaboration of physicists, engineers, technicians, and students, responsible for ground-breaking discoveries, such as that of the Higgs boson, an unstable, fundamental particle that gives mass to other particles such as electrons and quarks. Wolff relies on HTCondor to model numerous measurements of parameters and interactions of the Higgs boson, to determine if they are consistent with Standard Model predictions. The vast implications of computing resources like HTCondor also prove to be useful for the search for neutrinos and radio detection of air showers, work that is conducted in Argentina’s Pierre Auger Observatory. Researcher Kevin Almeida Cheminant presented on Monte Carlo simulations, intensive computational algorithms to predict uncertain events, and how they are needed to describe the development of air showers, which are cascades of subatomic particles and ionized nuclei that occur when a cosmic ray enters the Earth’s atmosphere. Furthermore, it was shared that HTCondor is a resource for the development of the Einstein telescope, a gravitational wave observatory expected to start construction in 2028. High performance computing and Cloud Research Technician Luca Tabasso presented on how HTCondor is being used for gravitational wave research on this project; similar to how it was used by LIGO, the group that was awarded the 2017 Nobel Prize for Physics for its detection of gravitational waves from the collision of two black holes. LIGO is an collaborator and significant user of HTCondor. These three projects, among others, presented on how HTCondor aids research efforts, further emphasizing the impact of global collaboration between high throughput computing and scientific research. To read more about individual presentations and review presentational materials, more can be found on the conference website, here.

Overall, the 2024 European Autumn HTCondor Workshop fostered new relationships, deepend older ones, and promoted freeform bantering and questioning that led to an increased understanding and use of HTCondor. Following the success of the 2024 Workshop, next year’s EU workshop will be hosted at FZU, the Institute of Physics of the Czech Academy of Sciences in Prague from September 16th-19th, 2025.

…

To learn more details about the 2024 European Autumn HTCondor Workshop, check out the workshop site here..