Summary of the presidents of the ASPLOS’23 program | SIGARCH

For 2023, ASPLOS has embarked on a new review template with multiple deadlines. Multiple deadlines are intended to encourage authors to submit their articles when they are ready, to facilitate the selection of certain articles for review, and to better distribute the reviewer’s workload. In this post, we provide an overview of our experience as ASPLOS’23 program co-chairs, offer thoughts on further improvements, and discuss long-term opportunities.

ASPLOS’23 had 3 deadlines: 31 March 2022, 7 July 2022 and 20 October 2022. Documents sent at each deadline could receive an acceptance, rejection or major review decision. Documents that have been given a major review decision will be given a concrete set of changes to make and 6 weeks to make them before being reviewed by the same reviewers. They were also granted additional space to accommodate requests for review. If the review requests had been met, the document would have been accepted. In 3 deadlines, we received 600 applications. Of these proposals, 151 were accepted: 128 appeared in ASPLOS’23 and another 23 will appear in ASPLOS’24. The overall acceptance rate was 25%. Out of 600 submissions, 90 papers received major review and 69 of these were ultimately accepted (77% acceptance rate). We have awarded 11 Distinguished Paper Awards. The ACM award guidelines place an upper limit of 10% of accepted documents that can be nominated for an award. We wanted to hit this ceiling: giving more awards is good for our community and to showcase the outstanding work of our researchers.

Multiple expiration and major revisions

Multiple deadlines allowed authors to submit their article when they were ready. The first deadline received the least number of submissions (90): the authors received a rather short notice about the new early deadline. We expect future years to see more consistent submissions within deadlines as our community gets used to the process. Multiple deadlines have also opened up the opportunity to give documents an important review decision. Most of the major review papers were eventually accepted, and the reviewers felt that the papers had been significantly improved through the major review process; 2 out of 11 Distinguished Work Awards were given to articles that underwent a major review decision, attesting to the improvement seen by the reviewers. An open question remains as to what the right threshold is for how many major audit decisions need to be issued. Have we given major revisions to documents that we would have accepted outright if major revision was not an option? While reviewing a revised document is easier than reviewing a new document, major revisions represent an increased workload for PC members. On the other hand, majorly reviewed docs typically didn’t need guidance and may not have needed as much work for the camera-ready version. Finally, we wanted all deadlines to offer the same opportunity; therefore, major revisions were offered at the autumn deadline that would appear in ASPLOS’24. This has two drawbacks: 1) there is a long window between when the paper is accepted and when it appears at the conference, and 2) it extends PC member service beyond the conference itself and stretches service to more than 1 year.

Auditor’s workload

At each deadline, we expanded our PC to cope with the higher-than-expected workload. Despite a large PC, PC members reviewed an average of 23 items, which is higher than a typical load of ~15 items. The increased load was offset by a longer window for service; those 23 documents were not all reviewed in a typical PC’s 2-3 month period. However, many reviewers found that frequent deadlines and long lengths of service took their toll on them; The ASPLOS service also overlapped with other commitments. 78% of our PC members felt the service on ASPLOS’23 was « A little more work » or « A lot more work » than previous ASPLOS PCs. To relieve the pressure on the PC, you can recruit larger PCs or invite PC members to serve for less than 3 deadlines. Additionally, we encourage those accepting an invitation to serve on the ASPLOS PC to think carefully about additional engagements as serving on multiple overlapping PCs is not good for individual researchers, their productivity, and the quality of their reviews. We have a large and diverse community and we need to ensure that the service load is well distributed across that community.

Going forward

To host 128 documents at ASPLOS’23, the program was organized in 3 parallel tracks in 3 days. Each talk had a time slot of 16.5 minutes: 12 minutes for presenting followed by 4.5 minutes for questions. The program included 2 poster sessions, to allow for more meaningful engagement between authors and participants. We expect next year’s ASPLOS program to be larger as we have already accepted 23 papers from the Fall 2022 major reviews. This leads to questions about how to scale the conference: more tracks, more days, shorter talks, featured papers, etc.?

The review model for ASPLOS also opens up the opportunity to publish ASPLOS proceedings as part of the Proceedings of the ACM series. Publishing ASPLOS as a journal in this form would benefit colleagues in institutions who still value the journals.

We are grateful to the steering committee for their input throughout the process, to our program committee members for their dedication to the process especially in the face of a high and unforeseen caseload, and to Tor Aamodt who, as general chairman, has Crafted an outstanding conferencing experience for all!

We welcome community thoughts on the overall process used in ASPLOS’23; this is a new experiment and there is always room for improvement. We invite you to comment below with your thoughts. The interested reader may also wish to view the Recording of the ASPLOS’23 work meeting which featured some open discussions about the process. More details can also be found in our 3 Program Chair Messages in the ACM DL: Tome I, Volume 2AND Volume 3.

About the Authors: Natalie Enright Jerger (University of Toronto) and Mike Swift (University of Wisconsin-Madison) served as co-chairs of the ASPLOS’23 program.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author, and do not represent those of ACM SIGARCH or its parent organization, ACM.