IISWC 25th Anniversary | SIGARCH

“If you can’t measure it, you can’t improve it.”- Lord Kelvin

The IEEE International Symposium on Workload Characterization (IISWC) just celebrated its 25th anniversary. This is an excellent opportunity to look back and forth at what has happened over the past 25 years and what is to come. This blog post shares some of what was said at the IISWC 25th Anniversary panel at the most recent conference, which was held in November 2022 in Austin, Texas. Speakers included John Carter (IBM), Lizy K. John (UT Austin), David Kaeli (Northeastern), Vijay Janapa Reddi (Harvard), Carole-Jean Wu (Meta), and Neeraja J. Yadwadkar (UT Austin). I had the honor of moderating the panel.

The quote at the beginning of Lord Kelvin’s blog post captures the essence of what IISWC is all about i.e. IISWC covers measurement techniques, characterization methodologies and workload analysis, which form the basis for any research cycle and development of architectures and systems to optimize computer system performance, power, reliability, security, etc. Without a proper understanding of workloads, it is simply impossible to design a well-optimized and balanced computer system for the workloads it will perform. Building and developing representative benchmark suites and methods for workload analysis is difficult and requires a lot of time and effort. IISWC provides a forum for researchers in academia as well as practitioners and developers in industry to present and discuss recent advances in workload characterization.

The IISWC’s mission is to promote research and application of workload characterization studies and techniques, and to provide a forum for presenting and discussing the latest research findings and insights related to workload characterization. To quote a former chair of the steering committee, “IISWC is the only symposium of its kind in the world. If you’re interested in what computing workloads are like today, IISWC is where you can get answers.

IISWC began in 1998 in Dallas, Texas as the Workshop on Workload Characterization (WWC), which was held concurrently with the IEEE/ACM International Symposium on Microarchitecture (MICRO). Lizy K. John and Ann Marie G. Maynard (IBM) co-founded and co-organized the first editions of WWC for several years until 2004. In 2005, the workshop became a symposium with a steering committee and a series of statutes to drive the event into the future and create a vibrant community around the conference. The conference has been held annually in a variety of locations throughout the United States, from the West Coast (Seattle, Portland, San Jose, San Diego), South (Austin, Dallas), and East Coast (Boston, Providence, Raleigh , Atlanta, Orlando), as well as virtually from Beijing, China.

The workloads analyzed over the years at IISWC have changed dramatically, as have the evolution of real-life computing workloads over the past 2.5 decades. While the initial emphasis was on Java and server workloads, embedded computing, memory analysis and tracing tools (1998-2002), the focus shifted towards commercial, bioinformatics, parallel workloads and high performance computing (2003-2007). The introduction of on-chip multiprocessors has shifted the emphasis to multi-core, multi-threaded, and transactional memory workloads; in addition, GPGPU computing began to emerge, as did internet-scale mobile and interactive workloads (2008-2012). The emergence of hyperscale data centers and the end of Dennard scaling pushed the conference’s focus towards cloud computing, big data, and power/energy characterization (2013-2017). The main focus of interest has recently shifted towards GPU and graph analytics workloads as well as deep neural networks and machine learning workloads (2018-2022). The word cloud above was taken based on print headlines over the past 25 years.

It is humbling to see the impact of the conference over the years. IISWC has developed several benchmark suites. Some of these have been widely adopted by the community or are expected to make an impact soon. MiBencha suite of benchmarks for embedded computing published at WWC in 2001, has been cited 4,637 times, according to Google Scholar. Rodiniaa suite of GPGPU benchmarks published at the IISWC in 2009, has been cited 3,320 times and STAMPa suite of transactional memory benchmarks published at the IISWC in 2008, was cited 1,277 times. ILLIXRa suite of virtual/extended reality benchmarks recently released at the IISWC 2021 was selected as an IEEE Micro Top Pick.

IISWC is much more than a conference that publishes benchmark suites. Indeed, most of the papers published on IISWC can be divided into three main groups: methodology, workloads and systems. Methodology papers cover broad topics such as benchmarking, profiling, tracking, simulation, hardware performance counters, sampling, phase analysis, workload synthesis, etc. Workload papers describe new benchmark suites and in-depth analyse, characterize and explain new application domains. Systems documents look at how workloads affect systems and how workloads affect systems, not just in terms of performance but also power, energy, thermals, reliability, and so on. A variety of methodological papers and extensive characterization studies have had a significant impact with over 100 citation counts.

The future of workload characterization and IISWC is bright. Workload analysis and understanding are the foundation of our field and our R&D cycle. Workloads are constantly changing and we, as a community, need to keep up with the times. At the end of the day, we’re designing computer systems that will run our future workloads, not just today’s benchmarks, let alone yesterday’s. The workloads that emerge on the horizon will inevitably be different from today’s workloads, as we are seeing the emergence of hybrid cloud, edge and Internet-of-Things computing, quantum computing, artificial intelligence, machine learning, deep learning recommendation , domain-specific computing, virtual and augmented reality, accelerator-rich and heterogeneous systems, robotics, healthcare, microservices and serverless computing, wearables, etc.

In terms of workload characterization methodologies, the use of new machine learning and data analysis techniques could foster a new breadth of characterization and analysis methodologies to be able to see the forest for the trees in the huge quantities of data that current performance contrasts, profiling, instrumentation and simulation techniques provide when analyzing increasingly complex computer systems. Additionally, we need cross-cutting technologies to accelerate innovation and reference technology for fast-growing domains to stay ahead of workload trends. We also need robust and reproducible methodologies for collaborating within the community, which has been a challenge for many years and could become even more challenging as machine learning gets involved in the system stack. We should develop methodologies to model and characterize workloads at extreme scale in hyperscale data centers and across the many mobile and edge devices equipped with heterogeneous hardware. Sustainability is a major challenge for our society and generation, and in line with the IISWC motto “to measure is to know”, we need to develop methodologies, metrics and characterization techniques to analyze and ultimately improve the sustainability of information systems. Given the scale of the challenges, this requires a community-led approach to unify our efforts to collectively accelerate progress and adoption.

In summary, IISWC has served the community well for the past 25 years and is expected to continue to do so for the next 25 years and beyond. Workload characterization is and remains a fundamental foundation for our field. There are great challenges and fantastic opportunities ahead of us to generate impact.

This post was written by Lieven Eeckhout, with input from the IISWC 25th Anniversary Speakers.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author, and do not represent those of ACM SIGARCH or its parent organization, ACM.