What are the most cited ISCA documents?

I was talking to my colleague from UC Berkeley recently Sophia Shao about 50th International Symposium on Computer Architecture (ISCA) on June 17, and we discussed which documents are highly cited. The conversation inspired me to try and find the 50 most cited papers from the 50 ISCAs.

Sadly, Google Scholar doesn’t provide a simple query to answer this question, so I tried other means. My colleague from Google Partha Ranganathan recently showed me the “ISCA Hall Of Fame(HOF), which is a list of the most prolific authors of ISCA papers that was developed and maintained by the good folks at the University of Wisconsin at Madison. With the help of Google Scholar, I searched for highly cited ISCA articles from the 118 HOF authors who individually published 8 to 41 ISCA articles and who collectively wrote 1075 ISCA articles. I have found that many highly cited articles by these authors have been published elsewhere (eg. SPRAY, HPCA, MICRO, sc), so keep that in mind. THE IEEE Xplore repository has a table of contents for almost all of the ISCAs from 1988 to 2022, so I used Google Scholar to see if articles with high Xplore citations are in the Top 50. I also checked the citation count of the Influential ISCA Paper Award winners, which are awarded 15 years after the publication of a paper. It started in 2003, so it covers 1988-2007. Finally, very late, I learned of an effort by Gaurang Upasani et al. to use an internal Google Scholar API to write to article on the main ISCA citations per year of the conference. We compared notes to debug both of our lists.

The 2-part table lists the top 50 most cited ISCA papers after this process, showing first author (underlined) and the authors HOF (in italics), if the item has won the ISCA award and the type (see below). Google Scholar citation count update hourly; this table is a snapshot of the April 25 quotes. These 50 documents are approximately 2% of the 2134 ISCA documents from 1973 to 2023.

Table 1 part A (PDF version with live links to documents)

top 25 ISCA articles by number of citations (PNG image)

Table 1 part B (PDF version with live links to documents)

Next 25 ISCA papers by number of citations

As a numbers guy I find the table fascinating, but keep in mind the cautionary comment of the HOF sitewhich I believe also applies to citation ranking:

“A true Hall of Fame should be determined by impact, not card counting.”

However, it’s hard not to notice Norm Jouppi— also at Google — has won 2 Influential ISCA Paper Awards and co-authored 2 of the top 10 and 5 of the top 50 list. Others who co-authored several top 50 papers are Joel Emer (5), Bill Dally (4), Anop Gupta (4), Douglas Burger (3), Kourosh Gharachorloo (3), John Hennessy (3), Honor Felix (3), and Dean Tullesen (3). These 9 of the 125 ISCA HOF architects co-authored 19 of the top 50 articles. In other words, of the top 35 HOF authors (≥15 articles), only 10 have multiple Top 50 articles, 15 have one article, and 10 have none.

I then dive one level deeper in this data. First I divided the top 10 and all 50 articles by type and topic. My 3 types are:

1. Microarchitecture: Architecture techniques that could be used inside many computers;
2. Architecture: a description or proposal of a complete computer architecture; AND
3. Tools: Tools to help architects design computers, such as simulators or benchmarks.

Looking at the mix of types in the pie charts below, it’s no surprise now that the tools that help architects are a larger fraction of the top 10 than the entire 50, but that wasn’t obvious to me before. In fact, the Top 50 list only has 5 instrument cards, but 3 are in the top 5.

Distribution of Top ISCA publications by type

Hopefully the architecture arguments in the pie charts below are self-explanatory. Since the mantra of chip design today is the area of ​​power performance, it’s no surprise that parallelism and power are big slices. Machine learning (ML) accelerators have an unexpectedly large share of the top 10 and all 50, since the current enthusiasm for deep neural networks only started a decade ago: the 9 ML articles in the table are from 2014 as of 2017. ML items in general are among the most highly cited in all of science and engineeringfor example, the ResNet ML card as of 2016 it has over 161,000 citations. Therefore, the huge popularity of ML today probably accelerates citations to ML accelerators.

The best ISCA papers by topic

I showed the data to Thierry Tambe, who was visiting Berkeley. As a young architect, he wondered which documents were the most cited in the last five years. There is no easy way to check recency in Google Scholar, so I manually adjusted the quotes. Based on the data I gathered for the first table, as long as an article has at least 700 citations, it is probably one of the most cited articles. The table below answers Thierry’s question.

Table 2. Top 50 ISCA documents cited since 2018

As you might expect, accelerated ML docs dominate the most recently cited docs, filling the top 7 slots and 9 of the 15. Sophia Shao noted that, unlike the Top 50, most were architectures, presumably because it is easier for small groups of researchers to design new domain-specific architectures than generic ones. These articles are also more recent than the Top 50: their median year is 2014 versus 2003 for the Top 50. My colleague Cliff Young noted the staying power of the SPLASH-2 document, which appeared 12 years before the next older document. The article at the top of Table 2 has added ~1200 citations since January 2022 compared to ~120 for the article at the top of Table 1, so it could become the most cited next year.

Finally, I observed how Top 10 and Top 50 citations change over time. The histogram below plots the number of articles cited by conference year. It takes a few years after an article has been published before it can attract 1000+ citations, so it’s not shocking that the last highly cited article is from 2017.

Distribution of the best ISCA articles by year of publication

The scarcity of documents from the 70s and 80s AND shocking. Speaking as the author of some of those papers, I have three observations that might explain what happened (aside from the fact that they were irrelevant!)

1. ISCA predates search engines and digital libraries. For example, ACM Digital Library it started in 1998. Finding related work in the 1970s and 80s was very taxing: you had to go to libraries to read physical papers and chase references, which reduced the number of references per item.
2. ISCA and ASPLOS changed paper length policies for references in ~2015, hoping to increase the number of citations to computer architecture documents. Digital libraries plus no limit major references. The 27 articles of the 1978 ISCA had an average of 12 references (316 total), while 40 years later the ISCA had 63 articles with an average of 58 citations or 5 times as many (3636 total). The 2018 ISCA total was ~12 times higher than in 1978.
3. There are also far more outlets for papers today thanks to services like arxiv and more architecture conferences accepting more papers. David Wood he pointed out that in 1990 ASPLOS was biannual so ISCA plus half of ASPLOS equaled ~50 journals. ISCA, ASPLOS (now annual), and HPCA (started 1995) have published ~240 papers in the last year, or ~5 times as many papers. Thus, there may be 5×5 or 25 times more citations from refereed architecture papers published in 2023 than there were in 1990.

The table on citations from 2018 suggests that recent articles are more likely to cite newer articles than older ones. If so, then the above three observations explain why the 1970s and 1980s may have been at a competitive disadvantage, but they weren’t necessarily irrelevant. These hurdles make it all the more impressive that a paper from the first ISCA in 1973 made the top 50 list (#43). Its author HOF was Jack Lipovskywho was also the first general president of the ISCA.

It seems fitting to leave the final words to Jack, whose remarks from 50 years ago remain just as relevant today:

This symposium could be, in hindsight, ten years from now, a decisive turning point in computer architecture. With the dissolution of the Joint spring and autumn computer conferences, a major forum for Computer Architecture has been lost. So we started an annual Computer Architecture Symposium, to be rotated from year to year around the world. …

Symposium papers point to the growth of computer architecture as a science. While it is difficult to explain the reasoning behind decisions made in an architecture, especially the architecture of a practical machine, this reasoning is the basis of a science. …

We hope that the participants underline the questions about the reasoning behind the architecture and that the authors prepare themselves for these questions. If this becomes a tradition at this annual symposium, it should steer authors toward scientific explanation of their architectures for subsequent symposiums.

— Jack Liposvki, Foreword to the Proceedings of the First Annual Computer Architecture Symposium,

University of Florida, December 9-11, 1973

About the author: David Patterson it’s a UC Berkeley Pardee Emeritus ProfessorA Distinguished Google EngineerTHE RIO laboratory Director, and the RISC-V International Vice President of the Board. Probably his most influential projects at Berkeley were HEAT AND RAID. The most important of his seven books is Computer architecture: a quantitative approach. ACM SIGARCH presented him with the Eckert-Mauchly Prize and the Alan D. Berenbaum Distinguished Service Award. He and his book co-author John Hennessy shared the ACM AM Turing Award 2017 and the 2022 NAE Charles Stark Draper Award for Engineering.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author, and do not represent those of ACM SIGARCH or its parent organization, ACM.