[Read: What audience intelligence data tells us about the 2020 US presidential election<\/em><\/a>]<\/p>\n The paper focuses on the ICLR review process, though other conferences use similar methodologies. Each AI paper submitted to the conference goes through several steps before being accepted or rejected. The process involves, reviewers, area chairs, and program chairs.<\/p>\n Area chairs are experts that have experience in specific domains, such as computer vision and natural language processing. They do not review individual papers but control the process. They guide reviewers, moderate discussions, and make recommendations based on the feedback they get from authors and reviewers.<\/p>\n Reviewers are the people who work on individual papers. Each paper is assigned to several reviewers, who read it in full and verify the code and data that comes with the paper to make sure the findings are valid and reproducible. They correspond with the author and the area chairs to clarify questions that need to be answered, and finally, they give their final recommendations on whether a paper should be accepted or rejected.<\/p>\n Program chairs are senior scientists and experts that make high-level decisions, including the final decision on which papers get rejected or accepted. They can also intervene in the review process if needed.<\/p>\n Experts can apply to become reviewers and area chairs at ICLR, but most are invited by the organizers.<\/p>\n The entire process is registered on the OpenReview platform, where everyone can see the feedback given by reviewers, authors, area chairs, and program chairs. OpenReview also accepts public comments from people who are not involved in the review process.<\/p>\n The authors of the paper, who remain anonymous as of this writing, were keen on answering the question: Do conference reviews consistently identify high-quality work? Or has review degenerated into censorship?<\/p>\n To make an informed decision, they used public information registered in OpenReview, including titles, abstracts, authors lists, emails, scores, and reviews for more than 5,000 papers submitted to ICLR conferences from 2017 to 2020. The authors obtained additional information about withdrawn papers after communicating with the administrators of OpenReview.<\/p>\n The researchers used different methods to find additional information, such as the impact of papers and authors on other AI research work. They also analyzed the paper data to determine the topic of each paper.<\/p>\n The first question the researchers answered was whether there was any amount of randomness involved in the acceptance or rejection of AI research papers. This means that, for instance, if an accepted paper went through the review process again, would it be accepted again?<\/p>\n To perform this test, the researchers created a regression model from paper scores and the final decision made by the area chairs. Then they simulated the process by generating scores the fit the threshold and checked to see how many times the model reproduced the same decision on the paper. They ran the simulation separately for the papers submitted to each year, and the results were disappointing.<\/p>\n \u201cWe observe a downward trend in reproducibility, with scores decreasing from 75% in 2017, to 70% in 2018 and 2019, to 66% in 2020,\u201d the authors write.<\/p>\n Even when they increased the number of reviewers in their simulation, the gains were not significant. \u201cWhile reproducibility scores increase with more reviewers, gains are marginal; increasing the number of reviewers from 2 to 5 boosts reproducibility by just 3%,\u201d the authors write. \u201cAs more reviewers are added, the high level of disagreement among area chairs remains constant, while the standard error in mean scores falls slowly. The result is paltry gains in reproducibility.\u201d<\/p>\n To improve the process, the authors recommend to use a small number of reviewers and ad ad-hoc reviewers where first-round reviews are uninformative.<\/p>\n Papers accepted at major AI conferences should help advance future artificial intelligence research. In their work, the authors of the OpenReview paper analyzed the impact of both accepted and rejected AI research papers. \u201cWe measure impact using citation rate, calculated by dividing citation count by the number of days since the paper was first published online,\u201d they write.<\/p>\n In other words, impactful papers should receive more citations from other AI researchers. Naturally, papers accepted and presented at AI conferences gain more exposure and by extension more citations than rejected work. But when comparing accepted papers among themselves, the researchers found a very small correlation between citation and review scores.<\/p>\n \u201cOur dataset shows that reproducibility scores, correlations with impact, and reviewer agreement have all gone down over the years,\u201d the authors write.<\/p>\n The authors of the paper were also interested in knowing whether there was a preference for AI research done at prestigious academic institutions or tech companies. \u201cWe found that 85% of papers across all years (87% in 2020) had at least one academic author,\u201d the authors write, adding that this finding per se does not imply bias toward and can fairly be related to the quality of research done at these institutions.<\/p>\n To further investigate, they \u201ccontrolled for paper score,\u201d which means they verified whether area chairs showed any bias when deciding on papers that had similar scores.<\/p>\n \u201cWe found that, even after controlling for reviewer scores, being a top ten institution leads to a boost in the likelihood of getting accepted,\u201d the authors write. This bias remained even when the researcher only considered the affiliation of the last author of the AI papers (last authors are usually the senior person who oversees the research or heads the lab where the research takes place).<\/p>\n There was a significant preference for Carnegie Mellon, MIT, and Cornell universities, the authors found. Another interesting point that confirms this finding is that author identity visibility improves its review score. The authors of an AI paper submitted to OpenReview remain anonymous to reviewers while it is going through the review process. But at the same time, the researchers can publish their paper in other mediums, such as the arXiv preprint server. The researchers found that AI papers appearing on arXiv before the review process tended to do better, especially if the researchers were associated with CMU, MIT, or Cornell.<\/p>\n The authors of the OpenReview analysis also found that, all things equal, area chairs generally preferred submissions authored by highly reputed individuals.<\/p>\n Interestingly, their research did not find a significant bias toward large tech companies such as Google, Facebook, and Microsoft, which house reputable AI researchers. At first glance, this is a positive finding, because big tech already has a <\/span>vast influence over commercial AI<\/a> <\/span>and, by extension, on AI research.<\/p>\n But as <\/span>other authors have pointed out<\/a>, the same academic institutions that are very well represented at AI conferences serve as talent pools for big tech companies and receive much of their funding from those same organizations. So this just creates a feedback loop of a narrow group of people promoting each other\u2019s work and hiring each other at the expense of others.<\/p>\n Also concerning is the under-representation of women in AI conferences. The gender gap is a well-known problem in the AI community (and the tech community in general), and it carries over to the paper review process. \u201cWe observe a gender gap in the review process, with women first authors achieving a lower acceptance rate than men (23.5% vs 27.1%),\u201d the authors of the OpenReview analysis observe in their paper.<\/p>\n The researchers add that while in 2019, women made up 23.2 percent of all computer science PhD students, only 10.6 percent of publications at ICLR 2020 had a female first author (the first author is the lead contributor of the AI research). Meanwhile, women constitute 22.6 percent of computer science faculty in the U.S. but only 9.7 percent of senior authors at ICLR.<\/p>\n \u201cWe observe a gender gap in the review process, with women first authors achieving a lower acceptance rate than men (23.5% vs 27.1%),\u201d the authors of the analysis add.<\/p>\n \u201cWhen the number of papers with merit is greater than the number of papers that will be accepted, it is inevitable that decisions become highly subjective and that biases towards certain kinds of papers, topics, institution, and authors become endemic,\u201d the authors conclude in their paper.<\/p>\n As artificial intelligence continues to gain prominence in different areas of life, it is important that it becomes inclusive and represents all different demographics. AI conferences serve as hubs to draw attention to influential work that can have a great impact on future applications and research. When the review process favors one group of people, the effects can ripple to other areas that benefit from AI research, including health care, finance, hiring, and the justice system.<\/p>\n AI conference organizers should certainly reconsider the review process and take measures to ensure all researchers get a fair shot at having their work presented at prestigious venues. The work also has a message for the rest of us, the people who are following and covering AI research. We should all be aware that while AI conferences are a good criterion for where the community is headed, there is plenty of good research that never find their way into the spotlight. I\u2019ve had some great experience unearthing <\/span>some of these gems<\/a> <\/span>while perusing the <\/span>machine learning subreddit<\/a>, the AI and deep learning <\/span>Facebook group<\/a>, and creating my own list of AI researchers and computer scientists to follow on Twitter. I\u2019m proud to have been able to give some of these researchers the exposure their work deserves. I\u2019m sure others can do even better than me.<\/p>\n This article was originally published by Ben Dickson on <\/span><\/i>TechTalks<\/span><\/i><\/a>, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech and what we need to look out for. You can read the original article here<\/a>.<\/span><\/i><\/p>\n Published October 27, 2020 \u2014 10:39 UTC <\/p>\nThe AI conference review process<\/h2>\n
![\"ICLR](\"https:\/\/i1.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/10\/ICLR-review-process.png?resize=696%2C392&ssl=1\")
![\"openreview](\"https:\/\/i1.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/10\/openreview-rejected-paper.jpg?resize=696%2C365&ssl=1\")
Investigating the AI research paper review process<\/h2>\n
Is the review process reproducible?<\/h2>\n
![\"AI](\"https:\/\/i2.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/10\/AI-paper-adding-reviewers.png?resize=696%2C430&ssl=1\")
What is the impact of accepted AI research?<\/h2>\n
![\"AI](\"https:\/\/i2.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/10\/AI-paper-review-score-vs-impact.png?resize=696%2C501&ssl=1\")
Institution and reputation bias in the AI review process<\/h2>\n
![\"top-tier](\"https:\/\/i2.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/10\/top-tier-universities.jpg?resize=696%2C392&ssl=1\")
Why is this important?<\/h2>\n
![\"thinking](\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/01\/thinking-human-statue.jpg?resize=696%2C464&ssl=1\")
<\/figure>\n
\n