A few months ago, I received an email that asked me to serve on the ICWSM 2014 Program Committee. The role of PC member for this conference entails reading several submitted papers to the conference and then writing up reviews that would then be used by the senior members to come to a decision about accepting or rejecting the papers. I was really excited to be asked to be a part of this process because it was my first time seeing what goes on behind the scenes of how papers get vetted, reviewed, and deliberated on by the research community. Unfortunately I could find few resources for how to write a proper review and did not come across any tailored to this specific research community (or even the general research communities of HCI, CSCW, or social computing).

Nevertheless, with the help of my research advisor who provided several useful tips, I came away from the process with a better understanding of how to write a good review. I also found reading papers from the point-of-view of a reviewer eye-opening and informative for myself as a researcher and writer. As a result, I am writing down some of my thoughts and findings in the hopes that they may be useful for other beginning reviewers going through the same experience. Please note that this is not meant to be a comprehensive how-to for writing reviews – only an account of the thought-process, revelations, and observations of a 1st time reviewer!

At First…

My initial reaction to being invited to review was excitement, followed closely by anxiousness. I was worried that as a beginning researcher I would not find anything substantial to say about any of the submissions, written as they were by more experienced researchers than myself. Surely any of the meager suggestions or recommendations that I could potential prescribe would have been thought of and taken care of already?

Once I actually received my assigned papers, however, I was surprised to find that there was a good amount of variation in the papers in terms of quality of writing and methodology, and incidentally, I could in fact come up with comments for each of the papers (though whether these comments were of any value I couldn’t yet be certain – more on that later).

Observations

Some things that I noticed while reading papers from the lens of a reviewer for the first time:

Grammatical errors, imprecise or undefined wording, and poor sentence structure were glaringly noticeable, and I could sense my perception of the paper quickly souring upon encountering more than a few of them. Noticing this, I tried my best to suppress my misgivings and overlook such errors because it was apparent that some of the papers were written by non-native English speakers, and I thought it unfair for them to be penalized.

On a wider note, I realized how incredibly important it is to be a good writer and to think deeply about organization and presentation. As a reviewer, it was almost a joy to read the papers that conveyed ideas clearly, presented well-thought-out graphs and tables, and told a well-structured story that flowed coherently from section to section. On the flip side, it was frustrating to read papers that had sections that seemed misplaced or irrelevant, or graphs and tables where it wasn’t clear what they were trying to express.

There were two papers sections in particular that stood out to me as surprisingly important:

Related Work

I could sense alarm bells going off when I read a Related Work section and could think of or could quickly Google related research that was not mentioned. It wasn’t necessarily a specific paper, but when a particular research direction or theme that was very relevant to the work of the paper was not given mention, it made me wonder what else the writers missed. As someone who is not the pre-eminent expert on every topic of the papers I read nor in the position of replicating each paper’s work, I couldn’t be certain that everything the writers were asserting was true. So reading a comprehensive Related Work section somehow made me feel more confident in the writers and more inclined to believe their assertions.

Discussion

The Discussion section I found to also be extremely important. One might think that in order to write a strong research paper, it’s best to emphasize the strengths of the method or highlight only the positive findings and obfuscate, ignore, or minimize bad or inconclusive findings. In fact, I found that the opposite is true. I found myself much more willing to take a paper’s findings and conclusions at face value if the authors openly talked about drawbacks, caveats, and failures they had while doing their work. It was also interesting to see how authors tried to understand or conceptualize their findings, even if it wasn’t rigorous or a main part of their paper. I found it much more interesting as a reader to be presented with findings supported by a plausible explanation or put in context of a larger model rather than being given a slew of numbers at the end with no discussion at all.

Advice

After writing my initial reviews, I sent them off to my advisor for him to give a quick look-over to make sure I wasn’t putting my foot in my mouth. He gave me some great tips which I summarize below:

Conclusion

After turning in my reviews, I got a chance to see how other reviewers reviewed the same papers. Some reviews brought up points I had overlooked or had diverging opinions from me, which was  useful for me to see the other points of view that I didn’t consider as well as the range of perspectives. The best reviews in my opinion were the ones that demonstrated their expertise in that particular niche by backing up their points with citations and including recommendations and papers to look at along with their critiques. I could see how these reviews would be really useful for the authors to refine their work. Finally, it was a validating experience to see when other reviews brought up similar points to my own and when the meta-reviewers cited my review or mentioned points from my review as helpful for forming their final opinion. Here was concrete proof that many of my comments were actually valuable.

So in the end, my worst fears of being wildly off the mark in my reviews never materialized, and I finished the experience more confident in my research instincts and more understanding of the mentality of paper reviewers and thus how to frame, organize, and style my research writing. I think being part of the review process will help me become a better writer and ultimately, a better researcher and active member of the research community, and I encourage program chairs and organizers of conferences to invite PhD students and newer members of the community to participate in the review process.

—–

*  This post has been cross-posted from my personal blog.

*  While I’ve learned a great deal from this process, there are many, many veteran reviewers out there who have many more tips that they’ve cultivated or learned over time. Therefore, I’m keeping a list of comments to this piece and further advice I receive from others here:

- “…always talk about the weaknesses and limitations of your approach in a paper, but don’t have that be the last thing you talk about. I remember once ending a paper with the “Limitations” section right before a short conclusion, and my MSR boss telling me how that leaves a negative last impression.”

- My advisor also mentioned that a great (and hilarious) resource for reviewers is “How NOT to review a paper: The tools and techniques of the adversarial reviewer” by Graham Cormode, which also has citations to other useful guides on reviewing.

- This guide written for CHI 2005 is several years old but is a really thorough look into how to write a proper review, including useful examples of both suitable and unsuitable reviews.

There’s ongoing discussion of our conference review process.  Unsurprisingly it seems to spike around the time people get reviews of their own work.  A lot of the griping tends to assert that reviewers aren’t doing a good job.  Some argue for abandoning the pre-publication review process in favor of accepting everything and reviewing post-publication.  But I’m a big supporter of the model where experts help authors improve their work and help filter material so I know where to look for the good work.

Without abandoning it, I think there are some great opportunities to improve the review process by adding some transparency.  I’d like to address three problems with the current pre-review process:

1. Reviewers know that their work will never be seen, so are missing all the external motivators (praise for good work, scorn for bad) for doing a good job; only altruism remains.
2. Authors who feel unfairly reviewed have to shut up and take it
3. New reviewers have no real models of good work to emulate

To address these, I propose the following addition to the review process.   When reviewers submit a review, give them a box they can check if they are willing to have their review published, specifying anonymity or in their own name.  Similarly, when authors read reviews, give them a checkbox to indicate their willingness to have those reviews made public, as well as a check-box to indicate their willingness for their initial submission to be made public.  These permissions could be used in a number of interesting ways.

First, reviewers with permission of authors could begin to maintain/publish a corpus of the reviews they’ve written.  Even when I end up as Reviewer 2, I try to invest a lot of work in making a good review, and I’d love to be able to show that to other people.  A good reviewer could get credit in the community for their thoughtful reviews.  I suspect having a record like this would also make that reviewer a first choice for reviewing the most interesting submissions in our field (if we wanted to add incentive, we could give such high quality reviewers first dibs on bidding to review specific papers).

Second, authors who disagreed with their reviews could post responses to them.  It wouldn’t change the outcome, but it could certainly give the authors some satisfaction.  And the feedback, if valid, might help the reviewer become a better reviewer in the future.

Perhaps most important, these public reviews and submissions could become a valuable education resource.  New reviewers could be directed to a corpus of high quality reviews that they could use as models for their own.  This would work even better in combination with publication of the original and final submissions.  I’ve been asked to help teach a class on reviewing next semester; I would love to be able to take some original submissions, ask the students to review them, then show the students the real reviews and, finally, the paper that was ultimately published.  We should be able to see the kinds of problems that reviewers noticed, and the ways the authors chose to deal with those problems in revision.

These are just the direct benefits; there are also strong indirect ones.  Even if optional, that waiting check-box will be on reviewers’ minds as they review; they can take the easy way out, but knowing that there’s a community desire for openness might encourage them to work a little harder at that review in order to make it public.  It won’t fix the really awful reviewers but will lead those on the boundary to do a little more.  This might not just mean reading the paper more carefully, but also taking the time to make sure that the tone of the review is respectful and encouraging of the authors.  Conversely, knowing that the community would prefer to publish the original submission might encourage authors to deliver one of higher quality.  Again, those who plan to have the committee do the authors’ work for them, framing the problem, finding related work, and suggesting better experiments, aren’t going to check that box, but those on the borderline might decide to wait for the next conference if their paper isn’t ready yet.

Some years ago I went to a HOTNETs conference where each paper was preceded by a commentary from one of the PC members.  If reviews could be published, it could be quite interesting to include them in the proceedings with each paper, to serve just that commentary role.  People who signed their reviews could get some nice credit for insightful commentary, and we could even award prizes to the best ones.

An important consideration is that a reviewer permitting their review to be published isn’t sufficient—the paper author has to agree as well.  Otherwise, the review might leak information about the submission that the author isn’t willing to share.

Are there downsides?  We might worry about increasing conflicts of interest.  A junior member might be worried about being associated with a negative review of a powerful colleague.  But I think this can often be addressed by allowing anonymous publication of those reviews.  And there’s always the option to keep reviews private.

In summary, I think this is an approach that creates several opportunities without forcing any undesired change.  I know that I would often be quite eager to share my own reviews (they become a useful commentary on the state of the art), and comfortable sharing most of the reviews I’ve received and my preliminary submissions.  How about you?

This past month I finished reviewing 4 papers for CHI and 6 for the WWW conference.  For CHI, 3 of the 4 papers described small, simple applications intended to test some user interface idea.  For WWW, 4 of the 6 were machine learning/model fitting papers that tested algorithms on a particular non-sensitive data sets, one ran a small user study, and one was an experimental system like the CHI submissions.  For these 9 of 10 papers, there was absolutely no technical barrier to publishing the source code and/or data used for the paper: no privacy concerns and no complicated system architecture dependencies.  (The last paper was a theory paper that needed no replicating.)

There’s been a good amount of discussion about replication of prior experiments and its importance to good science.  There’s even a workshop on it at CHI 2014.  For replication, lack of the prior system or data is fatal.

Right now everyone talks about the importance of replication, but we aren’t doing enough to make it the norm.  So here’s a simple proposal.  In future conferences, let’s require that every submission (and every final published version) include a single, final paragraph labeled “Replication”.  In it, the authors should describe which of their experimental materials will be necessary to replicate their experiment, and either (i) where to find them or (ii) why they can’t make them available.

For an experimental system, the authors should give a link to the source code, or explain why their code cannot be shared.  “My code is ugly” is a terrible excuse.  “My code won’t run on your system” is only slightly better: even if it won’t run as is, it may be easier for future replicators to modify the previous code instead of writing their own system from scratch.  But there are plenty of legitimate explanations, including “I work at a company and the code is protected intellectual property”–we may not like that one, but we can’t expect a single author to change company policy.

For data, the authors should give a link to their data gathering instruments (e.g. surveys or software) and data sets, or explain why it’s impossible to share the data—privacy concerns being the obvious one.  Of course, such concerns can often be addressed by properly scrubbing the data, and authors should explain why they can’t do that.  The right preparation can help—for example, it could be easy and useful to get consent from survey respondents to publish their raw responses.

Ideally, this replication paragraph won’t just be boilerplate.  Rather, the program committee will take it into consideration at they make decisions about what to accept.  A paper that fails to justify withholding code or data should be rejected, or should be accepted only conditioned on the code or data being made available.  But even if the PC doesn’t want to be so strict in the beginning, I suspect that the need to explain, in public, why they aren’t sharing their code or data will be a great prod  encouraging more authors to make that small extra effort, to the benefit of science.

I’ll brag that our group is doing reasonably well on this front.  Nowadays, all the software projects we build go up on github, either in the general group repository or in a project-dedicated one like exhibit or nb.  Using github significantly improves our software development process, and make public sharing happen by default.  We’ve even accomplished a little bit on data: when we published our study in CHI of lightweight note-taking, we posted a site where any users of our tool could “contribute their notes to science”, scrubbing them and marking them for public use.  Scientists can download the resulting corpus of notes for their own research.  At 2500 notes, our published corpus is only a small fraction of our fill 200,000-note collection, but it’s a start.  If other groups were making similar efforts and it became the norm, we’d be trying harder to grow that corpus.

I don’t get them often, but by some statistical fluke I last summer I got invitations to and attended the faculty summits at Google, Microsoft, and Facebook within a period of two weeks.  I’ll spoke about the summits themselves in a different post.  In this one, I’ll talk about one existential question that arose while attending them: given the tremendous resources of these companies, does the research we CS faculty do actually make a difference?

Coincidentally, the day of the Google summit, one of my past collaborators announced that he was abandoning his tenured faculty position to take up a job at Google. His primary reason was “to make a difference”. While some of us faculty believe that the quest for knowledge is its own justification, a lot of faculty do share that motivation to make a difference with our research. Is that possible in CS at the present time?

The usual narrative talks about all the important companies that emerged from academia, like Google from Stanford and RSA from MIT.  But it’s hard to assess the counterfactual: if universities hadn’t launched these companies, would the same innovations have appeared at existing companies?

There are some plausible arguments that our research is superfluous. Google, Facebook, and Microsoft are full of brilliant researchers and engineers who are just as capable as we are of coming up with fundamental innovations. And they have far more resources than we do to actually pursue those innovations. They far outnumber our students (the Taulbee survey says about 1500 CS PhDs graduate per year, which suggest that we have around 7500 enrolled.  The companies employ tens of thousands of computer scientists including, I suspect, more PhDs than we have enrolled).  Their equipment dwarfs what any academic can access—scanning the entire web is straightforward at these companies thanks to their spidering engines, but really challenging for any academic. Their user base is huge, permitting massive A/B studies at scales that we academics cannot command, that can therefore measure incredibly small effects.

Historically, academics have also been tasked with carrying out “high risk/reward” research—stuff that was unlikely to pay off but would have a big impact if it did.  Companies avoided such research.  That was and still is meaningful when creating a new chip architecture or solving artificial intelligence, which requires massive investment before you see if your risk has paid off.  But nowadays, in many domains of computer science, it’s possible to throw together a prototype system in just a few weeks.  I think this changes the risk calculus, since a company can afford to cheaply take many high risks knowing that some of them will pay off.  Google’s 20% time is a great example of how this can work.

On the flip side, I believe there is important research that companies could do but choose not to, where we academics still have an opportunity to make a difference. Perhaps this was best represented by a slogan at Facebook: “done is better than perfect”. I agree with the sentiment, but it does reflect that fact that these fast-moving companies don’t have time to really dwell on a problem. They also often don’t take the time to tell anyone else about what they did. Academics have the luxury to take their time, and I think there is value to doing so. If we can work out a more “perfect” solution, and document it by publication, then the next company to have that problem will have access to a solution that is better than the first try, and immediately available to them without demanding investment of research or development effort. It may be depressing for faculty to consider that, instead of being at the cutting edge of research, we’re “backfilling”, bringing into public view ideas that companies already knew. But there’s clear social benefit to doing so—a way for us to make a difference. As a great example, Martin Hellman “didn’t care what the NSA knew” when he developed public key cryptography because he sensed there would be huge value to its being known publicly.

As a selfish example, my group has developed list.it, a lightweight note taking client. It’s not all that different from commercial tools like EverNote or . It’s got about 20,000 users, which is great for an academic project but nothing compared to the commercial tools. And those tools’ builders have presumably done some careful analysis of their users’ notes and usage patterns, and used that information to improve their tools. But they haven’t told anyone else what they found. We can make a contribution by publishing our analysis, and hopefully advance the public state of the art so that future note-taking tools can start from a better baseline.

Another example  comes from work we did on load-balancing web servers. One result was the founding of Akamai, a typical example of the emergence of a company from academia.  But first came our publication of a load-balancing technique called consistent hashing. Since then, the technique has found a place in peer-to-peer systems and more recently in . If we’d just been a company we might not have bothered to publish the work—we might even have kept it secret for competitive advantage—forcing those later applications to reinvent the idea or come up with a different one.

We do also have the opportunity to have cutting-edge impact, when we tackle research that companies don’t care to do, because the business payoff seems too far off or non-existent.   In this category I’m particularly interested in tools to improve civic discourse and tools that will let individuals keep their personal data to themselves.  I don’t see a way to convince any company that improved civic discourse will help them sell their products.  And keeping data private will likely have a huge negative impact on the advertising revenue on which these companies rely.  So academia needs to push these topics.

But if we do want to have this kind of forward impact we need to play to our strengths rather than the companies’.  We need to remember that they can build things faster and better than we can.  Perhaps this means that if there’s an engineering project that we know how to do, then we shouldn’t do it.  Instead we need to concentrate on areas where we have questions and don’t have answers.  But even more important, we need to recognize that often building the system simply isn’t an interesting contribution—it’s something any company could have done.  Rather, our contribution to society comes from studying that system in use, investing energy in drawing meaningful lessons from it, and publishing them.