Mailing Lists: Why Are They Still Here, What’s Wrong With Them, and How Can We Fix Them?

Online group discussion has been around almost as long as the Internet, but it seems we still can’t create tools for it that satisfy everyone. Starting with the first mailing list in 1972 and evolving to modern social media tools, we see tensions between people’s desire to participate in productive discussions and the struggle to manage a deluge of incoming communication. Our group wants to build better tools to address this problem. As a first step we decided to learn from an existing tool. Mailing lists have survived almost unchanged from the earliest days of the Internet, and are still heavily used today. The fact that they’re still here suggests that they’re doing something right; the fact that so many new social media tools have tried to replace them suggest they’re doing something wrong.  We wanted to understand both sides. We interviewed a variety of mailing list users to understand what they loved and hated about their mailing lists. We discovered an interesting opportunity to combine some of the best features of email and social media to address the weaknesses of both.

To understand how different types of groups use mailing lists and why they continue to use them, we interviewed members of two active mailing list communities and surveyed 28 additional mailing lists of many different types. When asked whether they would be interested to switching to a different tool, such as Facebook Groups, or a discussion forum, or a subreddit, most people across the board were not interested in switching to a newer social media. In fact, only 12% indicated they were interested in switching to Facebook Groups, the most analogous tool to many people. When we asked why, people’s responses grouped into the following four themes:

  • Email is for work while social media is for play or procrastination. One interviewee was concerned about more cat pictures and other irrelevant or silly posts if his group moved to a Facebook Group and felt this was the wrong tone for the list. Other people felt that mailing list communication was actually somewhat in between work and play.
  • Email feels more private while social media feels more public. People mentioned images of people’s faces and hyperlinks to their profile as making the Facebook Groups interface feel more public. However, we were concerned to find out that most people surveyed and interviewed did not realize their mailing list archives were public. Nor could they properly estimate how many people read their emails. In most cases, people guessed an entire order of magnitude lower than the true subscription count.
  • There is a greater confidence that email will be seen. Not only do more people use email instead of Facebook, people also had a sense that email would be seen, while Facebook algorithms might make it uncertain who receives what.
  • Email management is more customizable. People enjoyed being able to set up their own filters and customize their notifications and experience of their mailing list.

Given all of these reasons for preferring mailing lists, have all of the social moderation features and controls in newer social media been created for naught? It seems the answer to this is also no. In our research, we found many tensions within mailing list communities, specifically issues arising from people within the same mailing list expressing very different opinions and perceptions about the list. The following three tensions stood out the greatest:

  • Tensions over type and quantity of content on the list. While some users enjoyed intellectual discussions on the list, others hated them. Same for just about any other category of content, such as humor, job listings, rental and item sales, etc. People even disagreed about the correct etiquette for reply-to-the-list versus reply-to-the-sender.
  • Tensions over desire for interaction versus hesitation to post. Most users expressed a desire for more discussion on their mailing list, yet the majority of these folks have never participated in a discussion themselves. When asked about the reasons people were deterred from posting, they mentioned concerns such as the fear of spamming others, fear of looking stupid, fear of offending, and fear of starting a heated debate.
  • Tensions over push versus pull email access method. Most users either received all their mailing list email in their main inbox (push) or filtered all their mailing list emails to a separate folder (pull). We found very different attitudes from people with these two different strategies. For instance, push-users were much more worried about missing email, were more likely to miss email, and were more hesitant to post out of fear of spamming. On the other side, pull-users read email when they felt like it and not when it arrived, were more likely to miss emails, and had a more relaxed attitude towards sending emails.

Some of these tensions have been mitigated in newer social media systems thanks to social moderation and other newer features. So what can we do given what we’ve learned? One thing we can do is to improve new social media systems by incorporating more of what people like about mailing list systems. Another thing we can do is to improve mailing lists by incorporating some features taken from social media. Some things that we consider are introducing slow propagation through the list using likes, allowing friends to moderate posts before they get sent farther, allowing topic tags and following of different topics and threads, and more. We emphasize improving mailing lists because it’s something that anyone can work on (you don’t have to work at Facebook!), it’s relatively easy to build and test since users continue to use their existing mail clients, and it’s really about time that mailing lists had some innovation.

In that vein, we’re actively working on a new mailing list system. It’s still in a very preliminary stage but you can check it out and join or even start your own mailing list group at You can also read our research paper published at CHI 2015 or look at slides and notes from the talk given at the CHI 2015 conference.

This work was conducted with Mark Ackerman of University of Michigan and David Karger at MIT CSAIL.

Thoughts and Reflections from a 1st Time Reviewer

A few months ago, I received an email that asked me to serve on the ICWSM 2014 Program Committee. The role of PC member for this conference entails reading several submitted papers to the conference and then writing up reviews that would then be used by the senior members to come to a decision about accepting or rejecting the papers. I was really excited to be asked to be a part of this process because it was my first time seeing what goes on behind the scenes of how papers get vetted, reviewed, and deliberated on by the research community. Unfortunately I could find few resources for how to write a proper review and did not come across any tailored to this specific research community (or even the general research communities of HCI, CSCW, or social computing).

Nevertheless, with the help of my research advisor who provided several useful tips, I came away from the process with a better understanding of how to write a good review. I also found reading papers from the point-of-view of a reviewer eye-opening and informative for myself as a researcher and writer. As a result, I am writing down some of my thoughts and findings in the hopes that they may be useful for other beginning reviewers going through the same experience. Please note that this is not meant to be a comprehensive how-to for writing reviews – only an account of the thought-process, revelations, and observations of a 1st time reviewer!


At First…

My initial reaction to being invited to review was excitement, followed closely by anxiousness. I was worried that as a beginning researcher I would not find anything substantial to say about any of the submissions, written as they were by more experienced researchers than myself. Surely any of the meager suggestions or recommendations that I could potential prescribe would have been thought of and taken care of already?

Once I actually received my assigned papers, however, I was surprised to find that there was a good amount of variation in the papers in terms of quality of writing and methodology, and incidentally, I could in fact come up with comments for each of the papers (though whether these comments were of any value I couldn’t yet be certain – more on that later).



Some things that I noticed while reading papers from the lens of a reviewer for the first time:

Grammatical errors, imprecise or undefined wording, and poor sentence structure were glaringly noticeable, and I could sense my perception of the paper quickly souring upon encountering more than a few of them. Noticing this, I tried my best to suppress my misgivings and overlook such errors because it was apparent that some of the papers were written by non-native English speakers, and I thought it unfair for them to be penalized.

On a wider note, I realized how incredibly important it is to be a good writer and to think deeply about organization and presentation. As a reviewer, it was almost a joy to read the papers that conveyed ideas clearly, presented well-thought-out graphs and tables, and told a well-structured story that flowed coherently from section to section. On the flip side, it was frustrating to read papers that had sections that seemed misplaced or irrelevant, or graphs and tables where it wasn’t clear what they were trying to express.

There were two papers sections in particular that stood out to me as surprisingly important:

Related Work

I could sense alarm bells going off when I read a Related Work section and could think of or could quickly Google related research that was not mentioned. It wasn’t necessarily a specific paper, but when a particular research direction or theme that was very relevant to the work of the paper was not given mention, it made me wonder what else the writers missed. As someone who is not the pre-eminent expert on every topic of the papers I read nor in the position of replicating each paper’s work, I couldn’t be certain that everything the writers were asserting was true. So reading a comprehensive Related Work section somehow made me feel more confident in the writers and more inclined to believe their assertions.


The Discussion section I found to also be extremely important. One might think that in order to write a strong research paper, it’s best to emphasize the strengths of the method or highlight only the positive findings and obfuscate, ignore, or minimize bad or inconclusive findings. In fact, I found that the opposite is true. I found myself much more willing to take a paper’s findings and conclusions at face value if the authors openly talked about drawbacks, caveats, and failures they had while doing their work. It was also interesting to see how authors tried to understand or conceptualize their findings, even if it wasn’t rigorous or a main part of their paper. I found it much more interesting as a reader to be presented with findings supported by a plausible explanation or put in context of a larger model rather than being given a slew of numbers at the end with no discussion at all.



After writing my initial reviews, I sent them off to my advisor for him to give a quick look-over to make sure I wasn’t putting my foot in my mouth. He gave me some great tips which I summarize below:

  • Start out reviews with a short summary of the paper. It’s useful for reviewers to remember what paper they’re reading about and helps authors know if they’re getting their main ideas across properly. Don’t forget to write within the summary what aspect of the paper is novel. This forces the reviewer to focus on the novelty of the work, which may affect the rating given to the paper. I recall cases where I actually struggled to put into words the novelty of a paper and as a result, realized I should bumped my ratings down.
  • Be careful of subjective statements that start with phrases such as “I would have liked…” because it’s unclear to the meta-reviewer and authors whether this is an objective problem with the paper or a subjective preference. The former is a strike against the paper while the latter may be interesting to the authors but shouldn’t count against the paper in terms of acceptance into the conference. When going through my reviews, I noticed that I had included several of these phrases – I guess because I was a first-time reviewer and afraid of coming off too strong or assured in my review, especially if I turned out to be wrong.



After turning in my reviews, I got a chance to see how other reviewers reviewed the same papers. Some reviews brought up points I had overlooked or had diverging opinions from me, which was  useful for me to see the other points of view that I didn’t consider as well as the range of perspectives. The best reviews in my opinion were the ones that demonstrated their expertise in that particular niche by backing up their points with citations and including recommendations and papers to look at along with their critiques. I could see how these reviews would be really useful for the authors to refine their work. Finally, it was a validating experience to see when other reviews brought up similar points to my own and when the meta-reviewers cited my review or mentioned points from my review as helpful for forming their final opinion. Here was concrete proof that many of my comments were actually valuable.

So in the end, my worst fears of being wildly off the mark in my reviews never materialized, and I finished the experience more confident in my research instincts and more understanding of the mentality of paper reviewers and thus how to frame, organize, and style my research writing. I think being part of the review process will help me become a better writer and ultimately, a better researcher and active member of the research community, and I encourage program chairs and organizers of conferences to invite PhD students and newer members of the community to participate in the review process.


*  This post has been cross-posted from my personal blog.

*  While I’ve learned a great deal from this process, there are many, many veteran reviewers out there who have many more tips that they’ve cultivated or learned over time. Therefore, I’m keeping a list of comments to this piece and further advice I receive from others here:

- “…always talk about the weaknesses and limitations of your approach in a paper, but don’t have that be the last thing you talk about. I remember once ending a paper with the “Limitations” section right before a short conclusion, and my MSR boss telling me how that leaves a negative last impression.”

- My advisor also mentioned that a great (and hilarious) resource for reviewers is “How NOT to review a paper: The tools and techniques of the adversarial reviewer” by Graham Cormode, which also has citations to other useful guides on reviewing.

- This guide written for CHI 2005 is several years old but is a really thorough look into how to write a proper review, including useful examples of both suitable and unsuitable reviews.

Allocating CHI Reviewers, 2014 Edition

As is now an annual tradition, I’ve performed my analysis of the allocation of reviewers for this year’s CHI conference.  The data from the CHI review process suggests that we can reduce the number of reviewers per paper (and thus reduce the workload on our community) without significantly affecting the outcome.  At present, every paper gets three reviews.  As in the past two years, I ask what would happen if we initially used only two reviewers per paper, then sent for a third review only when necessary (and what the right “necessary” criterion should be).

I used exactly the same methodology as in the past two years, which you can read about here and here.  This year I had accurate data thanks to PC chair Tovi Grossman, who took a snapshot of reviewing scores just before beginning the “discussion” phase of the review process (when reviewers can see each others’ comments and may change their review scores).  A total of 1542 papers were reviews, and ultimately 454 were accepted.

The results are extremely consistent with the past two years.  In particular, if after getting two reviews we immediately reject any papers with both review scores below 3 (the neutral score) then in expectation we skip 641 unnecessary reviews while rejecting only 10 of the 454 papers that “should have been” accepted.   This represents a savings of 641/5809=11% of the total amount of reviews, with a “noise” loss of only 10/454=2% of the accepted papers.  I believe this 2% is dwarfed by other errors (review inaccuracies), happens primarily to papers on the borderline of acceptance, and is worth it if we can save workload for the entire community.  To make the point stronger, I have confirmed that 2/3 of the skipped reviews were below 3—not only are we saving reviewer work, we are primarily saving reviewers work on papers they don’t like!

The Details

To assess my proposal, I use exactly the same methodology as last year.  I analyze the process of choosing two of each paper’s reviews at random as the “first” reviews, determining whether those two reviews would trigger a third review, and then bucket the result according to whether the paper was ultimately accepted or rejected.  For example, while analyzing a rule such as “don’t bother with a third review if the initial two scores are both less than three”, a paper that received scores of 1,2, and 4 would be seen as skipping a third review if the pair (1,2) were chosen (probability 1/3) but receiving a third review if pair (1,4) or (2,4) were chosen.  Skipping the third review was beneficial if the paper was ultimately rejected as it saved a “pointless” review, but damaging if it was ultimately accepted as it would lead to an “accidental” rejection of what should have been accepted.

Slightly complicating things was the fact that 86 papers received 4 or even 5 reviews (presumably a race condition where multiple reviewers were recruited simultaneously), 300 received only 2 (presumably the third arrived after the deadline), and 33 had only 1 review.  I properly adjusted probabilities for the five, four, and two-review cases, and dropped the 1-review cases as they could not be sensibly analyzed.  33 is too small a number to materially affect the analysis.

Below, the table of outcomes.  For each pair of scores I report the number of papers (in expectation) that would initially get that pair of scores (Score1 and Score2) and were ultimately accepted (Accepts) and Rejected (Rejects).   Now suppose all papers with initial score-pair at or below a given row of the table were rejected without a third review.  Then all the ultimately-rejected papers below this row would be saved from getting a third review, while all ultimately accepted papers below this row would be inappropriately rejected for lack of their third review.  For a given row, the “Saved Reviews” column counts the number of reviews that would be skipped using that row as a threshold, while the “False Rejects” column counts the number of papers that get rejected by that threshold but got accepted under the current 3-reviews scheme.

The table is sorted by maximum score (of the pair) which makes it easy to see what would happen if you rejected papers that didn’t achieve a given maximum score.  So for example, rejecting any paper that failed to get any score of 4 or higher would save 1152 reviews (almost 1/3 of the total) but lead to 113 mistaken rejections (almost 1/3 of accepted papers).

Score1 Score2 Accepts Rejects Ratio Cum Acc Cum Rej False Rej Saved Reviews
5 5 1.67 0.00 NaN 0.00 0.00 454.00 1542.00
5 4.5 9.33 1.67 5.60 1.67 1.67 452.33 1540.33
5 4 13.50 3.50 3.86 11.00 5.17 443.00 1536.83
5 3.5 12.17 5.67 2.15 24.50 10.83 429.50 1531.17
5 3 8.00 3.67 2.18 36.67 14.50 417.33 1527.50
5 2.5 4.00 10.17 0.39 44.67 24.67 409.33 1517.33
5 2 3.50 8.50 0.41 48.67 33.17 405.33 1508.83
5 1.5 4.00 2.00 2.00 52.17 35.17 401.83 1506.83
5 1 0.33 1.33 0.25 56.17 36.50 397.83 1505.50
4.5 4.5 10.50 1.00 10.50 56.50 37.50 397.50 1504.50
4.5 4 33.33 10.33 3.23 67.00 47.83 387.00 1494.17
4.5 3.5 20.17 10.83 1.86 100.33 58.67 353.67 1483.33
4.5 3 12.00 6.17 1.95 120.50 64.83 333.50 1477.17
4.5 2.5 14.00 15.83 0.88 132.50 80.67 321.50 1461.33
4.5 2 9.83 23.17 0.42 146.50 103.83 307.50 1438.17
4.5 1.5 3.33 13.17 0.25 156.33 117.00 297.67 1425.00
4.5 1 2.00 8.17 0.24 159.67 125.17 294.33 1416.83
4 4 36.83 11.67 3.16 161.67 136.83 292.33 1405.17
4 3.5 52.17 29.00 1.80 198.50 165.83 255.50 1376.17
4 3 23.67 31.83 0.74 250.67 197.67 203.33 1344.33
4 2.5 37.00 51.83 0.71 274.33 249.50 179.67 1292.50
4 2 20.50 67.00 0.31 311.33 316.50 142.67 1225.50
4 1.5 6.33 29.67 0.21 331.83 346.17 122.17 1195.83
4 1 2.00 23.50 0.09 338.17 369.67 115.83 1172.33
3.5 3.5 22.67 20.27 1.12 340.17 389.93 113.83 1152.07
3.5 3 20.33 44.33 0.46 362.83 434.27 91.17 1107.73
3.5 2.5 18.83 65.57 0.29 383.17 499.83 70.83 1042.17
3.5 2 12.00 93.83 0.13 402.00 593.67 52.00 948.33
3.5 1.5 5.67 38.93 0.15 414.00 632.60 40.00 909.40
3.5 1 2.67 23.83 0.11 419.67 656.43 34.33 885.57
3 3 4.50 15.33 0.29 422.33 671.77 31.67 870.23
3 2.5 10.00 55.67 0.18 426.83 727.43 27.17 814.57
3 2 6.00 71.00 0.08 436.83 798.43 17.17 743.57
3 1.5 1.00 33.83 0.03 442.83 832.27 11.17 709.73
3 1 0.00 20.00 0.00 443.83 852.27 10.17 689.73
2.5 2.5 2.00 47.93 0.04 443.83 900.20 10.17 641.80
2.5 2 3.50 115.40 0.03 445.83 1015.60 8.17 526.40
2.5 1.5 1.33 59.37 0.02 449.33 1074.97 4.67 467.03
2.5 1 0.67 41.00 0.02 450.67 1115.97 3.33 426.03
2 2 1.67 89.27 0.02 451.33 1205.23 2.67 336.77
2 1.5 0.33 110.93 0.00 453.00 1316.17 1.00 225.83
2 1 0.33 90.50 0.00 453.33 1406.67 0.67 135.33
2 0.5 0.00 0.33 0.00 453.67 1407.00 0.33 135.00
1.5 1.5 0.00 43.77 0.00 453.67 1450.77 0.33 91.23
1.5 1 0.33 57.93 0.01 453.67 1508.70 0.33 33.30
1 1 0.00 32.97 0.00 454.00 1541.67 0.00 0.33
1 0.5 0.00 0.33 0.00 454.00 1542.00 0.00 0.00


Shining some Sunlight on Conference Reviews

There’s ongoing discussion of our conference review process.  Unsurprisingly it seems to spike around the time people get reviews of their own work.  A lot of the griping tends to assert that reviewers aren’t doing a good job.  Some argue for abandoning the pre-publication review process in favor of accepting everything and reviewing post-publication.  But I’m a big supporter of the model where experts help authors improve their work and help filter material so I know where to look for the good work.

Without abandoning it, I think there are some great opportunities to improve the review process by adding some transparency.  I’d like to address three problems with the current pre-review process:

  1. Reviewers know that their work will never be seen, so are missing all the external motivators (praise for good work, scorn for bad) for doing a good job; only altruism remains.
  2. Authors who feel unfairly reviewed have to shut up and take it
  3. New reviewers have no real models of good work to emulate

To address these, I propose the following addition to the review process.   When reviewers submit a review, give them a box they can check if they are willing to have their review published, specifying anonymity or in their own name.  Similarly, when authors read reviews, give them a checkbox to indicate their willingness to have those reviews made public, as well as a check-box to indicate their willingness for their initial submission to be made public.  These permissions could be used in a number of interesting ways.

First, reviewers with permission of authors could begin to maintain/publish a corpus of the reviews they’ve written.  Even when I end up as Reviewer 2, I try to invest a lot of work in making a good review, and I’d love to be able to show that to other people.  A good reviewer could get credit in the community for their thoughtful reviews.  I suspect having a record like this would also make that reviewer a first choice for reviewing the most interesting submissions in our field (if we wanted to add incentive, we could give such high quality reviewers first dibs on bidding to review specific papers).

Second, authors who disagreed with their reviews could post responses to them.  It wouldn’t change the outcome, but it could certainly give the authors some satisfaction.  And the feedback, if valid, might help the reviewer become a better reviewer in the future.

Perhaps most important, these public reviews and submissions could become a valuable education resource.  New reviewers could be directed to a corpus of high quality reviews that they could use as models for their own.  This would work even better in combination with publication of the original and final submissions.  I’ve been asked to help teach a class on reviewing next semester; I would love to be able to take some original submissions, ask the students to review them, then show the students the real reviews and, finally, the paper that was ultimately published.  We should be able to see the kinds of problems that reviewers noticed, and the ways the authors chose to deal with those problems in revision.

These are just the direct benefits; there are also strong indirect ones.  Even if optional, that waiting check-box will be on reviewers’ minds as they review; they can take the easy way out, but knowing that there’s a community desire for openness might encourage them to work a little harder at that review in order to make it public.  It won’t fix the really awful reviewers but will lead those on the boundary to do a little more.  This might not just mean reading the paper more carefully, but also taking the time to make sure that the tone of the review is respectful and encouraging of the authors.  Conversely, knowing that the community would prefer to publish the original submission might encourage authors to deliver one of higher quality.  Again, those who plan to have the committee do the authors’ work for them, framing the problem, finding related work, and suggesting better experiments, aren’t going to check that box, but those on the borderline might decide to wait for the next conference if their paper isn’t ready yet.

Some years ago I went to a HOTNETs conference where each paper was preceded by a commentary from one of the PC members.  If reviews could be published, it could be quite interesting to include them in the proceedings with each paper, to serve just that commentary role.  People who signed their reviews could get some nice credit for insightful commentary, and we could even award prizes to the best ones.

An important consideration is that a reviewer permitting their review to be published isn’t sufficient—the paper author has to agree as well.  Otherwise, the review might leak information about the submission that the author isn’t willing to share.

Are there downsides?  We might worry about increasing conflicts of interest.  A junior member might be worried about being associated with a negative review of a powerful colleague.  But I think this can often be addressed by allowing anonymous publication of those reviews.  And there’s always the option to keep reviews private.

In summary, I think this is an approach that creates several opportunities without forcing any undesired change.  I know that I would often be quite eager to share my own reviews (they become a useful commentary on the state of the art), and comfortable sharing most of the reviews I’ve received and my preliminary submissions.  How about you?




Demanding a Replicability Paragraph in Conference Submissions

This past month I finished reviewing 4 papers for CHI and 6 for the WWW conference.  For CHI, 3 of the 4 papers described small, simple applications intended to test some user interface idea.  For WWW, 4 of the 6 were machine learning/model fitting papers that tested algorithms on a particular non-sensitive data sets, one ran a small user study, and one was an experimental system like the CHI submissions.  For these 9 of 10 papers, there was absolutely no technical barrier to publishing the source code and/or data used for the paper: no privacy concerns and no complicated system architecture dependencies.  (The last paper was a theory paper that needed no replicating.)

There’s been a good amount of discussion about replication of prior experiments and its importance to good science.  There’s even a workshop on it at CHI 2014.  For replication, lack of the prior system or data is fatal.

Right now everyone talks about the importance of replication, but we aren’t doing enough to make it the norm.  So here’s a simple proposal.  In future conferences, let’s require that every submission (and every final published version) include a single, final paragraph labeled “Replication”.  In it, the authors should describe which of their experimental materials will be necessary to replicate their experiment, and either (i) where to find them or (ii) why they can’t make them available.

For an experimental system, the authors should give a link to the source code, or explain why their code cannot be shared.  “My code is ugly” is a terrible excuse.  “My code won’t run on your system” is only slightly better: even if it won’t run as is, it may be easier for future replicators to modify the previous code instead of writing their own system from scratch.  But there are plenty of legitimate explanations, including “I work at a company and the code is protected intellectual property”–we may not like that one, but we can’t expect a single author to change company policy.

For data, the authors should give a link to their data gathering instruments (e.g. surveys or software) and data sets, or explain why it’s impossible to share the data—privacy concerns being the obvious one.  Of course, such concerns can often be addressed by properly scrubbing the data, and authors should explain why they can’t do that.  The right preparation can help—for example, it could be easy and useful to get consent from survey respondents to publish their raw responses.

Ideally, this replication paragraph won’t just be boilerplate.  Rather, the program committee will take it into consideration at they make decisions about what to accept.  A paper that fails to justify withholding code or data should be rejected, or should be accepted only conditioned on the code or data being made available.  But even if the PC doesn’t want to be so strict in the beginning, I suspect that the need to explain, in public, why they aren’t sharing their code or data will be a great prod  encouraging more authors to make that small extra effort, to the benefit of science.

I’ll brag that our group is doing reasonably well on this front.  Nowadays, all the software projects we build go up on github, either in the general group repository or in a project-dedicated one like exhibit or nb.  Using github significantly improves our software development process, and make public sharing happen by default.  We’ve even accomplished a little bit on data: when we published our study in CHI of lightweight note-taking, we posted a site where any users of our tool could “contribute their notes to science”, scrubbing them and marking them for public use.  Scientists can download the resulting corpus of notes for their own research.  At 2500 notes, our published corpus is only a small fraction of our fill 200,000-note collection, but it’s a start.  If other groups were making similar efforts and it became the norm, we’d be trying harder to grow that corpus.

PhD: Part 1

Sieuferd Small Screenshot
Fig. 1: An early version of SIEUFERD, the Schema-Independent End-User Front-End for Relational Databases.

It’s my sixth year in graduate school; my committee has been formed, my PhD thesis proposal has been submitted, and I am coding along on SIEUFERD, the research system from which I hope to squeeze my remaining research contributions. The project and its moniker, the Schema-Independent End-User Front-End for Relational Databases, began during two semesters of undergraduate Independent Work at Princeton with Brian Kernighan and David Walker, and I’m still using the same old demo database that I assembled back then. In 2009, Paul Grogan, Yod Watanaprakornkul, and I implemented what eventually became my Master’s thesis project, a spreadsheet-based database UI known as Related Worksheets (published in CHI 2011). For many slide decks since, Related Worksheets served as a mockup of things I hoped to achieve in my longer-term PhD project.

SIEUFERD’s goal is to provide a general-purpose user interface to relational databases. It takes its inspiration from two decades’ worth of graphical database applications that were developed, at great expense, to serve niche markets such as seafood trading, music school administration, and refugee camp management, and attempts to generalize their standard UI idioms into a single, universal application that provides most of their features in a schema-independent manner. The proof that this can be done lies in existing general-purpose “killer apps” such as Excel and ; the challenge lies in achieving the same for the generalized relational database use case (think CRUD over data modeled by entity-relationship diagrams). Ted Benson and I presented a vision paper about this at CIDR 2011.

Well, my advisors (David Karger and, also for my master’s thesis, Rob Miller) always told me that “a Master’s Thesis is a double-spaced conference paper, and a PhD is three more of those”. By that measure, I have just completed the first installment of my PhD trilogy, having returned from Atlanta to present the first SIEUFERD-related paper at InfoVis 2013. Unexcitingly titled, this paper deals with the important subproblem of taking data retrieved from a relational database and displaying it on the screen in a way that resembles the standard UIs of traditional tailor-made database applications, without actually requiring anyone to design that UI. And once we are dealing with a general-purpose visualization rather than a hard-coded dialog box, we can slowly but surely add features such as spreadsheet-style cursors for keyboard and mouse navigation, frozen table headers, printing and pagination, and eventually copy/paste, find/replace, undo/redo, sorting and filtering, and so on. While we take these features for granted in a general-purpose application like Microsoft Excel, many are rare to see in niche applications because of the prohibitive cost of implementing them for a small target user population.

The next “PhD installment” will be about the UI that actually makes a user able to connect to an arbitrary relational database and start generating queries of the kind for which the results were visualized in the previous subproject. This should, excitingly, be the first point at which the research system becomes useful for actual real-world tasks (in the “report generation” category), promoting it from its current “toy” status. Back to work!

Can Academics Make a Difference in CS Research?

I don’t get them often, but by some statistical fluke I last summer I got invitations to and attended the faculty summits at Google, Microsoft, and Facebook within a period of two weeks.  I’ll spoke about the summits themselves in a different post.  In this one, I’ll talk about one existential question that arose while attending them: given the tremendous resources of these companies, does the research we CS faculty do actually make a difference?

Coincidentally, the day of the Google summit, one of my past collaborators announced that he was abandoning his tenured faculty position to take up a job at Google. His primary reason was “to make a difference”. While some of us faculty believe that the quest for knowledge is its own justification, a lot of faculty do share that motivation to make a difference with our research. Is that possible in CS at the present time?

The usual narrative talks about all the important companies that emerged from academia, like Google from Stanford and RSA from MIT.  But it’s hard to assess the counterfactual: if universities hadn’t launched these companies, would the same innovations have appeared at existing companies?

There are some plausible arguments that our research is superfluous. Google, Facebook, and Microsoft are full of brilliant researchers and engineers who are just as capable as we are of coming up with fundamental innovations. And they have far more resources than we do to actually pursue those innovations. They far outnumber our students (the Taulbee survey says about 1500 CS PhDs graduate per year, which suggest that we have around 7500 enrolled.  The companies employ tens of thousands of computer scientists including, I suspect, more PhDs than we have enrolled).  Their equipment dwarfs what any academic can access—scanning the entire web is straightforward at these companies thanks to their spidering engines, but really challenging for any academic. Their user base is huge, permitting massive A/B studies at scales that we academics cannot command, that can therefore measure incredibly small effects.

Historically, academics have also been tasked with carrying out “high risk/reward” research—stuff that was unlikely to pay off but would have a big impact if it did.  Companies avoided such research.  That was and still is meaningful when creating a new chip architecture or solving artificial intelligence, which requires massive investment before you see if your risk has paid off.  But nowadays, in many domains of computer science, it’s possible to throw together a prototype system in just a few weeks.  I think this changes the risk calculus, since a company can afford to cheaply take many high risks knowing that some of them will pay off.  Google’s 20% time is a great example of how this can work.

On the flip side, I believe there is important research that companies could do but choose not to, where we academics still have an opportunity to make a difference. Perhaps this was best represented by a slogan at Facebook: “done is better than perfect”. I agree with the sentiment, but it does reflect that fact that these fast-moving companies don’t have time to really dwell on a problem. They also often don’t take the time to tell anyone else about what they did. Academics have the luxury to take their time, and I think there is value to doing so. If we can work out a more “perfect” solution, and document it by publication, then the next company to have that problem will have access to a solution that is better than the first try, and immediately available to them without demanding investment of research or development effort. It may be depressing for faculty to consider that, instead of being at the cutting edge of research, we’re “backfilling”, bringing into public view ideas that companies already knew. But there’s clear social benefit to doing so—a way for us to make a difference. As a great example, Martin Hellman “didn’t care what the NSA knew” when he developed public key cryptography because he sensed there would be huge value to its being known publicly.

As a selfish example, my group has developed, a lightweight note taking client. It’s not all that different from commercial tools like EverNote or . It’s got about 20,000 users, which is great for an academic project but nothing compared to the commercial tools. And those tools’ builders have presumably done some careful analysis of their users’ notes and usage patterns, and used that information to improve their tools. But they haven’t told anyone else what they found. We can make a contribution by publishing our analysis, and hopefully advance the public state of the art so that future note-taking tools can start from a better baseline.

Another example  comes from work we did on load-balancing web servers. One result was the founding of Akamai, a typical example of the emergence of a company from academia.  But first came our publication of a load-balancing technique called consistent hashing. Since then, the technique has found a place in peer-to-peer systems and more recently in . If we’d just been a company we might not have bothered to publish the work—we might even have kept it secret for competitive advantage—forcing those later applications to reinvent the idea or come up with a different one.

We do also have the opportunity to have cutting-edge impact, when we tackle research that companies don’t care to do, because the business payoff seems too far off or non-existent.   In this category I’m particularly interested in tools to improve civic discourse and tools that will let individuals keep their personal data to themselves.  I don’t see a way to convince any company that improved civic discourse will help them sell their products.  And keeping data private will likely have a huge negative impact on the advertising revenue on which these companies rely.  So academia needs to push these topics.

But if we do want to have this kind of forward impact we need to play to our strengths rather than the companies’.  We need to remember that they can build things faster and better than we can.  Perhaps this means that if there’s an engineering project that we know how to do, then we shouldn’t do it.  Instead we need to concentrate on areas where we have questions and don’t have answers.  But even more important, we need to recognize that often building the system simply isn’t an interesting contribution—it’s something any company could have done.  Rather, our contribution to society comes from studying that system in use, investing energy in drawing meaningful lessons from it, and publishing them.

Semantic Web for End Users: Keynote from ESWC 2013 now Online

A final addendum to my series of three posts on the Semantic Web and End Users.  The talk that I summarized in those posts is now online here, synchronized with the slides I presented.  I think I did a pretty good job getting through 217 slides in 75 minutes, and there’s some interesting Q&A at the end.  I remain very optimistic about the potential for the Semantic Web to improve the end user experience, and look forward to exploring this issue further with you all.

soft viagra

Keynote at ESWC Part 3: What’s Wrong with Semantic Web Research, and Some Ideas to Fix it

I’ve just returned from the European Semantic Web Conference, where I gave a keynote talk on “The Semantic Web for End Users”.   My talk addressed the problem that has interested me for eighteen years: making it easier for end users to manage their information.  The thesis was that

  • The current state of tools for end users to capture, manage, and communicate their information is terrible (last week), and
  • The Semantic Web presents a key part of the answer to building better tools (also last week), but
  • Not enough work is being directed toward this problem by the community (today’s post).

Since I had a lot to say (217 slides) I’ve broken the summary into three separate posts aligned with these three bullets; today’s is the last.


Our Story So Far

In my first post, I argued (using the work of Voida et al.) that our traditional application, designed around fix schemas, fail users when those user wish to use their own schemas or connect information from different schemas.  In the second, I argued that the Semantic Web suggest an exciting alternative, of applications that can be used with any schema, permitting end users to adapt their schemas as they see fit.  I gave four examples of such Semantic Web applications.


Papers in Our Semantic Web Conferences

Unfortunately, research on these end-user applications is almost completely absent from our Semantic Web conferences.

Preparing for this keynote, I looked over the program of ESWC 2013.  Of the 36 main-track papers, the vast majority were devoted to the generic underlying technologies of the semantic web—ontology matching (9), entity linking (6), information extraction (6), rdf querying (6), inference (3), and specific rdf ontologies (3).  Amidst all this infrastructure, there’s only one I’d label an end-user application: Kaljurand and Kuhn’s paper “Multilingual semantic wiki based on Attempto Controlled English and Grammatical Framework.”  Indeed, only 5 of the 36 main-track papers had any screenshot of a working system at all!  ESWC also offers an “in use track”,  it is particularly odd that only 2 of the 6 papers in this track came with a screenshot—the other 4 were basically narratives about experts putting their data on the Semantic Web—useful, but certainly not end-user applications.

A glance back at ISWC 2012 shows a similar lack of end user applications; much the same topics are covered as in ESWC 2013.

I have a number of possible explanations for what is going on here; I suspect the truth is some mixture of them.


Less Semantic, More Web

The Semantic Web is a relatively new research community, and one would imagine that it was established to tackle a relatively new research direction.  But for many of the papers at this conference, it isn’t clear that’s the case.  The mainstays of the conference–knowledge representation, inference, and ontologies—have seen decades of study within the artificial intelligence community.  That’s because these topics are all essential for the long-term objective of modeling cognition and developing true artificial intelligence.

This is obviously an important goal, but I question the importance of its connection to the Semantic Web.  Isn’t such work on knowledge representation and reasoning still going on in the AI community? Given the fundamental nature of these problems, does the fact that we are doing our inferences over web data rather than (say) an expert system knowledge base change the problem at all?  And if there is nothing specific to the Semantic-Web about this work, what is the value of partitioning it from the AI community?

The major novelty in the Semantic Web arises from its innovation over the web, rather than its innovation over semantics.  The Web’s revolution was in making it easy for everyone to author, manage and share information.  It wasn’t really about novel systems—all the pieces already existed.  Instead, it was about a novel arrangement of those pieces that empowered end users.  The introduction of structured data can drive that revolution forward, but only if we continue to think about how end users will use that technology.


Hammers and Nails

A common risk in academic research is getting too caught up in our hammers—powerful solution techniques—and losing track of the nails—the problems that need to be solved.  I fear this may have happened in our community.  Back in the early days, we convinced ourselves that a web of structured data would be useful.  Now, we’re devoting all our energy to a hypothetical infrastructure for that web.
But we ever fully worked out just how that structured web would actually be used.  Sure, if we solve AI then we can send off our autonomous agents to do all our work for us on the Semantic Web—but on the other hand, if we solve AI, those agents will be able to understand text and won’t need the Semantic Web.  Anyway, that solution seems a long way off.

I think we have to do a much better job of demonstrating, to ourselves and to others, the more immediate benefits of the Semantic Web.  And we can only do this by showing how the Semantic Web can solve problems that end users have right now.  I don’t mean the generic utopian vision—I mean nails.  We have to describe specific end-user problems and demonstrate specific Semantic Web applications that will solve those problems.  If we fail to do that, if we create hammers without nails, I doubt we’ll ever build the right hammers.  Someone else will solve those problems without using Semantic Web tools, and the Semantic Web will be left behind.

More of our research should start with the identification of a current end-user problem, because then we’ll know that there’s a real reason to create the solution.  Each of the four example applications I described last time did so: personal information management and integration under your own schema (Haystack), making spreadsheets work better (Related Worksheets), publishing cool interactive data visualizations (Exhibit), and automatically coping with incoming information streams (Atomate).

I’ll specifically highlight Atomate, because it’s actually quite close in spirit to the agents proposed in the Semantic Web visioning paper,  but with an important difference.  We don’t know how to build autonomous Semantic Web agents yet.  But we can lower our sights, give people a simple controlled natural language for query specification and get something that we can build and use right now.  As such, we can point to it and say “look, if the Semantic Web were widespread, everyone could immediately benefit from something like this!”

And supporting my warning that the Semantic Web might be left behind as irrelevant, I’ll remind you of the If This Then That tool I described in the last post.  This is a tool that is tackling the same problem, but solving it without using Semantic Web technology.  This makes it inferior in ways I discussed last time.  But it is clearly superior in the most important way: it’s actually out there, being used by people to solve their current problems.  If we wait too long to offer something better based on the Semantic Web, people will get comfortable with the lesser solution.  An O’reilly post last year speaks approvingly of “API-centric architectures” (like IFTTT) and disparagingly of the Semantic Web—if we don’t take steps to demonstrate the superiority of our approach, then theirs will win by default.

As an aside, I’ll propose a specific remediation here: next year’s Semantic Web Challenge should be to create a Semantic Web version of If This Then That.  We could compare different instantiations of the conveniently names SWIFTTT on usability, power, and usefulness.  IFTTT shows that if we actually created one that worked, there’d be clear demand for it.



I’ve been focusing on the papers, but I also want to talk about the demo track and two pervasive issues I saw there.  The demo track had plenty of applications.  But it suffered from the same problem that I wrote about in a blog post last year, which initiated my thinking that led to this keynote.  Far too many of these demos called themselves Semantic Web applications because they stored their data in RDF.  This makes no sense as the definition of a Semantic Web application.  From the perspective of the end user, the particular storage model ought to be invisible.  Any system that stores its data in RDF could store it in a SQL database instead with its user none the wiser (indeed, the user is unlikely to know what RDF and SQL are).  Instead, I’ll state once more what I consider the key characteristic of Semantic Web applications:

A Semantic Web application is one whose schema is expected to change.

Few of the applications demoed at ESWC met this description.  Rather, the majority seemed to have a fixed schema for their data which was hard-coded into the user interface.  The fact that the underlying data was RDF might make it easy to change the schema in the data model, but there’s no hint of how that change might easily be propagated to the user interface.



Another serious problem with the demos was the lack of evaluation.  At first glance this is understandable: evaluation is the difference between a demo and research paper.  But consider this question: where are all the evaluations of last year’s demos, which should have shown up as research papers in this year’s conference?  There aren’t any.  Which suggests that our builders are stopping at demo and never actually evaluating what they built.

If true, this is a fundamental flaw in our community.  As a scientific discipline we must have evaluation; without it we can make no progress.  Can anyone other than the creator actually use the system they built?  Does anyone see any benefit to doing so?  Without positive answers to these questions, our demos tell us nothing.  As academics, we aren’t building tools for use; we’re building tools as experiments that yield data for us to analyze.  If we don’t do that, we’re wasting our time.

This problem extended to some of the in-use papers as well.  I attended the in-use track and, after several of the talks, asked some variant of “how did making this a Semantic Web application make it better than a traditional application?”  Some speakers didn’t really have an answer; other did but hadn’t put it in their paper.  I particularly remember a paper on Hafslund Sesam, whose presenter, when prompted, went on at length about how much easier it had been to build their system using Semantic Web technologies instead of traditional databases.  This was great to hear, but (1) this argument is barely made in the paper itself and (2) there is no (evaluation-based) proof—only opinion.  Where is the comparable system that tried to do things the old fashioned way and was harder?  How do we know that Semantic Web technologies were actually better here, as opposed to being what the developers found most familiar?

The three examples I sketched last time all included some form of evaluation.  In Related Worksheets, we concentrated on usability: could end-user answer questions faster with these than with traditional spreadsheets?  Similarly, with Atomate we asked whether end users could actually create rules using our controlled natural language, as well as asking whether end user perceived value in having such a tool.  With Exhibit, we’ve begin (and are continuing) to ask what kind of data and visualizations end users want (and are able) to publish on the web.  With Related Worksheets and Atomate we did “lab studies”—bringing users into our lab to carry out specific tasks we defined for them.  These are generally great for understanding usability, pretty easy to set up, and not too time consuming.  There are simple metrics like task correctness and time to completion that can be used to compare different tools.  Exhibit is more of a “field study”—we’ve put the tool out in the world and gotten some adoption, and are now asking what users are doing with it.  These studies tend to be messier, requiring more time and effort to deploy the tool and analyze the data, but also more realistic.



Winding up this long post, I hope I’ve convinced you that the Semantic Web reflects some of the key insights, on schema variability, that I think are critical to improving people’s ability to manage information.  But we aren’t doing the work.   We’re devoting far too much energy to studies of knowledge representation, reasoning, and information extraction that have traditionally appeared in artificial intelligence conferences, and perhaps should continue to do so.  We build applications, but we call them demos and don’t evaluate them.  Many of them aren’t really Semantic Web applications; they’re just traditional applications that happen to be storing their data in RDF.   We’re letting a great opportunity pass us by; I hope we can think about ways to seize it.

Finally, for those who’ve made it this far, here are my slides and those of Voida et al.

Keynote at ESWC Part 2: How the Semantic Web Can Help End Users

I’ve just returned from the European Semantic Web Conference, where I gave a keynote talk on “The Semantic Web for End Users”.   My talk addressed the problem that has interested me for eighteen years: making it easier for end users to manage their information.  The thesis was that

  • The current state of tools for end users to capture, manage, and communicate their information is terrible (yesterday’s post), and
  • The Semantic Web presents a key part of the answer to building better tools (this post), but
  • Not enough work is being directed toward this problem by the community (tomorrow).

Since I had a lot to say (217 slides) I’m breaking the summary into three separate posts aligned with these three bullets.  Come back tomorrow for the next.


Our story so far

Yesterday, I discussed the dire state of information management for end users.  I argued that our traditional applications are designed around fixed schemas, and that whenever an end user wants to use their own schema, or connect information from different schemas, these traditional applications fail them.  Users are forced to settle for generic tools like spreadsheets and spread their data over multiple tools.  Voida et al.’s Homebrew Database paper (a must read) shows how awful the results are.


The Semantic Web can Help

The Haystack Client

The Haystack Client is managing a schema-mixed collection of email messages, news items, and people. All are displayed in a list collection, but each type is shown using its own “lens”. Facets can be used to filter the collection. Context menus expose operations suited to each type of item.

Our first attempt to address the “schema diversity” problem was Haystack, a tool that could be used to store and visualization arbitrary information.  Haystack could store arbitrary user-defined entities with arbitrary properties and relations to other entities, and also allowed its user to customize visualizations of those entities.  You could create something that looked quite like a traditional application, over whatever schema you decided was useful.

We created the first version of Haystack before the Semantic Web was visioned.  However, it was obvious after the fact that Haystack was a Semantic Web application (more specifically a Semantic Desktop) and when RDF was released as a web-standard data model, we adopted it as the native model for later versions of Haystack.

Haystack reflects what I consider the key novel perspective of the Semantic Web community—the idea of a web of data reflecting vast numbers of distinct schemas.  While the database community has devoted significant effort to data integration, their canonical example has been, e.g., the combination of a few large corporate databases when two companies merge.  It hasn’t really addressed the far more anarchic situation of a different schema on each web site.

I believe that this setting demands a new kind of application.  Instead of traditional applications with their hard-coded schemas and interfaces, we need applications like Haystack whose storage and user interface can effectively present and manipulate information in any schema that their user encounters or creates.  This is a challenging task since we tend to rely on knowing the schema to create good user interfaces; however, I believe the challenge can be met.


Concrete Examples

The related worksheets system being used to display information about courses.  Each course has reading and sections, with presentation nested inside the relevant cells of the courses table.

The related worksheets system being used to display information about courses. Each course has reading and sections, with presentation nested inside the relevant cells of the courses table.

To support this argument, I presented three of these flexible-schema Semantic Web applications.  The first is Related Worksheets, being developed by my student Eirik Bakke.  Eirik recognized the incredible dominance of spreadsheets as a schema-flexible data management tool, and asked how we can make spreadsheets better for this task without changing their fundamental nature.  His approach is to improve spreadsheets to better present and navigate the entities and relationships represented in them.

A typical spreadsheet may have, e.g., one table consisting of university courses (one row per course) referring to another table consisting of course readings (one row per reading) and another table of course instructors.  In a traditional spreadsheet this “reference” is just a textual correspondence—there’s a cell in the course table that names the title of a reading that’s in the readings table.  But if you recognize that the reading is actually an entity, you can do better. First, you can present information about each reading nested inside the cell in the course listing table, so you can immediately see more information about the reading without having to go find it in the readings table.  Second, you can “teleport” from the reading shown in the course table to the corresponding row in the readings table, where you can see or modify more data about the reading (and, e.g., teleport onward to the author of the reading).  A user study showed that these features can significantly improve end users’ speed at extracting information from the worksheet.

    The Exhibit framework being used to present information about presidents of the United States, plotted on a map and a timeline. Facets on the left offer filtering by religion and party, along with text search.

The Exhibit framework being used to present information about presidents of the United States, plotted on a map and a timeline. Facets on the left offer filtering by religion and party, along with text search.

I then presented is Exhibit, a tool that lets end users author interactive data visualizations on the web.  The motivation for Exhibit was the recognition that while professional organization are able to create fancy data-interactive web sites that offer templating, sorting, faceted browsing, and rich visualizations, end users generally don’t have the necessary programming and database administration skills necessary to do so, and thus tend to publish only text and static images.

My student David Huynh recognized that a large number of the professional sites fit a common pattern, and that it was possible to add a quite small extension to the HTML vocabulary that was sufficient to describe these professional sites just in HTML.  The vocabulary describes common elements such as views (lists, tables, maps, timelines), facets (for filtering the data shown in the views), and lenses (HTML templates for individual items).  Any end user can drop these extended HTML elements into a web page, point them at a data file (spreadsheet, json, or csv) and instantly publish an interactive data visualization.  To make it even easier, Ted Benson and Adam Marcus created Datapress by integrating Exhibit into WordPress, so you can “blog your data” using WordPress’ built-in WYSIWYG editor.

There are now over 1800 exhibits online, covering an incredible spectrum of data sets—from ocarinas  to failing bridges, European Court for Human Rights cases, pollution measurements in Spain,  map stores, classical music composers, strange sports, ,  mining information, teacher bonuses in Florida  and an Urdu-English dictionary.

By the way, anybody who wants to try exhibit for themselves can just copy one of the ones on the web and start playing with it.  For example, if you’re an academic, perhaps you could use a nicer publications page.  Just download mine and replace the data with your own.  But if you want a more careful introduction, take a look at this tutorial I put together.

Atomate being used to generate the rule "". The user adds one word at a time; each step uses a dropdown/autocomplete to ensure only valid words are added.

Atomate being used to generate the rule “remind me to take out the trash when I get home on Tuesday evening.” The user adds one word at a time; each step uses a dropdown/autocomplete to ensure only valid words are added.  Click the image for a video demonstration.

The last tool I described was Atomate, built by my student Max van Kleek and Brennan Moore to demonstrate how end users could author automation rules to reduce their effort handling incoming social media and other information streams.  For example, a user might want to be notified when their calendar shows that a certain band is performing and their social media stream reports that a particular friend is in town, so that they can attend the performance together.  A big challenge is coming up with a query language that is simple enough for end users.  We settled on a controlled natural language—a query language that looks like English but is actually unambiguous filters over the properties and values in the user’s structured data collection.  Drop-down menus and autocomplete ensure that the user is only able to create meaningful queries.  You can click the image on the right to see a demonstration video.

A user study of Atomate revealed that users were able to create meaningful queries when given a specific task, that they recognized the general utility of this system, and that they were able to envision specific ways (particular rules they could write) to use it for their own benefit.

Since the publication of the Atomate work, we’ve seen some of its approach appear at If This Then That, a web site that can connect to numerous web sites to pull and push data, and that lets end users specify triggers to activate when that data changes and actions to take that may modify other data.



I’ve now outlined four applications that, in my mind, leverage the “special sauce” of the Semantic Web—the idea that applications must be designed to work effectively over whatever schemas their users choose to create or import.  This creates major challenges in the design of user interfaces, since we often want to leverage a hard-coded schema to determine the ideal domain-specific interface.  But there are ways around this problem, either using generic interfaces like spreadsheets (Related Worksheets) or natural language (Atomate), or putting more of the user interface authoring in the hands of the end user (Haystack and Exhibit).  Each of these tools demonstrate that it is possible to give an end-user a tool that can work with arbitrary schemas.

Given the potential, I’m disappointed with the level of effort being invested in this kind of work by the Semantic Web community.  In my next post, I’ll discuss what work I think is missing, how to do it well, and changes we might make to our Semantic Web conferences to encourage it.