Our paper (CHI 1994-2013: Mapping two decades of intellectual progress through co-word analysis) received an Honorable Mention at the CHI 2014 conference. The paper is a graph-based analysis of papers published at CHI between 1994 and 2013. In this blog entry we try to explain the motivation for the paper and provide some useful background.
Motivation
If you are an experienced HCI researcher, how would you respond if your new PhD students ask you the following questions:
- What is HCI research and what are the mainstream topics of the field?
- Does HCI have any overarching theory that I can start to base my research on?
- Does HCI have any mainstream research methodology or ways of doing research that I can use in my research?
- Has HCI accumulated any knowledge that I can apply to new contexts?
- What are the emerging or fading research topics within HCI?
Our paper is an attempt to answer the above questions, and in fact was motivated by a new researcher joining our group.
Methodology
In our paper we analysed a total of 3152 CHI articles published during 1994-2013 and their 16035 keywords. Our analysis relies on techniques from social network analysis, hierarchical cluster and graph theory, and makes use of co-word analysis artifacts such as strategic diagrams and co-occurence graphs.
Co-word analysis rests on the assumption that a paper’s keywords constitute an adequate description of its content as well as the links the paper established between problems: two keywords co-occurring in the same paper are an indication of a link between the topics to which they refer to. The presence of many co-occurrences around the same word or pair of words points to a locus of strategic alliance within articles that may embody a research theme.
Results
By extracting the keywords from all papers and converting them into a co-word graph, we can then conduct a cluster analysis to identify research themes (Figure 1).
Figure 1. The main clusters of keywords (i.e. themes) we identified. (Please click the pictures to zoom in)
We then proceeded to map all the themes using a so-called “strategic diagram” (Figure 2). The x-axis of this diagram shows the strength of interaction between a specific research theme with others (i.e., centrality). The y-axis reflects the density of the research theme, or the internal cohesion of a specific research theme. The diagram naturally breaks up into four quadrants, and each quadrant signifies different types of research themes. For example, mainstream/motor themes are in the top-right quadrant. The result of this mapping is shown in Figure 3, which refers to all the clusters in Figure 1.
Figure 2. Strategic diagrams rely on density and centrality.
 |
 |
|
Figure 3. Strategic diagram for CHI for the period 1994-2003 (left), and 2004-2013 (right). |
|
The main characteristics we identified through our analysis is that neither of the 2 periods of CHI proceedings has any clusters in the top-right quadrant. This means that there is no “mainstream” or “motor theme” along which research is conducted. Because similar analyses have been conducted for other fields for research, we were able to reconstruct the strategic diagrams previously reported in those fields. In Figure 4 we reproduce the strategic diagrams for Psychology, Consumer Behavior, Software Engineering, and Stem-cell research, as reported in literature (see paper for references). What we find is that all these disciplines do have mainstream themes that drive research, unlike the CHI proceedings.
|
Psychology |
Consumer Behavior |
|
|
|
|
Figure 4. Strategic diagrams for various disciplines. |
Discussion
In the paper we discuss this glaring skewing of the CHI community, and we also report some further findings from our analysis. We note that HCI has become more cohesive over time, since the average centrality and density across all the clusters increased from the period 1994-2003 (0.571, 2.305) to the period 2004-2013: (0.635, 3.127). Also, the network density has grown from 0.148 in 1994-2003 to 0.206 in 2004-2013
We also note that HCI lacks motor themes, and has lots of under-developed yet transversal research themes. At the same time, some important topics from the first period are not so important today. This suggests that paradigm change is taking place in the field. Only 42 of 94 keywords (44.7%) between 1994-2003 are found again as top keywords between 2004-2013.
Based on the results, we argued that:
- As it stands, the only tradition in HCI is that of having no tradition in terms of research topics.
- The accumulated knowledge in HCI is almost exclusively grounded on very specific technological contexts. In addition, due to the rapid pace of technology designed for humans, knowledge in HCI tends to be highly contextual instead of universal like in the field of biology or physics.
- By nature, HCI research is like nomads chasing water and grasslands, making it challenging for the community to accumulate knowledge.
- Therefore, current HCI research is lacking motor themes and will also lack motor themes in the near future.
Finally, we provide answers to the questions we mention at the beginning of this article.
- What are the mainstream research themes in the field of HCI?: There are no dominant mainstream themes, but there are some relatively more centralised topics.
- Does HCI have any overarching theory that I can start to base my research on? No widely accepted theory has been used to drive research themes.
- Does HCI have any mainstream research methodology or ways of doing research that I can use in my research? No, select the method that is appropriate to the context.
- Does HCI have accumulated knowledge that I can apply to the new contexts? No, only contextual knowledge available.
- What are the emerging or fading research subfields or topics of HCI? Please see the paper for more details. Fading or faded topics include: annotation, digital libraries, documents, dynamic query, programming by demonstration, end-user programming, information foraging, information scent, awareness, video. Emerging themes include social networks, SNS, social media, twitter, Facebook, crowdsourcing, human computation, emotion, affect.
As part of the paper’s presentation to CHI 2014, a number of additional points were made to draw some links between our community’s lack of motor themes and the value we place on “implications for design”. Specifically, we argued that “implications for design”:
- Put practitioners’ needs above researchers’
- Give preference to contextual knowledge, because design requires grounding, and therefore forces compartmentalized thinking
- Are often just a well-written argument, and they provide neither a tool, theory, or data.
- Demotivate repeating studies, because my research can help you design, not do more research
- Demotivate incremental research, because it can be hard to argue for “improved” design implications.
The takeaway message of the talk: our community needs to publish more tangible & re-usable research: tools, data, theories. This can hopefully allow us to “get behind” a handful of themes and make them the motor themes of our discipline.
References
[bibtex key=Liu:Chi:2014]
Links
- Slides from the CHI 2014 talk
**Best Talk Award
- Dynamic chart with keywords from period 1994-2003: http://goo.gl/BAjyMt
- Dynamic chart with keywords from period 2004-2013: http://goo.gl/v8j1Nh
- Software for conducting this analysis in R can be found under tools.