Annotating Qualitative Social Science

A vibrant conversation is underway about how research data should be curated, managed, and shared. While these were not initially prominent questions in qualitative social science traditions, more recently, discussions have ensued across a wide range of scholarly contexts.¹In the last three years, for example, see: Aremu, F. & MacLean, L.M. 2016; Batty, F. 2016; Blattman, C. 2015; Breuning, M. & Ishiyama, J. 2016; Capoccia, G. 2016; Carsey, T. 2014; Dafoe, A. 2014; Ellett, R. & Massoud, M.F. 2016; Elman, C. & Kapiszewski, D. 2014; Elman, C., & Lupia, A. 2016; Flaherty, C. n.d.; Fuji, L. A. 2016; Gastinger, M. 2015; Gelman, A. 2015; Grynaviski, E. 2016; Hall, P. A. 2016; Hayes, J. 2015; Htun, M. 2016; Isaac, J.C. 2016; Ishiyama, J. 2014; Janz, N. 2015; LeBas, A. 2016; Leeper, T. J. 2014; Lupia, A. & Alter, G. 2014; Lupia, & Elman, C. 2014; Lynch, M. 2016; McDermott, R. 2014; Moravcsik, A. 2014; Moravcsik, A. 2016; Morrison, 2016; Patty, J. 2015; Pepinsky, T. 2015 and 2016; Rohlfing, I. 2015; Sil, R., Castro, G., & Calasanti, A. 2016; Subotic, J. 2016; Tripp, A. 2016; Wilson, R. 2015a, 2015b, and 2015c; Yao, J. 2016; and Yashar, D. 2016.

Different approaches to research provide a variety of contributions to knowledge-making, and this methodological pluralism is a source of great strength in social inquiry. Not surprisingly, however, this diversity is also reflected in the wide range of scholars’ positions on openness. Our view is that particular research communities should reach their own consensus on these questions based on their members’ collective judgements. They are in the best position to decide what kinds of information they should provide to each other in order to enhance the credibility and legitimacy of their conclusions.

Often these local conversations are framed as cost-benefit calculations. On the benefit side, the call for transparency in part springs from the recognition that procedurally contingent knowledge claims are strongest when they can be reproduced, which in turn requires open data and materials (Miguel et al. 2014). This suggests that scholars should strive to share the data they used, and to describe how they generated and analyzed those data to reach their conclusions, or explain why they cannot (Lupia and Elman 2014; Elman and Kapiszewski 2014). Simultaneously, there is broad recognition that openness may sometimes be costly, particularly when social science data, both qualitative and quantitative, are infused with ethical and legal complications.

Where scholars stand on these cost-benefit calculations will strongly influence their position on transparency. Academics who think there are considerable knowledge-making advantages to openness and whose work involves fewer ethical and logistical constraints are likely to be more optimistic about the promise of openness. Those who believe openness yields meager epistemic dividends and produces substantial risks will likely have a negative view. It is thus important that scholars making such calculations rely on accurate information and make appropriate comparisons.

For example, one compelling statement about openness suggests that data, as well as documentation about how data are generated (i.e., “materials”), should be “FAIR”—that is, findable, accessible, interoperable, and reusable (Wilkinson et al. 2016).²Just as importantly, of course, FAIR-ness also ensures that data can be used for secondary analysis. Making data and documentation FAIR requires that scholars actively engage in data management planning. Scholars may consider generating and implementing a data management plan to be a significant burden. Yet scholars who collect data carefully and document their procedures are likely already undertaking a large part of what a formal data management plan calls for. Hence, the steps required to make their research more transparent may not impose unreasonable burdens.

Moreover, costs and benefits are likely to change as technology develops further. The notion that innovation can facilitate openness is hardly novel, and is widely shared among quantitative scholars. Technological development also holds great promise for augmenting openness in qualitative research. For instance, transparency requires that data and materials be available to readers in ways that facilitate understanding and evaluating research. Data and analyses are interwoven throughout the text of qualitative scholarship.³While we use “article” throughout this statement, the challenges hold equally true for, and our proposed solution works equally well with, any web-based digital scholarship including blogs, preprints, working papers, and so on. To optimize openness in such work, digital data sources (e.g., archival documents, audio recordings, interview transcripts, ethnographic field notes) and relevant analytic information must be immediately available where the data sources are invoked in the publication (i.e., across the span of an article). They also must be accessible from the article as it is displayed on a journal’s web page (i.e., on the publisher’s platform). However, making data proximate in these ways implies fiduciary and technical responsibilities that exceed what most publishers are willing or able to deliver.

A second transparency requirement concerns protection. Tensions can sometimes arise between transparency and the protection of human participants. Proponents of openness explicitly acknowledge this tension when calling for scholars to make their data “as open as possible [but] as closed as necessary” (ERAC 2016, 15; emphasis added). The key question scholars must ask themselves concerns what is “necessary.” Technology is providing an increasingly sophisticated toolbox of techniques for sharing sensitive data. For example, de-identification might allow some types of data to be shared while still protecting human participants. Sensitive data can be protected via encryption as they travel through cyberspace. Moreover, access controls can be applied to limit the number of people who can see sensitive data and how much of the data they can view. Safely sharing sensitive data—including developing and administering systems to authenticate users and encrypting data and materials—is another burden that few publishers are likely to want to assume.

FAIR-ness, proximity, and protection can be provided in qualitative research through digital enrichment using the open annotation framework. “Open annotation” allows for the generation, sharing, and discovery of digital annotations across the web (Sanderson et al. 2017). The Qualitative Data Repository (QDR, www.qdr.org) and Hypothes.is (https://hypothes.is/), a nonprofit technology firm that develops open source software enabling the creation, storage, and sharing of web-standard annotations, are collaborating to build on that framework. Specifically, they are developing a new approach to transparency in qualitative research: annotation for transparent inquiry (ATI). ⁴ATI draws on an earlier technique, Active Citation (Moravcsik 2010), which fulfilled similar transparency goals, but lacked the FAIR-ness of ATI.

Using ATI allows authors to produce a data supplement to their article consisting of a set of digital annotations. Each annotation is anchored to a segment of article text published on the web and contains one or more of the following elements:

• A source excerpt: typically 100 to 150 words from a textual source; for handwritten material, audiovisual material, or material generated through interviews or focus groups, an excerpt from the transcription;
• A source excerpt translation: if the excerpt is not in English, a translation of the key passage(s);
• An analytic note: discussion that illustrates how the data were generated and analyzed and how they support the empirical claim or conclusion being annotated in the text;
• A persistent link to the underlying data

Digitally enriching qualitative scholarship using ATI requires participation by and partnership between publishers and repositories, and allows each stakeholder to do what it does best.

Repositories host data and materials (including annotations), making them FAIR and protecting them. Publishers publish articles and facilitate making the relevant data proximate on their platforms. Annotation software carries out the coordination between the two types of stakeholder.

Importantly, access to sensitive data and materials can be controlled, as they are stored in a trusted digital repository. Indeed, the utility of ATI is premised on a counterintuitive observation: controlling access to data will lead to more data being made accessible. By empowering scholars to provide unfettered access to as much data as possible (given legal and ethical imperatives) while protecting sensitive data as much as necessary, ATI should dramatically increase the amount of qualitative data and materials that are shared. As such, ATI has the potential to enhance the clarity with which descriptive and causal claims are made in qualitative research, to transform how transparency is achieved, and to impact the way qualitative social science scholarship is evaluated. This new approach to transparency should thus have a direct impact on the credibility and legitimacy of qualitative scholarship and its utility for evidence-based policy.