Why are non-data driven representations of data-driven research in the humanities so bad?

One of the more frustrating aspects of working in data-driven research today is the representation of such research by people who do not use data. Why? Because it is not subject to the same rules of evidence. If you don’t like data, it turns out you can say whatever you want about people who do use data.

Take for example this sentence, from a recent special issue in Genre:

At the heart of much data-driven literary criticism lies the hope that the voice of data, speaking through its visualizations, can break the hermeneutic circle.

Where is the evidence for this claim? If you’re wondering who has been cited so far in the piece you can guess it’s Moretti. That’s it. Does it matter that others have made the exact opposite claim? For example, in this piece:

In particular, I want us to see the necessary integration of qualitative and quantitative reasoning, which, as I will try to show, has a fundamentally circular and therefore hermeneutic nature.

But does a single piece of counter-evidence really matter? Wouldn’t the responsible thing be to try to account for some summary judgment of all “data-driven literary criticism” and its relationship to interpretive practices?

To be concerned about the hegemony of data and data science today is absolutely reasonable and warranted. Data-driven research has a powerful multiplier effect in its ability to be covered by the press and circulate as social certainty. Projects like “Calling Bullshit” by Carl Bergstrom and Javin West are all the more urgent for this reason.

But there is another dimension of calling bullshit that we shouldn’t overlook. It’s when people invent statements to confirm their prior belief systems. To suggest that data is omnipotent in its ability to shape public opinion misses one of the great tragedies of facticity of our time: climate collapse (a phrase I prefer to climate “change” which is too wishy washy a word for where we’re headed — “change is good!”).

In other words, calling bullshit is a multidimensional problem. It’s not just about data certainty. Its also about certainty in the absence of data. Its about rhetorical tactics that are used to represent phenomena without adequate evidence, something that happens all too often in the humanities these days when it comes to understanding things as disparate as the novel or our own discipline.

As authors, journal editors, peer-reviewers, researchers and teachers we need to wake up to this problem and stop allowing it to pass with a mild nod of the head. We need to start asking that hard question: Where’s your evidence for that?

Data Visualization and Reading – An Interview

Mark Algee-Hewitt and I recently took part in an interview with Elyse Graham for a special issue of English Studies on “Data Visualization and the Humanities.” You can read her introduction here and our interview here.

We touch on a bunch of topics about visualization: like whether data visualization is exclusively exploratory, whether the humanities has testable hypotheses, whether it matters if visualizations are “beautiful.” Here’s my response to the question of the next big thing in literary studies:

I think more and more pressure is going to be put on the question, “Where’s your evidence for that?” When you’re generalising about modernity, the novel, or even an author’s corpus and his or place within their culture—what is the evidence, the full scope of evidence, that supports that generalisation?

The general claim about our discipline, at the moment, is that it’s a theory exporter and method importer. A lot of talk about the digital humanities is framed as what will happen as we import methods from other disciplines—but the interesting work will happen as we accommodate those methods to the traditions within our own discipline.

I’ve worked in a number of collaborations with engineers, computer scientists—people from a number of outside disciplines. And as much as that’s important—connecting with new languages from other disciplines—there’s even greater value in bringing those languages into your own discipline, so that you can begin to have intra-disciplinary conversations; that way you can dig deep into the methods and issues that concern you most. When you collaborate across disciplines sometimes ideas dissipate in order to keep everyone interested and happy. I began my career as a translator (of German literature) and I think the most important role today is to learn to translate between these different disciplinary communities. Intimate knowledge of two scientific communities will allow us to build wholly new ways of thinking and researching. This is the part I am most excited about.

Connectivity. A Conference

Looking forward to this event tomorrow. Bringing together researchers from different disciplines to develop models of cultural connectivity. Connectivity has become the dominant framework through which contemporary knowledge is increasingly understood. From networks to clouds to close reading to reconstructing historical social worlds, making connections is at the core of what academics are expected to do. And yet the very ubiquity of the term has largely hidden it from critical view. This workshop is devoted to exploring the diversity of what it means to be connected.

Culture + Computation: New Syllabus in Cultural Analytics LLCU 614

 

I am pleased to add this year’s syllabus for my graduate course, “LLCU 614, Cultural Analytics: The Computational Study of Culture.” The aim of the course is twofold: 1) to introduce students in the humanities to the computational and quantitative methods for studying culture in order to move beyond the use of anecdotal evidence and 2) to introduce students in computer science to the importance of theory for studying culture, i.e. to avoid a naive approach to data analysis. As I mention in my opening class, this course is about valuing different ways of looking at cultural questions and also conceding major methodological flaws in our current disciplinary orientations. Everyone in the room has something valuable to add given their disciplinary training and everyone also brings essential blind spots to the study of culture, including myself. This course is about making us all more sophisticated cultural analysts.

Why your dissertation needs data

screen-shot-2016-11-25-at-10-02-41-am

Dear Future Graduate Students,

It’s that time of year to start thinking about grad school. Recruiting is not easy for me. My general sentiment around graduate training is, let them decide. Advertising or persuasion is for places like Trump University not scholarship. But I think we are at a bit of a crossroads in our field and I am concerned that too many people aren’t making good choices, potentially because of what they’re hearing from their faculty. After all, the ratio of people doing computational humanities to those who are not is tiny. The messaging is bound to be skewed. It seems important therefore to go out on a limb (yes it feels like a limb) and try to articulate why you should orient your work towards a more data-driven approach. So here goes.

Why does your dissertation need data? Because it opens up so many more questions. When your only method is to read as much as possible, first, you’ll always come up short. You can never read enough and you’ll always know it. This is one of the reasons we like to parade our erudition. It’s to cover over our knowledge of what we know we don’t know. Second, you have no principled way of making judgments about all that you have read as a whole. You have no way to contextualize those insights, to put it in conversation with the things you haven’t read. To put it another way, you have no way to generalize about what you are finding. If you want to talk about the politics of modernism or the spectrality of televisual personalities, watching or reading alone isn’t going to get you there in a convincing way. Data isn’t the be all to end all. But it does solve problems. It answers questions that you will not otherwise be able to pose.

There’s another reason too, one that I think is almost more important because it isn’t about a particular subject area. Rather, it’s about your position in the field more generally. Every day thousands of dissertations are uploaded to ProQuest. And every day we know a little bit less about our respective fields. The more research there is, the harder it is to have a sense of the field as a whole — and where your place is within it.

I remember, very distinctly, a moment I had one day wandering through the stacks as a graduate student at Columbia University, the home of Melvil Dewey. I remember thinking to myself, holy s%*t, look at all these books. What is the point of me writing one more? The aggregate value of one more book decreases every day. But the ability to use data to understand that whole to which you yourself are a contributor: that is invaluable. And you can’t get there by reading alone. Only data can do this, for better and for worse.

I know people will tell you it’s a bad idea. Or that it’s a fad. It’s not. It’s an essential part of the research process. You should be thinking about programs that will help you integrate it into your research, be able to guide you towards using it effectively and thoughtfully, and above all champion methodological plurality rather than dogma. If you’re hearing something else then you aren’t being given very good advice.

Validation and Subjective Computing

Like many others I have been following the debate between Matthew Jockers and Annie Swafford regarding the new syuzhet R package created by Jockers, which has been given a very nice storified version by Eileen Clancy. As others have pointed out, the best part of the exchange has been the civility and depth of replies, a rare thing online these days.

To me, what the debate has raised more than anything else is the question of validation and its role within the digital humanities. Validation is not a process that humanists are familiar with or trained in. We don’t validate a procedure; we just read until we think we have enough evidence to convince someone of something. But as Swafford has pointed out, right now we don’t have enough evidence to validate or — and this is a key point — invalidate Jockers’ findings. It’s not enough to say that sentiment analysis fails on this or that example or the smoothing effect no longer adequately captures the underlying data. One has to be able to show at what point the local errors of sentiment detection impact the global representation of a particular novel or when the discrepancy between the transformed curve and the data points it is meant to represent (goodness of fit) is no longer legitimate, when it passes from “ok” to “wrong,” and how one would go about justifying that threshold. Finally, one would have to show how these local errors then impact the larger classification of the 6 basic plot types.

As these points should hopefully indicate, and they have been duly addressed by both Jockers and Swafford, what is really at stake is not just validation per se, but how to validate something that is inherently subjective. How do we know when a curve is “wrong”? Readers will not universally agree on the sentiment of a sentence, let alone more global estimates of sentimental trajectories in a novel. Plot arcs are not real things. They are constructions or beliefs about the directionality of fortune in a narrative.  The extent to which readers disagree is however something that can and increasingly must be studied, so that it can be included in our models. As we’ve recently undertaken here at .txtLAB, in order to study social networks in literature we decided to study the extent to which readers agree on basic narrative units within stories, like characters, relationships, and interactions. It has been breathtaking to see just how much disagreement there is (you’d never guess that readers do not agree on how many characters there are in 3 Little Pigs — and it’s in the title). Before we extract something as subjectively constructed as a social network or a plot, we need to know the correlations between our algorithms and ourselves. Do computers fail most when readers do, too?

What I’m suggesting is that while validation has a role to play, we need a particularly humanistic form of it. As I’ve written elsewhere on conversional plots in novels, validation should serve as a form of discovery, not confirmation of previously held beliefs (see the figure below). Rather than start with some pre-made estimates of plot arcs, we should be asking what do these representations tell us about the underlying novels? Which novels have the worst fit according to the data? Which ones have the worst fit according to readers? How can this knowledge be built into the analytical process in a feedback loop rather than a single, definitive statement? How can we build perspective into our exercises of validating algorithms?

While I don’t have any clear answers right now, I know this is something imperative for our field. We can’t import the standard model of validation from computer science because we start from the fundamental premise that our objects of study are inherently unstable and dissensual. But we also need some sort of process to arrive at interpretive consensus about the validity of our analysis. We can’t not validate either.

The debate between Jockers and Swafford is an excellent case in point where (in)validation isn’t possible yet. We have the novel data, but not the reader data. Right now DH is all texts, but not enough perspectives.

Here’s a suggestion: build a public platform for precisely these subjective validation exercises. It would be a way of basing our field on new principles of readerly consensus rather than individual genius. I think that’s exciting.

This diagram captures the different stages of computational reading and the different types of practices each stage entails. Traditional close reading encompasses the first stage of “belief.” Current understandings of distant reading bring us as far as “measurement.” This model advocates for the continuation of the process in an oscillatory fashion, moving back and forth between close and distant forms of reading in order to approach an imaginary conceptual center. The initial sample (here Augustine’s Confessions) is chosen and understood with reference to a larger category (here “The Novel”), as is the new sample of quantitatively significant texts derived from the model (“Sample2”). “Sample2” is also mediated by the larger sample from which it is drawn (“Whole’”, here my subset of 450 novels that are representative of “The Novel”). The process of interpreting “Sample2” is both one of validation – did the model work – and also one of refinement – in what other ways can we understand and thus measure this group of texts? The overall process is represented as a spiral that does not return to the initial sample, but gradually, though never completely, converges on an imagined generic center.
This diagram captures the different stages of computational reading and the different types of practices each stage entails. Traditional close reading encompasses the first stage of “belief.” Current understandings of distant reading bring us as far as “measurement.” This model advocates for the continuation of the process in an oscillatory fashion, moving back and forth between close and distant forms of reading in order to approach an imaginary conceptual center. The initial sample (here Augustine’s Confessions) is chosen and understood with reference to a larger category (here “The Novel”), as is the new sample of quantitatively significant texts derived from the model (“Sample2”). “Sample2” is also mediated by the larger sample from which it is drawn (“Whole’”, here my subset of 450 novels that are representative of “The Novel”). The process of interpreting “Sample2” is both one of validation – did the model work – and also one of refinement – in what other ways can we understand and thus measure this group of texts? The overall process is represented as a spiral that does not return to the initial sample, but gradually, though never completely, converges on an imagined generic center.

 

.txtLAB Internship Winter 2015

.txtLAB is offering 3 internships for the Winter Semester available to undergraduate and graduate students. The internships will involve participation in lab research and the creation and completion of individual projects in cultural data mining. The internships will run from Feb. 1 through April 30. Application deadline is January 15, 2015. For more information, please go here.

txtLAB_Internship_Poster

PhD Fellowship in German and Digital Humanities @ .txtLAB

As part of the multi-university partnership, “NovelTM: Text Mining the Novel,” .txtLAB is sponsoring a 5 year fellowship open to students with a background in German literature or cultural studies and a demonstrated interest in digital humanities. The fellow will be part of the Department of Languages, Literatures, and Cultures at McGill University and an active participant in all activities related to the multi-university partnership, “Text Mining the Novel,” a six-tear research project funded by the Social Sciences and Humanities Research Council of Canada that involves nine universities across North America. In addition to the regular coursework, the fellow will craft a dissertation project in conjunction with the research partnership and have the opportunity to participate in exchanges at other member universities; attend annual training events at the HathiTrust Research Center; and present at annual workshops and conferences.

The ideal candidate will have native or near-native fluency in German and some exposure to methods in computational text analysis. The funding package will be worth approx. $25,000/year and will include opportunities for teaching assistantships.

For more information, please visit http://novel-tm.ca

Departmental website:

http://www.mcgill.ca/langlitcultures/graduate/programs/german-studies

Contact:

Andrew Piper

William Dawson Scholar of German and European Literature

Department of Languages, Literatures, and Cultures

McGill University

andrew.piper@mcgill.ca

https://txtlab.org