Topic Stability, Part 2

In my previous post I tried to illustrate how different runs of the same topic modelling process can produce topics that appear to be slightly semantically different from one another. If you keep k and all other parameters constant, but change your initial seed, you’ll see the kind of variation that I showed.

The question that I want to address here is whether we can put a number to that variation, so that we can understand which topics are subject to more semantic variability than others.

I’ve gone ahead and written a script in R that calculates the average difference between a given topic and the most similar topic to it from all other runs. You can download it in GitHub.

I use KL-divergence to calculate the difference between topics, meaning I am asking how much information is lost when we approximate a topic’s term probability distribution by the most similar topic from another model. I then take the average KLD of all these comparisons for a given topic. I have run 10 models (small N!) and so that is the basis of this average.

When I run this on my 10 models what I see is that there is a fairly wide range of “stability” across topics (mean KLD = 0.87, standard deviation = 0.68, table pasted in at the bottom along with table of topic words). The topic with the lowest average information loss between runs is the “French” topic (jean, roman, pierre, saint, review, french, donne, france, rousseau, marie) followed by the junk topic (think, people, know, way, say, just, things, something, time, now) but more interestingly also the “race” topic (black, white, african, race, racial, slave, africa, slavery, south, negro). These have average KLD scores between 0.14 and 0.15.

The topics with the highest divergence are the French-arab studies topic (french, france, arabic, first, arab, studies, two, many, since, time, century) with a score of 3.5 (!) and some sort of novelists topic (romance, dickens, tom, house, gothic, hawthorne, melville, poe, mark, conrad) with a score of 2.6. Interestingly the next highest is the poetry topic which looks very stable when you observe the top few words but obviously must have a lot more variability further down the list (poetry, poet, poets, poetic, eliot, poems, poem, prose, pound, eliots).

Two things immediately come to mind: the first is that these high-scoring topics are very high scoring with respect to the average and also low scorers. They definitely feel like outliers. Frankly, I wouldn’t invoke the French-arab studies topic as a topic. It is too unstable to do a deep dive into it. Essentially it keeps turning up as something different each time. It might even make sense to subset any analysis of a model by removing topics with these really strong variabilities. Since they are so evanescent, do we really want to conclude anything based on them? In this sense, I think this kind of measure can be very valuable in gaining confidence about the stability of topics or subsetting on a group of more stable topics.

The second idea that occurs to me is from that poetry topic. That really surprised me. I wonder what would happen if you condition on just the top words. Then again, are variations on the word “poet” really a topic? Maybe that is what this is telling us is that the topic of “poetry” is actually very unstable depending on how you model it. While you will have a strong register of top words, underneath that tip is an iceberg of variability.

If we return to the culture studies topic I discussed in the previous post, we can see that it scores almost dead in the middle of the pack (29th in terms of stability). In other words, that variability I was seeing was average for this model, neither low nor high, i.e. “some.”

Next steps are going to be to compare this to document probabilities. Does semantic instability correlate with document instability? In other words, when a topic changes its word distributions, does this mean we are getting very different distributions of articles?

Second, I want to compare this with David Mimno’s work on topic “coherence”. Are stability and coherence capturing the same thing?

Finally, I could imagine that running this across numerous values of k might be valuable — do we see a significant decline in the average topic stability as k increases? I assume stability will decrease with the rise in k, but maybe there is something telling about when it declines faster. This might be another diagnostic for choosing k. But that’s a lot of run time, so don’t expect anything on this anytime soon.