Skip to main content
search

Text as data is the idea that written and spoken words can be measured, counted, and compared like any other kind of evidence. It is the engine behind most computational communication research, from tracking how the news covers an election to studying how a rumor changes as it spreads. Thanks to AI, it has also changed more in the last three years than in the twenty before.

The phrase became standard after a 2013 paper by Justin Grimmer and Brandon Stewart, which laid out both the promise and the pitfalls of automated content analysis. Their warning still frames the whole area: “All quantitative models of language are wrong, but some are useful.” This guide, part of our Computational Communication section, walks through the four main methods and the one habit that makes any of them trustworthy.

Key takeaways

  • Text as data trades nuance for scale: you lose some meaning in each document to see patterns across thousands or millions.
  • There are four main approaches: dictionaries, supervised machine learning, topic models, and large language models.
  • No method is best in general. The right choice depends on your question, your data, and how much human labeling you can afford.
  • Validation against careful human judgment is the step that separates evidence from decoration.

What Text as Data Really Means

In a traditional content analysis, trained coders read every item and apply a codebook. That works for a few hundred articles. It does not work for ten years of parliamentary speeches or forty million posts.

Text as data methods solve the scale problem by converting each document into numbers a computer can work with. A collection of documents is called a corpus. Each document becomes a set of features, which might be simple word counts or, in newer methods, a long list of numbers called an embedding that captures meaning. Once text is numeric, you can sort it, cluster it, and track it over time.

The cost is that every conversion throws something away. Word counts ignore word order. Embeddings are hard to interpret. Knowing what your method ignores is half the skill.

Step Zero: Building the Corpus

Before any analysis, someone decides which texts are in and which are out. That decision shapes every result that follows. A study of “public opinion” built only from posts on X describes people who post on X, nothing more.

Cleaning matters too. Researchers often lowercase text, remove very common words, and trim words to their stems. These look like housekeeping, but a well-known 2018 study showed that different preprocessing choices can lead to different substantive conclusions from the same documents. Good practice is to report your choices and test whether your findings survive reasonable alternatives.

Method 1: Dictionaries

A dictionary method counts words from a predefined list. If you want to measure negativity, you count negative words. The best-known tool for social media is VADER, published in 2014, which scores text from negative to positive using a word list rated by people plus a handful of rules for things like capital letters and exclamation marks.

Dictionaries are fast, free, and fully transparent. Their weakness is context. The word “sick” is negative in a hospital review and positive in a skateboarding forum. Sarcasm, negation, and specialist language all trip them up, and a dictionary built for one domain often performs poorly in another.

Method 2: Supervised Machine Learning

In supervised learning, people label a sample of documents by hand, and a model learns to copy their decisions. You then apply the trained model to the rest of the corpus. Modern versions fine-tune language models such as BERT, which read words in context instead of one at a time.

This approach is usually more accurate than a dictionary because it is tailored to your categories and your data. The price is labor. You typically need hundreds or thousands of carefully labeled examples, and the model inherits any inconsistency in the labels.

Method 3: Topic Models

Topic models work without labels. You hand the model a corpus, and it groups documents or words into clusters that tend to appear together. The classic method is Latent Dirichlet Allocation from 2003. A popular newer option is BERTopic, which clusters documents by meaning using embeddings.

Topic models are best for discovery, when you do not yet know what is in your data. They do not tell you what a cluster means. A person still has to read examples, name each topic, and decide whether it is coherent. In this talk, the creator of BERTopic explains how the pieces fit.

Text as data methods compared: dictionaries, supervised learning, topic models, and large language models
The four main text as data approaches and their trade-offs. Summary by Momentum Hub, following Grimmer and Stewart (2013).

Method 4: Large Language Models

The newest approach is to write your codebook as a prompt and ask a large language model to apply it. No training data is needed, which is why adoption has been so fast.

The evidence is encouraging but mixed. A 2023 study in PNAS found that ChatGPT outperformed crowd workers on several annotation tasks by about 25 percentage points on average, at under $0.003 per annotation, roughly thirty times cheaper than a crowd platform. But a replication effort across 27 tasks from 11 published data sets found that performance varied widely by task and prompt, and concluded that human validation is still required. Open models you can run yourself, which we covered in our look at open models, help with privacy and reproducibility.

Validation Is the Whole Game

Every text as data method produces a number. The question is whether the number means what you think it means. Grimmer and Stewart’s most repeated advice is simply to validate, again and again.

In practice, that looks like this:

  • Hand-code a random sample with at least two trained people and measure how often they agree.
  • Compare the machine output with that human gold standard, category by category.
  • Read the errors. Mistakes that cluster in one group or topic can bias your conclusions even when overall accuracy looks high.
  • Report the numbers, including the unflattering ones.

If a study using text as data does not describe its validation, treat its findings as provisional.

How to Choose a Text as Data Method

A few honest rules of thumb:

  • If your concept maps cleanly onto specific words, start with a dictionary and validate it on your own data.
  • If you have clear categories and can label a few hundred examples, supervised learning or a validated LLM prompt will usually win.
  • If you are exploring an unfamiliar corpus, begin with a topic model, then confirm what you find with a second method.
  • If your texts are sensitive, prefer tools that run locally over sending data to an outside service.
  • Whatever you choose, budget time for human coding. It is not optional.

Key Terms in Text as Data

Six terms come up constantly in text as data research. Knowing them makes most methods sections readable.

  • Tokenization. Splitting text into units, usually words or word pieces, that a computer can count or process.
  • Bag of words. A representation that keeps word counts and ignores word order. Simple, and surprisingly useful.
  • Embedding. A list of numbers that places a word or document in a space where similar meanings sit close together.
  • Zero-shot. Asking a model to do a task from instructions alone, with no labeled examples.
  • Gold standard. A carefully human-coded sample treated as the reference for judging a method.
  • Precision and recall. Precision asks how many flagged items were correct. Recall asks how many of the true items were found.

Any text as data project should be able to report precision and recall against a gold standard. If it cannot, be cautious.

Quick Questions, Answered

Do I need millions of documents to use text as data? No. The methods help whenever there is more text than you can read carefully, and that threshold can be a few thousand customer comments.

Will large language models replace the older methods? Not entirely. Dictionaries remain useful for transparency, and trained classifiers are cheap to run at very large scale. Many strong studies combine approaches.

What is the most common mistake? Skipping validation, followed closely by treating a tool’s default settings as neutral. Every setting is a research decision.

Final Thoughts: Words Are Evidence, If You Check Them

Text as data lets us ask questions about communication that were once impossible to answer, and AI has lowered the cost of entry to almost nothing. That is exactly why the checking matters more than ever. The method you pick is less important than whether you can show it measures what you claim. For a wider view of where these tools fit, see our overview of how big data is reshaping communication research and our earlier piece on the potential of computational communication.

Obada Kraishan

A Computational Social Scientist, Research Scholar, and Software Engineer specializing in machine learning, computational methods, and full-stack development. He leverages these skills to advance research and create innovative web solutions.

Skip to content