ChatGPT for text annotation

Chat GPT has made an undeniable splash. In addition to producing content, it can also analyze it.

This research group had a body of tweets analyzed by either trained human annotators, Amazon Mechanical Turk crowdsourced annotators, or ChatGPT.

The analysis classified the tweets according to: Relevance, Detection of predefined topics, Stance detection, General frame detection, and Multiple policy frame detection (14 classes).

Chat GPT outperformed crowdsourced workers from Mechanical Turk in accuracy of the text classifications, and ChatGPT’s intercoder agreement also exceeded that of crowd-workers and trained annotators.

In the above paper, the classification done on the source text (tweets) is referred to as an “annotation”. However, educators working with student annotators (with Perusall or hypothes.is for example), will be getting students to analyze a source text, and then the body of annotations produced will become yet another source text for possible analysis.

How could that be analyzed? If the class size is small, perhaps it is still manageable for one instructor to manually analyze student annotation output. If the class size is large, perhaps there is a role for some automated assessment of annotation output.

Since ChatGPT can analyze any text it is given, it could also analyze a body of annotations produced by students. Knowledge construction (KC) activities on the part of student annotators, previously analyzed manually by researchers, might well be automated in the near future via ChatGPT or another LLM.

Automated analysis is built into the Perusall platform already.

Automated text analysis platforms, like ChatGPT, or other machine learning algorithms, will compete with each other in the future on accuracy and sophistication. One should also keep in mind that two human coders do not always agree with each other when classifying text into a category, or assigning a quality score for a given passage of text!!! Students should not assume that human judgement is infallible.

ChatGPT has been used to produce summaries of student conversations in course chat forum where students are generating explanations and discussing evidence. This would likely be a similar text output as collaborative online annotation, or course discussion forums.

ChatGPT accurately summarized student chats from a collaborative online learning environment, but it may get confused by hyperbole or other idiosyncratic output by the students. The authors suggest that viewing the automated discussion output summaries might allow teachers to then focus more on frustrated or confused students – a helpful aid.