Patrick J. Burns

Associate Research Scholar, Digital Projects @ Institute for the Study of the Ancient World / NYU | Formerly Culture Cognition, and Coevolution Lab (Harvard) & Quantitative Criticism Lab (UT-Austin) | Fordham PhD, Classics | LatinCy developer

How much Latin does ChatGPT “know”?

Abstract for workshop presentation given at CANE2023

Abstract

ChatGPT—a new chatbot-style large language model from OpenAI—has quickly caught the attention of educators. That said, while a number of discussions have addressed English-language assignments such as college essays, I have yet to see conversations emerge concerning its implications for Latin instruction. It is clear however from experimenting with ChatGPT that there is a significant amount of Latin text in its training data and enough Latin-themed English material to provide adequate context for “chat” about these texts. So, how much Latin does ChatGPT “know”? And how might this “knowledge” affect how we teach—and learn—the language. I have elsewhere discussed the pedagogical possibilities for Latin chatbots, connecting computationally-assisted chat to the tradition of colloquia in learning the language. Testing the Latin capabilities of ChatGPT has only accelerated the need to think expansively and critically about Latin chat in a classroom context. When confronted, for example, with the prompt—“Can you fix the grammar in this sentence? ‘Gallia divisa in tres pars sunt.’”—not only did ChatGPT return an accurate, corrected version, but it also explained its changes and provided grammatical context for the changes. The workshop is divided into three parts: 1. introduce large language models to an audience of Latin researchers and teachers; 2. demonstrate the performance of ChatGPT on a set of sample prompts; and 3. open up discussion about the use of large language models in the Latin classroom. This third part could be run as a kind of Q&A session—only with the questions directed to the chatbot and its answers used as provocations for further discussion among the workshop participants. By the end of this workshop, participants will gain a critical understanding of the challenges and opportunities that computational language technologies present whether in the form of Duolingo, Google Translate, or ChatGPT.