Earlier today I was working on a new LibGuide about ChatGPT and Higher Education and ran across a YouTube video by physician Benjamin Tran claiming that you could use ChatGPT to write a systematic literature review in an hour. Since I'm in the process of writing one, I was curious how this was possible, but assumed that he was using the AI to cut down the time required writing and analysis.
I was wrong.
Take a gander for yourselves.
The type of review Tran describes here is not a systematic literature review. It's a quick and dirty literature of sources available on the Internet via PubMed and some additional web sources. Beyond the issue of a limited set of sources, many of which are not scholarly, this search is not replicable because we don't know what ChatGPT searches, what words and phrases it relies on, or even the date range of the sources found was. It's the kind of search you would expect from an undergrad on a short deadline (and we've all done them).
Tran's tutorial is interesting in that he shows how to use ChatGPT to conduct searches and to start writing a literature review based on a standard format, and create some citations. This is an interesting look at what we might expect students to use ChatGPT for in their work. Tran does a really good job of explaining the problems and limitations of using an AI for this type of work, which makes this a valuable resources.
The problem is one of terminology. What Tran created using ChatGPT was not a "systematic" literature review. If he weren't an MD, I would write this off as just a misunderstanding of what a systematic literature review is, but this type of review originated in the health sciences. The goal is to make literature reviews a replicable process and remove some of the bias that we bring to our interpretation of research articles by documenting the process, registering protocols, and analyzing the literature in teams using tools and rubrics.
That's not what this is, so it's also an example of the need to be careful with claims of what ChatGPT and its cousins are capable of in academic work.
No comments:
Post a Comment