ChatGPT and Systematic Literature Reviews

Earlier today I was working on a new LibGuide about ChatGPT and Higher Education and ran across a YouTube video by physician Benjamin Tran claiming that you could use ChatGPT to write a systematic literature review in an hour. Since I'm in the process of writing one, I was curious how this was possible, but assumed that he was using the AI to cut down the time required writing and analysis.

I was wrong.

Take a gander for yourselves.

The type of review Tran describes here is not a systematic literature review. It's a quick and dirty literature of sources available on the Internet via PubMed and some additional web sources. Beyond the issue of a limited set of sources, many of which are not scholarly, this search is not replicable because we don't know what ChatGPT searches, what words and phrases it relies on, or even the date range of the sources found was. It's the kind of search you would expect from an undergrad on a short deadline (and we've all done them).

Tran's tutorial is interesting in that he shows how to use ChatGPT to conduct searches and to start writing a literature review based on a standard format, and create some citations. This is an interesting look at what we might expect students to use ChatGPT for in their work. Tran does a really good job of explaining the problems and limitations of using an AI for this type of work, which makes this a valuable resources.

The problem is one of terminology. What Tran created using ChatGPT was not a "systematic" literature review. If he weren't an MD, I would write this off as just a misunderstanding of what a systematic literature review is, but this type of review originated in the health sciences. The goal is to make literature reviews a replicable process and remove some of the bias that we bring to our interpretation of research articles by documenting the process, registering protocols, and analyzing the literature in teams using tools and rubrics. 

That's not what this is, so it's also an example of the need to be careful with claims of what ChatGPT and its cousins are capable of in academic work.

Systematic Literature Reviews

In September 2021 I started working on a systematic literature Review with another librarian here at the university after helping one of the faculty in our School of Education get started on a similar project with some of her colleagues. systematic literature reviews are a type of study that developed in the health sciences as a way to make literature reviews more like a mixed methods study that could be replicated, and have slowly been adopted in other social sciences, including Library Science. At this point, there have been relatively few of these projects published in library journals, so it seemed like a project that would be interesting and have a good chance of being published. Librarians here are encouraged to publish, but publication is just one of the "creative scholarly projects" that we are evaluated on each year or if we seek promotion.

We just presented our work at the Association of College & Research Libraries Conference, which was held in Pittsburgh this year, but our presentation was not a complete view because our analysis is not complete. Trying to juggle the schedules of two librarians and a paraprofessional staff member (who also has an MLIS) makes getting a study of this sort done, especially when one of the team members is working on a doctoral degree.

Also, systematic literature reviews are hard.

Very, very hard.

Here it is in a nutshell:

1. Choose a topic
2. Create a study protocol
3. Develop search terms and phrases
4. Document search phrases and information sources searched
5. Download all of the articles and/or request via ILL
6. Remove duplicates
7. Exclude articles that aren't relevant
8. Analyze articles using CASP or similar checklists
9. Identify trends and themes
10. Write it all up using the PRISMA checklist
11. Publish!

After many fits and starts, we're at stages nine and ten. My goal is for us to get this thing written up and submitted to a journal by July. 

I'll probably expand on the process over the next few weeks, but one of the things I've been thinking about is how this would look for a literature review in History - not that I imagine that anyone would publish such a thing if it existed.