Unlocking ChatGPT’s Potential in Systematic Literature Reviews

ChatGPT can aid in conducting systematic literature reviews, but it requires detailed instructions and human oversight for accuracy.

#llm#chatgpt#research#prompt
schedule Sep 27, 2024
face leeron

A study recently explored whether ChatGPT could assist with one of the most labor-intensive academic tasks: conducting a systematic literature review (SLR). This task, essential in research, involves analyzing multiple academic papers to identify patterns and gaps in existing studies.

The study compared ChatGPT's ability to perform an SLR with a traditional human-led approach, using 33 academic papers as the subject. While ChatGPT successfully summarized the main themes and challenges across the literature, it showed several limitations.

One major issue was that it sometimes pulled information from irrelevant sections of papers, such as the introduction or literature review, instead of focusing on research findings. Additionally, the model struggled with accuracy in identifying page numbers and lacked the ability to filter papers from academic databases—a task still best suited for human researchers.

Despite these challenges, ChatGPT demonstrated potential as a research assistant when guided by precise and detailed prompts. The researchers highlighted that with the right instructions and human oversight, ChatGPT could help streamline the SLR process, particularly in summarizing large amounts of information. They proposed guidelines for effectively using the AI tools.

Guiding Principles:

To effectively use ChatGPT for a systematic literature review (SLR), researchers should:

  • Pre-screen papers manually: Use established methods like PRISMA to select appropriate papers.
  • Provide detailed prompts: Clearly define tasks at each step, specifying what to analyze.
  • Define research questions and frameworks: Use a specific theoretical framework to guide ChatGPT’s analysis.
  • Specify sections for analysis: Instruct ChatGPT to focus on relevant portions, like results, to avoid irrelevant content.
  • Verify page numbers manually: Since ChatGPT can misidentify page numbers, always cross-check them.
  • Document the research process: Keep records of ChatGPT's interactions for reference and repeat analysis if necessary.
  • Collaborate with ChatGPT: Use the AI as an assistant and triangulate findings with human input for better accuracy.

    • ChatGPT can't do a complete literature review on its own yet, but it can help speed up the process and offer useful insights as a support tool.

    article
    Huang, Q., & Wang, Q. (2024). Exploring the Use of ChatGPT for a Systematic Literature Review: a Design-Based Research. arXiv, 2409.17426. Retrieved from https://arxiv.org/abs/2409.17426v1

    Subscribe to my Newsletter