Study Finds Generative AI Significantly Boosts Graduate-Level Writing Efficiency and Quality

A new study on the impact of generative AI on graduate-level writing finds that, with proper instruction, it can drastically reduce writing time and elevate average grades, demonstrating its transformative potential in academic and professional writing. The findings have important implications for educational strategies in teaching writing with AI assistance and for professional writing practices. 

The research examines the effects of generative AI tools, such as ChatGPT and Microsoft Copilot, on the productivity and quality of writing among graduate students engaged in graded assignments within an actual classroom setting. After receiving targeted instruction on the effective use of these tools, students experienced not only a significant reduction in writing time but also a noticeable improvement in the quality of their work. This enhancement was observed across both native English speakers and ESL (English as a Second Language) students, demonstrating the broad applicability of generative AI in academic contexts.

The study, conducted by researchers at Carnegie Mellon University, is published as a working paper on the Social Science Research Network. 

“Our findings show that with proper guidance, generative AI can be a powerful tool for efficiently creating sophisticated graduate-level content,” says Jordan Usdan, the lead researcher and adjunct instructor of Generative AI: Applications, Implications, and Governance at Carnegie Mellon’s Heinz College. “This research underscores the importance of a tailored educational curriculum to fully leverage these tools. We're only beginning to explore AI's potential to revolutionize learning and enhance human expression.”

The capabilities of generative AI are particularly relevant in educational and professional settings, where demand for high-quality, efficient writing is high. But despite its widespread adoption, many find it challenging to use the technology effectively.

In this study, researchers measured how classroom instruction about the tool affected graduate students’ use of generative AI on a graded writing assignment. Participants were 27 graduate students enrolled in a course toward a degree program at Carnegie Mellon in spring 2024, a mix of native English speakers and ESL graduate students from a variety of academic disciplines.

Students were first assigned a professional memo writing task that required research, data analysis, policy development, and policy analysis, to be completed without the assistance of generative AI. Following this, they received comprehensive instruction on generative AI, covering system development, operation, prompt engineering, and common pitfalls, along with practical exercises across various research and writing sub-tasks. Finally, students were given a similar professional writing task where they had the option to use generative AI—an option that all participants chose to utilize.

Generative AI reduced students’ average writing time by 65% and improved their average writing quality from a B+ to an A. These benefits occurred for both native English-speaking students and ESL students, though they were slightly more pronounced for the ESL students, suggesting a leveling effect that helped ESL students catch up to initially higher-performing, faster peers. These improvements were consistent across both strong and weak writers."

When students were polled about their experiences using generative AI, they said they found it to be most useful for reducing the amount of time it took them to write, for summarizing information, and for general assistance in writing. They also found it helpful for web research, policy development, and analysis, though to a slightly lesser extent. ESL students found generative AI to be significantly more useful in the writing process than did native English speakers.

“Our results are further evidence that teaching students how to appropriately leverage generative AI for writing assistance is becoming an essential job for educators,” says Harley Chang, a learning engineer at Carnegie Mellon’s Eberly Center for Teaching Excellence & Educational Innovation, who coauthored the study. “Given these benefits, all colleges should consider incorporating generative AI instruction into curricula in undergraduate and graduate programs.”

While the study demonstrated significant improvements in both grades and writing efficiency with the use of generative AI, the authors acknowledge the challenges of isolating AI's impact within a real-world classroom setting. However, students themselves credited 66% of their productivity improvements to the AI rather than to additional writing practice. Another limitation is that the study's findings are based on a self-selected sample of participants, which may limit the generalizability of the results.

“Generative AI has strong potential to help not only student writers but also professionals in the workplace,” notes Allison Connell Pensky, a senior data science research associate at Carnegie Mellon’s Eberly Center for Teaching Excellence & Educational Innovation, who co-authored the study. “The pedagogical method we used may also be helpful for workplace training to maximize the potential benefits of generative AI while managing its challenges and ethical considerations.”

The research was funded by Carnegie Mellon University.

###

About Heinz College of Information Systems and Public Policy
The Heinz College of Information Systems and Public Policy is home to two internationally recognized graduate-level institutions at Carnegie Mellon University: the School of Information Systems and Management and the School of Public Policy and Management. This unique colocation combined with its expertise in analytics set Heinz College apart in the areas of cybersecurity, health care, the future of work, smart cities, and arts & entertainment. In 2016, INFORMS named Heinz College the #1 academic program for Analytics Education. For more information, please visit www.heinz.cmu.edu.