Automatic Text Analysis of Reflective Essays to Quantify the Impact of the Modification of a Mechanical Engineering Course

Document Type

Article

Publication Date

1-9-2024

Department

Department of Mechanical Engineering-Engineering Mechanics

Abstract

Investigations on the utility of reflective essays in engineering education are firmly grounded in the theory of metacognition. Besides being a tool to measure metacognition, insight acquired from assessing reflective essays can provide context for future instruction and assessment. Some challenges that hinder the critical assessment of reflective essays is the assessment of the large volume of text generated, and the difficulty in quantitatively assessing threshold concepts that are embedded in reflective essays and are communicated via free-form writing, whose presence and recurrence is indicative of an identity transformation during an education experience of learners.

This paper demonstrates an automated, quantitative assessment process composed of Text Mining (TM), Natural Language Processing (NLP), and Recurrence Quantification Analysis (RQA) to assess the presence of a specific thematic element in reflective essays for the purpose of confirming the impact of the modification of the assessment structure in producing a change in students’ focus. The thematic element of interest an attitude of teamwork or “working in teams” in an immersive course in team-driven, model-based engineering design.

The novel innovation of this approach is that the input (text from hundreds of reflective essays, sourced one at a time) when passed through this process quickly produces an output that quantifies the presence of thematic elements and their recurrence thereby quantitatively signaling a change in student focus towards a desired outcome.

Publication Title

Journal of STEM Education: Innovation and Research

Link to Full Text

Share

COinS