Validating Assessment Instruments for Use in Engineering Education: A Primer for Conducting and Interpreting Factor Analysis

Document Type

Conference Proceeding

Publication Date

6-23-2024

Department

Department of Engineering Fundamentals; Department of Cognitive and Learning Sciences

Abstract

Much of the researchers’ recent work focuses on validating the MUSIC inventory of Jones and associates. This inventory assesses students’ self-perceptions of motivation-related factors in an academic environment. Previous validation work took place in general education or free elective courses. It has not seen widespread use in engineering education. Thus, our near-term goal is to establish the MUSIC inventory, or an adaptation of it, for use in engineering education research.

Establishing the validity of an instrument does not occur in isolation from its use in a given context. For the MUSIC inventory to be a valid tool in engineering, we sought to validate the inventory through a series of studies. Confirmatory factor analysis (CFA) of the five-factor scale measuring the constructs of eMpowerment, Usefulness, Success, Interest, and Caring has indicated that the Interest factor may be problematic (i.e., cross-loading with items on the usefulness factor, among others).

In this report, we will describe our work in validating the MUSIC inventory to exemplify the process of establishing the validity of a pre-existing instrument when used in relatively unexplored domains while conducting discipline-based education research (DBER). More specifically, we strategically examine potential validation issues related to the Interest factor of the MUSIC inventory when used in a first-year engineering education (FYE) course.

Our overriding goal is to offer a primer to DBER researchers who may not be familiar with confirmatory and exploratory factor analysis techniques (CFA and EFA) used in multi-dimensional instrument validation. We take as a case the techniques used in our current efforts to validate the MUSIC inventory within engineering education. We discuss confirmatory factor analysis (CFA) and the underlying assumptions necessary to perform CFA with ordinal data, such as normality and homogeneity of variance. We will describe the various model options when conducting CFA and summarize how to assess a model’s fit. If model fit is lacking, we demonstrate returning to exploratory factor analysis (EFA) to assist in interpreting the model. Throughout, we will use our work with the MUSIC inventory as a case study to illustrate the methodology.

Our case study underscores the central importance of replication studies. Although not common in engineering education, replication is the core method used to validate original study conclusions, utilize assessment tools within new contexts, apply conclusions, or utilize cognitive models within varying contexts (e.g., external validation). If new validation efforts result in contradictions from the original work, one must either abandon the method(s) or update the tool, model, or findings given the new conditions and/or results. For us, we believe that the MUSIC inventory (or a modified version of the tool) can be an effective accelerant in identifying the importance of intrinsic motivation in students for learning.

Publisher's Statement

Publisher's permanent URL can be found here

https://peer.asee.org/48250

Publication Title

ASEE Annual Conference and Exposition, Conference Proceedings

Link to Full Text

Share

COinS