"Understanding Confusion: A Case Study of Training a Machine Model to P" by Ramanakumar Sankar, Kameswara Mantha et al.
 

Document Type

Article

Publication Date

12-9-2024

Department

Department of Mechanical and Aerospace Engineering

Abstract

Citizen science has become a valuable and reliable method for interpreting and processing big datasets, and is vital in the era of ever-growing data volumes. However, there are inherent difficulties in the generating labels from citizen scientists, due to the inherent variability between the members of the crowd, leading to variability in the results. Sometimes, this is useful — such as with serendipitous discoveries, which corresponds to rare/unknown classes in the data — but it might also be due to ambiguity between classes. The primary issue is then to distinguish between the intrinsic variability in the dataset and the uncertainty in the citizen scientists’ responses, and leveraging that to extract scientifically useful relationships. In this paper, we explore using a neural network to interpret volunteer confusion across the dataset, to increase the purity of the downstream analysis. We focus on the use of learned features from the network to disentangle feature similarity across the classes, and the ability of the machines’ “attention” in identifying features that lead to confusion. We use data from Jovian Vortex Hunter, a citizen science project to study vortices in Jupiter’s atmosphere, and find that the latent space from the model helps effectively identify different sources of image-level features that lead to low volunteer consensus. Furthermore, the machine’s attention highlights features corresponding to specific classes. This provides meaningful image-level feature-class relationships, which is useful in our analysis for identifying vortex-specific features to better understand vortex evolution mechanisms. Finally, we discuss the applicability of this method to other citizen science projects.

Publisher's Statement

© 2024 The Author(s). Publisher’s version of record: https://doi.org/10.5334/cstp.731

Publication Title

Citizen Science: Theory and Practice

Creative Commons License

Creative Commons Attribution 4.0 International License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Version

Publisher's PDF

Plum Print visual indicator of research metrics
PlumX Metrics
  • Usage
    • Abstract Views: 6
    • Downloads: 2
  • Captures
    • Readers: 1
  • Mentions
    • News Mentions: 1
see details

Share

COinS