Programming by Voice: Exploring User Preferences and Speaking Styles

Document Type

Conference Proceeding

Publication Date

7-2023

Department

Department of Computer Science

Abstract

Programming by voice is a potentially useful method for individuals with motor impairments. Spoken programs can be challenging for a standard speech recognizer with a language model trained on written text mined from sources such as web pages. Having an effective language model that captures the variability in spoken programs may be necessary for accurate recognition. In this work, we explore how novice and expert programmers speak code without requiring them to adhere to strict grammar rules. We investigate two approaches to collect data by having programmers speak either highlighted or missing lines of code. We observed that expert programmers spoke more naturally, while novice programmers spoke more syntactically. A commercial speech recognizer had a high error rate on our spoken programs. However, by adapting the recognizer's language model with our spoken code transcripts, we were able to substantially reduce the error rate by 27% relative to the baseline on unseen spoken code.

Publication Title

Proceedings of the 5th International Conference on Conversational User Interfaces, CUI 2023

ISBN

9798400700149

Link to Full Text

Share

COinS