Predicting Tap Locations on Touch Screens in the Field Using Accelerometer and Gyroscope Sensor Readings
Published in International Conference on Human-Computer Interaction, 2020
Schmitt, E. & Voigt-Antons, J.-N.
Research has shown that the location of touch screen taps on modern smartphones and tablet computers can be identified based on sensor recordings from the device’s accelerometer and gyroscope. This security threat implies that an attacker could launch a background process on the mobile device and send the motion sensor readings to a third party vendor for further analysis. Even though the location inference is a non-trivial task requiring machine learning algorithms in order to predict the tap location, previous research was able to show that PINs and passwords of users could be successfully obtained. However, as the tap location inference was only shown for taps generated in a controlled setting not reflecting the environment users naturally engage with their smartphones, the attempts in this paper bridge this gap. We propose TapSensing, a data acquisition system designed to collect touch screen tap event information with corresponding accelerometer and gyroscope readings. Having performed a data acquisition study with 27 participants and 3 different iPhone models, a total of 25,000 labeled taps could be acquired from a laboratory and field environment enabling a direct comparison of both settings. The overall findings show that tap location inference is generally possible for data acquired in the field, hence, with a performance reduction of approximately 20% when comparing both environments. As the tap inference has therefore been shown for a more realistic data set, this work shows that smartphone motion sensors could potentially be used to comprise the user’s privacy in any surrounding user’s interact with the devices.
Recommended citation: Schmitt, E. & Voigt-Antons, J.-N. (2020, July). Predicting tap locations on touch screens in the field using accelerometer and gyroscope sensor readings. Paper presented at the International Conference on Human-Computer Interaction (pp. 1-10). Springer, Cham. https://doi.org/10.1007/978-3-030-50309-3_43