Finger Orientation for Mobile Touch Input

Cinque Terre

Touch input on recent touch-based mobile devices is limited to single 2D coordinates of touches. This project is about extending the touch input vocabulary of mobile devices by two new dimensions – the pitch and the yaw angle of the finger touching the device. We first address the technical feasibility of determining the finger orientation on off-the-shelf touchscreens using convolutional neural networks (CNN). The CNN expects the raw measurements of the mutual capacitive touchscreen (also called a capacitive image) as input and outputs the estimated pitch and yaw angle with a mean absolute error (MAE) of 10.0° for pitch and 17.6° for yaw. To train this model, we collected a ground truth data set using a high-precision motion capture system (OptiTrack). The results were presented at the ACM International Conference on Interactive Surfaces and Spaces (ISS ’17) in Brighton, UK in October 2017.

With advances on the technical side, we further studied the ergonomic constraints of using the finger orientation as an input modality. The results help UI designers to develop inputs based on the finger orientation from an ergonomic perspective. The results were presented at the ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’18) in Barcelona, Spain in September 2018.

Models and Data Set

The models, data set, and Jupyter notebooks for the finger orientation estimation model are available on this project’s GitHub page.

Publications

PDF
Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks.
Sven Mayer, Huy Viet Le, and Niels Henze. 2017. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, Brighton, United Kingdom. 
@inproceedings{mayer2017estimating,
author = {Mayer, Sven and Le, Huy Viet and Henze, Niels},
title = {Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks},
booktitle = {Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces},
series = {ISS '17},
year = {2017},
isbn = {978-1-4503-4691-7},
location = {Brighton, United Kingdom},
pages = {220--229},
numpages = {10},
url = {http://doi.acm.org/10.1145/3132272.3134130},
doi = {10.1145/3132272.3134130},
acmid = {3134130},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {Finger orientation, capacitive sensing, mobile device, touchscreen},
}
PDF
Designing Finger Orientation Input for Mobile Touchscreens.
Sven Mayer, Huy Viet Le, and Niels Henze. 2018. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '18). ACM, Barcelona, Spain. 
@inproceedings{mayer2018designing,
author = {Mayer, Sven and Le, Huy Viet and Henze, Niels},
title = {Designing Finger Orientation Input for Mobile Touchscreens},
booktitle = {Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services},
series = {MobileHCI '18},
year = {2018},
isbn = {978-1-4503-5898-9},
location = {Barcelona, Spain},
pages = {29:1--29:9},
articleno = {29},
numpages = {9},
url = {http://doi.acm.org/10.1145/3229434.3229444},
doi = {10.1145/3229434.3229444},
acmid = {3229444},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {ergonomic zone, ergonomics, finger orientation, mobile, non-comfort zone, pitch, surface, touch, yaw},
}