Keywords

1 Introduction

Many techniques (e.g., [4, 7, 8, 11, 12, 17]) give designers the chance to prototype touch-sensitive objects: objects with touch interaction including grasp; the capability of sensing touch force, including grasp force, will further enrich the vocabulary of touch interaction. Such capability would allow designers to prototype objects with a rich set of touch interactions. For example, designers can prototype a grasp-sensitive controller and test its usability, where the user must grasp it strongly in an appropriate position to use it.

Previously, we reported a technique to estimate touch force based on active acoustic sensing [13]. The technique applied our active acoustic sensing [12] and support vector regression (SVR) to estimate how an existing solid object is touched, along with the touch force, by simply attaching a vibration speaker and contact microphone to the object.

Fig. 1.
figure 1

Tool for sensing the grasp force.

Full size image

In this paper, we applied our active acoustic sensing to sense grasp force. This technique estimates the force without machine learning, and thus is simpler and more lightweight than our previous technique [13]. We also implemented a prototyping tool based on the technique (Fig. 1). With this tool, a designer can make an existing solid object grasp-sensitive by attaching a vibration speaker and contact microphone to the object by using machine learning, as we did in [12]. After learning, the tool shows how the object is grasped, along with the grasp force.

2 Related Work

Various techniques for building force-sensitive surfaces have been researched, especially for prototyping.

2.1 Circuit-Based Approaches

PrintSense developed by Gong et al. [4] is a sensing technique for planar, curved, and flexible surfaces; it allows for single-layer printing of sensors that can sense multi-touch input, hover, several levels of touch pressure, and deformation. Rosenberg and Perlin used a matrix of force-variable resistors to build UnMousePad [15], which is a pressure-sensitive flexible multi-touch touchpad; such a matrix had been used previously to build a thin flexible pressure sensor [10]. PyzoFlex by Rendel et al. [14] used a ferroelectric material to form a pressure-sensitive bendable surface, which can even sense hover. While these devices can perform a wide range of stable pressure-sensing on surfaces, the sensitive area is located on the devices themselves. By contrast, our technique makes the surface of an existing object grasp- and force-sensitive.

2.2 Vision-Based Approaches

A touch sensing technology [5] based on frustrated total internal reflection (FTIR), which is widely used for prototyping multi-touch surfaces, can be used to estimate the touch force (e.g., [1, 3, 6, 16]). RetroDepth by Kim et al. [9] used a stereo camera and retro-reflective surfaces to estimate the three-dimensional (3D) contours of interacting objects on and above the surfaces precisely; the precision is so high that subtle changes in the 3D locations of the fingertips can be identified when the user presses the surface of a malleable object; the locations can be used to estimate the touch pressure. In contrast to the above techniques, our acoustic-based technique can sense grasp force on existing solid objects.

2.3 Acoustic Approaches

A resonant pressure sensor [2] is a fluid pressure sensor consisting of the following three elements: a diaphragm with a variable self-resonant frequency characteristic, a drive transducer to vibrate the diaphragm, and a pickup transducer to capture the vibration. The sensor estimates pressure by exploiting the fact that the vibration frequency of the diaphragm depends directly on the pressure applied to the diaphragm. While our technique uses a similar sensing principle to make the surface of an object grasp- and force-sensitive, the technique also observes the steepness and power of the frequency response of the object to estimate the grasp force.

Our previous technique [13] estimates how an existing solid object is touched using support vector classification (SVC), as well as the touch force using SVR, through attachment of a vibration speaker and contact microphone to the object. While the technique we present in this paper uses the same hardware, it estimates the grasp force without machine learning, and thus is simpler and more lightweight than our previous technique.

3 Sensing Grasp Force Using Active Acoustic Sensing

Our technique estimates the grasp force based on the principle that the resonant frequency response of an object, which can be observed by vibrating the object and performing spectrum analysis of the signal captured from the attached microphone using fast Fourier transform (FFT), changes with the strength of the grasp.

Fig. 2.
figure 2

Test object: a ceramic bowl.

Full size image

To test this, we observed the changes in the resonant frequency response of a ceramic bowl, observing the responses when we did not grasp it, grasped it weakly, or grasped it strongly, as shown in Fig. 2.

In these observations, we used the same hardware as in [13]. As the two piezoelectric elements, we used a bimorph piezo-electric element (THRIVE K2512BP1, 25 mm \(\times \) 12 mm \(\times \) 0.23 mm) after cutting it into halves to reduce the footprint. Both elements were attached to the bowl by using double-sided adhesive bonding tape (3M SPG-12). The signal currents to and from the elements were amplified and sent to a computer (Apple MacBook Air, CPU: Intel Core i7 1.7 GHz, RAM: 8 GB) via a USB audio interface (Native Instruments Audio Kontrol 1). The computer plays sinusoid sweep signals repeatedly from 20 kHz to 40 kHz, whose frequency increases linearly in 20 ms, at a 96 kHz sampling rate through the speaker. In parallel, the computer also converts the signal captured by the microphone into the resonant frequency response using FFT.

Fig. 3.
figure 3

Changes in the resonant frequency response with grasp. (Color figure online)

Full size image

We found the following in the resonant frequency response (Fig. 3) when the grasp force increased:

  • The power of the response generally decreases.

  • The steepness of the response generally decreases.

  • The power decreases at some frequencies (e.g., Fig. 3A), while it increases at other frequencies (e.g., Fig. 3B).

  • The number of peaks decreases at some frequencies (e.g., Fig. 3C).

Therefore, we tested the following four metrics to estimate the grasp force:

  • The sum of the resonant frequency response (sum_af),

  • The variance of the resonant frequency response (var_af),

  • The sum of the spectrum of the resonant frequency response (sum_sp), and

  • The centroid of the spectrum of the resonant frequency response (g_sp).

Estimating the grasp force using one of the above metrics is simple; if the metrics decreases, we can conclude that the force should be strong.

4 Preliminary Experiment

We tested the four metrics on three objects: a knob, control lever, and plastic toy, as shown in Fig. 4. We used the same hardware as in the previous section. In this experiment, we grasped each object 20 times. For each grasp, we changed the grasp force according to the animated guide [13], which we used as the ground truth of the grasp force.

Fig. 4.
figure 4

Objects tested in our preliminary experiment.

Full size image
Fig. 5.
figure 5

Results of the preliminary experiment. (Color figure online)

Full size image

The results are illustrated in Fig. 5. In this figure, “guide” means the ground truth. We found that all of them performed well, since these metrics increase monotonically in accordance with the ground truth. Therefore, we used sum_af, which has the lowest calculation cost, to implement the tool shown in Fig. 1.

5 Conclusions and Future Work

We presented an active acoustic sensing technique for sensing grasp force. With this technique, using machine learning, an existing solid object can be made grasp-sensitive by attaching a vibration speaker and a contact microphone to the object. After learning, the technique estimates how the object is grasped, along with the grasp force. Our technique estimates the grasp force based on the change in the resonant frequency response of an object with the strength of the grasp: the steepness and power of the response generally decrease when the grasp force increases.

In future work, we plan to conduct a user study to examine how our technique will perform with multiple users. We also plan to increase the number of piezo-electric elements to examine how the performance of our technique increases.