1 Introduction

In recent years, it has become possible to use information terminal at many locations under various conditions due to spread of smartphone. The contributing factors include improvement in portability by miniaturization and operability in touch panel. Along with spread of large-size touch panel for liquid crystal display, not only small-size information terminals such as smartphone but also those with large size such as large-screen tablet terminals have achieved widespread use. Tablet terminal is excellent in its large screen capable of handling a lot of information at a time. However, it is difficult to operate a large and heavy tablet terminal just in one hand. It is also difficult to use a tablet terminal “in a state that one hand is detained” by other purposes such as to hang on a strap and hold a baggage. Conventional operating method has a problem that use of large size information terminal is restricted. In addition, it is impossible to operate such tablets either by one hand or both hands when the hands get dirty by cooking or they are gloved due to cold weather.

In this study, we propose an interaction for operating objects displayed on a screen by blowing a breath on a screen of information terminal. It is aimed to make it possible to operate even “in a state that hands are detained by other purposes” by allocating “various breaths” to each operation of a tablet terminal with a large screen. Moreover, availability of interaction based on a breath is evaluated by implementing applications compliant with the interaction in a tablet terminal.

2 Related Studies

Several studies have been made for operation of information terminal using something other than fingers. For example, Zarek et al. have proposed SNOUT for operating smartphone with the nose [1]. Since it is difficult to expect all application developers to design a screen exclusively for the nose, they have resolved the problem by virtually superposing layers with selection determination area for nose interaction on the screen in order for operation by the nose to become easier. They have described as a result of accuracy verification of target selection by the nose that it is possible to select objects by the nose as long as the determination area is about one-and-a-half times as large as object size displayed on the screen. However, they have also explained that eyestrain is caused due to the close distance between the eyes and the screen.

There is also a study on operation using tilt and movement of information terminal. Some system operates a selection pointer (like a mouse cursor) in such a way to roll a virtual ball on the screen by detecting a tilt of information terminal with an acceleration or gyroscope sensor [2]. Such an interaction has been mounted that performs cancelling operation by a gesture to shake the terminal (shake) [3,4,5]. There are also systems other than that to perform various operations by gestures by a hand holding an information terminal.

Even though there are some previous studies on a device to operate using a breath, the purpose and requirements of the device are different. It has been conventionally studied on expiration (inspiration) input device to operate with a breath. Many of their purposes aim at providing input operation support for severely disabled people with physical inability and entertainment system under VR and MR environment. A device called as expiration (inspiration) switch has been put into practice in a study aiming at facilitating input operation by severely disabled people. Expiration switch has made it possible for users to perform input operation with the switch by blowing (inhaling) a breath from a tube in their mouth. Information to be input by this switch is limited to ON-OFF binary information, however, it is not able to make operation of PC easy.

Kitayama and Nakagawa have proposed an expiration mouse by further developing the expiration switch [6]. Different from conventional expiration switch, the expiration mouse has realized a way of operation by a mouse cursor by utilizing strong/weak breath. Because strong/weak expiration and inspiration, i.e. four elements in total, are allocated to transfer in upper/lower and left/right directions, however, it is impossible to directly select objects to be operated.

In relation to a contact type device with a tube to be held in the user’s mouth, Kume and Dong have pointed out problems on restriction of the mouth as well as a sanitary risk [7]. Therefore, they have proposed a non-contact type device equipped with a load cell and detected a breath from wind pressure using the load cell. In case of non-contact type, however, it is required for a user to blow a breath at a sensor installed on the device.

In a study conducted by Iga et al., expiration and inspiration was detected in a state of non-contact using audio signals of a microphone mounted on a headset [8]. In addition, performing a pointing operation depending on a direction of the head part using a magnetic sensor attached on the head set, the operation using a breath was reflected on an object targeted by the user. Since the device is a wearable type, it has become unnecessary to target at the sensor part every time when blowing a breath but users have to wear it for use. Further, a direction to which a breath is actually blown may shift from a position of the targeted object to be operated because the pointing operation is performed by changing a direction of the head.

3 Our Goal

In this study, we propose a system in which a user blow a breath on the actual object to be operated without wearing and contacting the device. While interaction based on breath has been proposed by conventional studies, breath is used just for generating on-off binary elements and has no direct relationship with objects to be operated. We would like to propose a framework to operate by directly blowing a breath on illustrations and letters displayed on a touch panel just like a way we directly touch them with a finger. Therefore, we propose a system to identify position of breath by multiple sensors mounted on a tablet terminal and implement and assess the system. Then, we create an application for a tablet terminal which is operated with a breath and discuss the practical utility based on the assessment.

4 Breath Interaction

4.1 Properties of Breath

As breath is a common interaction used also for daily living, people take actions using a breath such as to “blow dusts away” and “warm hands”. It is believed to be possible to make users imagine operating method intended by a designer without an instruction manual by showing them metaphors which resemble such interaction based on breath. Breathing is a motion that is required for maintenance of life and can be controlled by human as well. Therefore, it is believed that problems on information terminal operation restricted by situation and environment may be settled by implementing operation by using breath which is always available for human.

As interaction of common information terminals is represented by direct operation, operation and its object correspond in a one-on-one manner in many of them. According to a study by Iga et al. [8], breath has a characteristics that one operation is performed for multiple objects in a one-to-many manner such as an action “to blow out multiple candles”.

4.2 Elements of Breath

In order to allocate various types of breaths to multiple operations, how many types of breaths exist is considered at first. In Table 1, actions of human to “blow” a breath are classified. In case of playing wind instruments, sound is expressed differently by blowing a breath strongly/weakly for a long/short period of time with the mouth contacting with the instruments. In case of blowing a candle out, breath is blown strongly for a short period of time without contacting the mouth with the candle. In case of just flickering the candle, breath is blown weakly for rather a long period of time. Position for blowing a breath is important when there are multiple candles. Thus, breath is blown aiming at objects to be operated by selecting number of objects and range, and duration of the effect to be maintained may vary depending on the length of blowing of a breath. Therefore, we have determined it possible to achieve non-contact breath operation by detecting position, strength, and length of a breath.

Table 1. Types of breaths.
Full size table

Then, we would like to examine an action to “inhale” a breath. It is a common action to take in something with the mouth contacting with, such as an action to take in beverages using a straw. On the other hand, as there is no example to “take in something targeted” without contacting with the mouth, operation of inhaling has not been mounted in the current non-contact type device. Measuring position, strength, and length by a sensor when a breath is blown, multiple operations with a breath are implemented.

5 Proposed System

5.1 Overview of System

Overview of a system used in this study is shown in a Fig. 1. Capacitor microphones were arranged as a sensor evenly at 12 points of the end part of a tablet terminal screen. Breath is identified by using values of microphones. Based on identification of presence/absence state of blown breath, position, strength and length of a breath is identified if the breath was present.

Fig. 1.
figure 1

Overview of a tablet system

Full size image

5.2 System Configuration

The system was implemented in a tablet terminal Acer ICONIA TAB A500. A liquid crystal display of the tablet terminal is 13.6 cm long and 21.7 cm wide and capacitor microphones were arranged around and parallel to the screen. The capacitor microphones are placed facing toward center of the screen. Arduino, a microcomputer board, controls the 12 capacitor microphones. By sampling of microphones at 180 Hz, values obtained from the microphones are forwarded to a tablet terminal connected in series with Arduino. The tablet terminal identifies position, strength, and length from values obtained.

5.3 Breath Detection Treatment

How to treat detection is shown below from presence/absence, length and strength to position of breath in order.

Presence/absence of Breath

Presence/absence of a breath is treated by each microphone. Based on values obtained from each microphone, presence/absence of a breath is determined using values measured 30 times (approx. 0.17 s) in the past from the time of determination. It was determined that “breath was present” if number of microphone’s values exceeding ± 0.49 V from the value of 2.5 V at the time of no sound is 10 or larger out of 30. If breath was determined to be present by any one of the 12 microphones, breath is regarded to be present in the whole system.

Length

Length of a breath is obtained by counting them up if “breath was determined to be present” for more than one microphones. Length of a breath is counted out by continuously counting them up until “breath is determined to be absent”.

Strength

Strength of a breath in each microphone is obtained by integrating amount of change during the past 30 times of measurements. In other words, absolute values of differences between 2.5 V at the time of no sound and measured value are added. The whole strength of a breath is obtained by adding up strengths detected by all microphones after calculating strength detected by each microphone.

Assuming that strength detected by each microphone is Mi (i = 1 to 12), strength detected by microphone 1 is obtained by formula (1).

$$ M_{i} = \int_{ - 30}^{0} {\left| {Strength \,Value - 2.5V} \right|} $$
(1)

Strength of the whole breath BS is obtained by formula (2).

$$ BS = \sum\nolimits_{i = 1}^{12} {M_{i} } $$
(2)

Strengths detected by all microphones are treated to be added because the stronger is a breath the larger is the change in value observed in multiple microphones.

Position

In order to calculate coordinate of a breath, a coordinate was allocated virtually to microphones as shown in the Fig. 2. Assuming strength of a breath detected by each microphone Mi as a weight, position for a breath to be blown is identified by measuring gravity of a breath. In such a case, values measured by each microphone 30 times in the past are used and treatment is performed such that the newer is the value the stronger is its impact on the result.

Fig. 2.
figure 2

The position of microphone

Full size image

Coordinate of a breath B (Bx, By) is obtained by formulas (3) and (4). Possible values of Bx and By range from 1 to 5 and from 1 to 3, respectively.

$$ B_{x} = \frac{{\mathop \sum \nolimits_{i = 1}^{12} \left( {P_{ix} \times M_{i} } \right)}}{{\mathop \sum \nolimits_{i = 1}^{12} M_{i} }} $$
(3)
$$ B_{y} = \frac{{\mathop \sum \nolimits_{i = 1}^{12} \left( {P_{iy} \times M_{i} } \right)}}{{\mathop \sum \nolimits_{i = 1}^{12} M_{i} }} $$
(4)

Here, coordinate of a breath T (Tx, Ty) on the screen is obtained by formulas (5) and (6) using horizontal and vertical resolutions 1280 px and 752 px of the screen of tablet terminal to be implemented.

$$ T_{x} = \frac{{B_{x} }}{5} \times 1280 $$
(5)
$$ T_{y} = \frac{{B_{y} }}{3} \times 752 $$
(6)

6 Assessment Experiment of Breath Interaction

6.1 Objective

Our objective is to detect coordinate, strength, and length of a breath blown by human. Detection accuracy is believed to differ depending on several elements such as deviation from the target caused by inaccurate breathing by human and the way of blowing a breath. Using a device and algorism with a setting that human blows a breath, detection accuracy is measured for position, strength, and length of a breath.

6.2 Objects to Be Measured and Experiment Method

Subjects were 7 male and female university students. They were supposed to hold a tablet terminal with both hands and blow a breath on the tablet terminal. They were instructed how to hold and tilt the tablet terminal as well as to blow their breath with a reasonable method. Distance between the screen of tablet terminal and their face was measured during the experiment. As experiment is performed separately for maximum length, blowing strength, and coordinate accuracy of a breath, experimental method is explained below one by one.

Assessment Experiment of Detection of Breath Position

Subjects blow their breath aiming at the center of target displayed on the screen. Upon detection of a breath by the system, position of the target is changed in a random manner. The procedure was determined to be tried 200 times.

Assessment Experiment of Detection of Breath Strength

Subjects were asked to blow their breath by each of five grades (weakest-weak-medium-strong-strongest), respectively. They repeat to blow their breath by a strength as indicated on the screen. The system does not feedback the strength of a breath blown by the subjects. The procedure was determined to be tried 200 times.

Assessment Experiment of Detection of Breath Length

Instructing subjects to blow a breath as long as possible, how long they are able to blow a breath is measured. Since subjects are required to continuously blow their breath strongly enough for the system to response, whether the system has continuously detected the breath was displayed on the screen. The procedure was determined to be tried 10 times.

6.3 Results

Distance between the tablet terminal screen and their face during blowing a breath was 15 to 20 cm which was closer than that of ordinary use. Any of the subjects blew a breath by bringing the tablet terminal closer to their face. No large difference was observed in the distance between the tablet terminal screen and face of subjects.

Coordinate

As a result of tallying breath detection errors of seven subjects, average value and standard deviation was 2.52 cm and 1.56, respectively. Since the values of detection errors are normally distributed according to a normal probability paper, detection error to obtain 95% of confidence interval is required to tolerate 5.26 cm. Therefore, we design a system in this study based on an assumption that detection error of 5.26 cm by radius from center of target is generated by operation of a breath to be implemented.

Strength

Subjects were asked to blow their breath by each strength of five grades. The Fig. 3 shows a result of a subject who were able to most successfully blow a breath by selected strength. It has been determined to be difficult for the system to achieve confidence level of 95% for selective blowing at five grades. It was possible for all subjects to selectively blow with a confidence level of 95% by two grades (weakest and strongest). Therefore, interaction is implemented with two grades of breath strength in the system.

Fig. 3.
figure 3

Result of blow a breath by each strength of five grades.

Full size image

Length

Average length of longest breath is shown by subject in the Table 2. As it is preferable that many users operate breathing without difficulty in the interaction to be proposed in the study, the shortest result of 2.6 s is set as a condition to determine “long breath”.

Table 2. Average length of longest breath.
Full size table

6.4 Types of Breath Used for Interaction

Position (coordinate), strength, and length of a breath are implemented by the conditions below which were ever obtained:

  1. 1.

    Tolerate detection error of 5.26 cm by radius from a center of icon and button to be operated.

  2. 2.

    Breath strength: Selectively blow by two grades of weak and strong.

  3. 3.

    Breath length: Determined to be “short breath” if it is 2.6 s or less and “long breath” if it exceeds 2.6 s.

7 Demonstration Experiment Using Application

In order to mount an interaction using a breath in a tablet terminal, we have developed three applications dedicated for breath operation. Qualitative assessment is performed for operability using each application.

7.1 Home Screen (Launcher Application)

A launcher application was implemented by using metaphors of a windmill to be rotated by wind and a candle flame to be put out by strong wind (see Fig. 4). A screen layout with icons arranged annularly was adopted. The icons arranged annularly rotate clockwise or counterclockwise by selectively blow a breath toward left or right. It is possible to select (activate) an icon in the direction of 6 o’clock (lower side of screen) by blowing a breath strongly.

Fig. 4.
figure 4

Home screen by using metaphors of a windmill to be rotated by wind.

Full size image

7.2 Browsing of Electronic Books

An electronic book browsing application was implemented by using a metaphor of fallen leaves blown away by wind (see Fig. 5). For screen layout to browse books, two modes, i.e. reading mode and thumb nail mode, were prepared. In the reading mode, pages of a book are displayed on the left and right side of the screen in a double-spread state. Thumb nail mode is capable of displaying many pages of a book. It is possible to turn pages just like blowing them away by blowing a breath in the left or right direction. The two modes are switched by blowing a breath strongly.

Fig. 5.
figure 5

Reading mode in electronic books.

Full size image

7.3 Map Application and Web Browser

Map and web browser applications were implemented by using a metaphor of fallen leaves blown away by wind (see Fig. 6). Scrolling is performed in such a way that coordinate is transferred at the center of screen by blowing a breath on it. In addition, it is possible to zoom out just like blowing away by blowing a breath strongly as well as to zoom in by blowing a breath longer.

Fig. 6.
figure 6

Map application.

Full size image

7.4 Experiments

Experiments were performed without notifying subjects what types of interaction was available. We have observed how and by what types of breathing the subjects tried to obtain operation results. In order to assess whether metaphors used for the applications are available, we asked subjects to imagine and explain how the applications worked by blowing a breath just by showing them the metaphors of windmill, candle and fallen leaves. Then, we asked them to imagine and explain how operations are performed with breathing by showing them screens of home selection, electronic books, maps and browser. Finally, asking them to operate with breathing, we asked them to make remarks about any part with something different they felt from operations they had imagined.

7.5 Results

At first, subjects tried to express “tap”, “double tap” and “flick” by a breath while being unable to notice the new interaction based on the assumption of breath. In the process, however, they have learned the various metaphors based on a breath immediately. It has been revealed that they are able to use the tablet terminal freely by the interaction using a breath once they learned the way of operation.

Subjects often used position and strength of a breath but did not use a breath length for the operation. They described they thought long breath as a measure just to blow on an object at a distance or to continue such actions that were activated by a short breath and did not think that another interaction was assigned to a long breath.

Subjects felt something strange for the fact that there was no operation of “inhaling a breath”. They recognize breath as a set of actions to “blow” and “inhale” regarding “inhaling” as a reverse operation of “blowing”. Therefore, it is required to review a set of operations to “blow” and “inhale” in constructing an interaction based on a breath in the future.

We have received as one of characteristic opinions from the subjects that they feel existence of themselves when operating breath interaction compared with a case of touch interaction. There was also an opinion about a metaphor of candle that they had a strong awareness of relationship of subjects themselves blowing a breath with a candle as an object to blow a breath. They had another opinion about a screen of map that they had a sense to have looked down a geography from above the sky as if they themselves were a satellite. Therefore, it is believed that breath interaction may provide users with physicality in addition to a sense of unity as if they stay in the same space as the operation object.

8 Conclusion

In this study, we have proposed and assessed a device to operate by blowing a breath and algorism. It has been proved it possible to detect position, strength, and length of a breath in a non-contact and non-wearing state by a device using microphones. It has been also suggested to be possible to realize operations with a breath by blowing a breath on the object of operation itself displayed on a screen.

In consideration of the fact that subjects imagined operation of both “blowing” and “inhaling” first, it is required for us to consider to mount “inhaling” operation which is imagined by subjects as a reverse operation of “blowing” while detecting “inhaling”.

Since it was implemented in a tablet terminal, we received many opinions from subjects compared with touch interaction. Therefore, it is also required to compare with touch interaction in difference of operational feeling and possibility to operate without being limited by conditions and environment.