Portable auricular device for real-time swallow and chew detection

1 Introduction

Food intake monitoring is necessary to prevent critical conditions like obesity or malnutrition. In Germany 12.4% of senior citizens, living in a nursing home, are suffering from exsiccosis, the resulting condition of dehydration [1]. One approach to draw conclusions of a person’s food and fluid intake is to monitor swallow and chew events.

Currently used systems have high computational and/or hardware costs. A real-time MRI or a videofluoroscopy are examples of very expensive methods that are not suitable for long time measurements [2], [3]. Other systems that work with acceleration sensors [4] or a combination of EMG and bioimpedance sensors [5] are principally portable, but special preparations are needed. One preparation is the attachment of sensors and electrodes to the throat of a subject, which increases the risk of injury.

Best results, when automatically detecting swallow and chew events, are provided by recording the electrical activity of skeletal muscles (EMG) [5], [6] or by recording the acoustic signal they produce [7].

Due to the complexity of these methods, they are not suitable for longtime studies. Thus, studies with the aim of providing a conclusive result on the nutritional status of a person are usually done by using questionnaires which either are filled out by the subject or by the nurse.

The proposed system, composed of both, hardware and software, is capable of recording air pressure signals in the acoustic meatus and to detect swallow and chew events in real-time. The hardware costs are kept low as well as the computational intensity.

2 Material and methods

The device used to record the pressure signal consists of a miniature pressure sensor the size of 2 × 2.5 mm² (Bosch Sensortec BMP280) and a low-power microcontroller platform. The sensor is integrated into a silicon in-ear plug (Figure 1) which is inserted into the acoustic meatus. The plug seals off the acoustic meatus from the outside environment and thus allows the sensor to detect changes in the in-ear air pressure.

Figure 1:

Air pressure sensor integrated in a silicone in-ear plug.

The signal is recorded and processed by a microcontroller and can be saved on a SD-Card. One advantage of a low-power microcontroller concept is that it can be integrated into a behind-the-ear hearing aid to provide further information about the subject.

To automatically detect swallow and chew phases/events, a binary classifier has been trained by means of manually annotated data sets. To accumulate the needed data, several experiments were performed with subjects swallowing or chewing a food bolus. Using a hardware trigger connected to the microcontroller a parallel signal has been recorded, allowing to mark specific areas of the pressure signal. Relevant features have been extracted using these data sets.

To face challenges of interfering signals, specific constraints have been introduced. The detection of swallowing events is done in a controlled experiment of a subject drinking 50 ml of water out of a cup while sitting. The only prerequisite when testing the chew event was that the subject had to sit still, while still allowing normal movement of jaw and limbs.

2.1 Swallow detection

Two features were used to detect swallowing phases (^IF_S and ^IIF_S). The first feature represents the time the signal p(t), which is bias cleared and filtered, drops under a certain threshold limit. This characteristic drop occurs when the liquid is put into the mouth.

(1)FSI(t)=∑i=0L|t⁢hkS(t+i)−1|

The classifier ^thk_S(t) (eq. 2) indicates if the pressure signal p(t) drops under the threshold limit T . Equation (1) provides a function which is equivalent to zero if the drop under the threshold limit is of length L.

(2)kSt⁢h⁢(t)={1 if p⁢(t) < T0 else

The first classifier equation can be formed with ^IF_S(t) as the first feature, and ^IC_S its threshold limit (eq. 3).

(3)kSI⁢(t)={1 if  IFS(t) < ICS0 else

For the second feature the power of the delta function ^IIF_S(t) = (Δp(t))² is used. Its classifier (eq. 4) indicates when the change in the pressure signal p(t) is greater than a certain threshold limit ^IIC_S. This peak in ^IIF_S(t) represents the quick change in the pressure signal which occurs when a subject swallows the food bolus.

(4)I⁢IkS(t)={1 if I⁢IFS(t) > I⁢ICS0 else

The final classifier k_S (eq. 5) is a result of the combination of both classifiers. A shift S is added to the second classifier, as the characteristic peak in its feature occurs later in comparison to the peak of the first feature.

(5)kS(t) = IkS(t) ∧ I⁢IkS(t+S)

The top image of Figure 2 shows five distinctive swallow events with the external triggered events (manual annotation, red asterisk) and the automatically detected swallowing phases (green asterisk). The bottom image shows the simultaneously recorded EMG signal. It is apparent that the phase labeled ‘swallowing phase’, coincides with the corresponding peaks in the EMG signal. Only the last swallow phase is missing from the data.

Figure 2:

Top: Pressure signal p(t) of five swallow events with manually annotated triggers (red asterisk), detected events (green asterisk). Bottom: Simultaneously recorded EMG signal. EMG electrodes attached on the larynx.

2.2 Chew detection

Identifying the presence of a certain frequency domain in the pressure signal is the approach used to detect chewing phases. In the relevant domain of about 1 to 2.5 Hz, a significant shift in the amplitude spectrum can be observed. Figure 3 shows the spectra of the pressure signal p(t). On the left, the pressure signal of a person in a resting position can be viewed. On the right the signal of a person chewing is shown. To obtain a computable feature, the Fourier transformation ℬⁱ of the time phase bⁱ is integrated over the relevant frequency domain and used as feature F_C (eq. 6). The vector bⁱ is defined as a time phase (or block) of p(t) with a certain length and overlapping.

Figure 3:

Single-Sided amplitude spectrum of the pressure signal p(t). Left: Resting situation, Right: Chewing situation.

(6)FC⁢(i)=∫f=12.5⁢Hzℬi⁢(f)

The classifer (eq. 7) decides if a phase of p(t) is member of the class ‘chewing’ or not. The top image of Figure 4 shows the pressure signal p(t) of a subject chewing rusk (hard bread) with the classifier k_C. The bottom image shows the simultaneously recorded EMG signal and the external trigger. The block length is set to 6.25 s and the overlapping to 3 s. These values were the result of finding a reasonable compromise between computational costs and sensitivity of the Fourier transformation to changes in the frequency domain.

The activity of the EMG signal, which coincides with the external trigger, shows when the relevant chewing muscles are active.

(7)kC⁢(i)={1 if FC⁢(i) > CC0 else

Figure 4:

Chewing hard bread. Top: Pressure signal p(t) with detected chew phases. Bottom: Simultaneously recorded EMG signal with external trigger. EMG attached at relevant chewing muscles.

3 Evaluation

To evaluate the proposed system, consisting of both hardware and detection methods, it was tested with three different subjects (age 22–29 years, 2 femals, 1 male). For this purpose four tests were designed. Special attention was given to the design of the chewing tests to cover a wide scope of food bolus textures.

Test procedures:

1. Drinking, 5 × 50 ml

2. Chewing, apple

3. Chewing, hard bread (rusk)

4. Chewing, chewing-gum

To evaluate the methods, the tests were performed with the subjects while simultaneously marking the pressure signal with the external trigger. Afterwards the results of the automatically detected swallow and chew phases were compared with the annotations made by the external trigger.

A specificity of 65% and a sensitivity of 64% were achieved for all swallow events. When regarding the chewing tests, a specificity of 28% and a sensitivity of 88% were reached. The low percentage for specificity is due to unintentional jaw movements during phases of non-chewing, which the classifier recognized as chewing.

Measurements for one subject were unsuccessful. It is assumed that the cause of this was the divergent anatomy of the subject’s external meatus. Possibly the in-ear plug did not fit well into the ear and did not provide an airtight sealing to the environment. Another cause could be that the sensor was out of place or obstructed, not allowing the sensor to work properly.

In addition to the test results, the validity of the proposed system is evident when comparing the results of the classifiers with the simultaneously recorded EMG signal.

4 Conclusion

A novel biosignal measurement system has been proposed. The air pressure signal in the acoustic meatus was used to detect swallow and chew events in real-time using simple pattern detection algorithms.

Direct comparison with the de facto gold standard (EMG), and tests with subjects both delivered good results using the purposed method and apparatus.

Swallow and chew events were detected properly while keeping computational and hardware costs very low. The proposed hardware setup can easily be integrated into a portable hearing aid device. The computational costs of the algorithms used are kept low enough to run autonomously on a portable system.

In order to achieve higher validation with respect to a subject’s divergent anatomy and behavior, the proposed system must be tested with more subjects. The methods have to be improved to handle artifacts without the above-mentioned constraints.