True wireless stereo (TWS) active noise cancelling (ANC) wearables (headphones and earbuds) have been more popular in recent years; in 2023, the number of shipments worldwide of these devices will almost exceed that of smart wristbands and watches. Recent developments in artificial intelligence, transparency mode, and ANC have resulted in a large extension of the on-head time for wearables. Wearables are widely used by users for activities such as exercise, concentration, and mood adjustment in addition to listening to music. Still, for the consumer market, hearable health remains mostly unexplored.
Presenting at MobiCom 2023, we present “APG: Audio plethysmography for Cardiac Monitoring in Hearables,” a revolutionary approach to active in-ear health monitoring.
 Without requiring additional sensors or sacrificing battery life, audio plethysmography (APG) allows ANC wearables to track a user’s physiological data, including heart rate and heart rate variability. APG is very resistant to motion artefacts, complies with safety standards by maintaining an 80 dB margin below the limit, is not impacted by seal conditions, and works with all skin tones.
In the ear canal, health-sensing
The arteria auricularis profunda, or deep ear artery, is the source of blood flow to the auditory canal. The auditory canal is deeply penetrated by this artery, which produces a complex network of smaller arteries. An excellent place to sense health is the ear canal because minute changes in blood vessel form brought on by pulse and blood pressure can result in minute differences in the volume and pressure of the canals.
To allow health applications including blood pressure tracking, heart rate monitoring, and sleep monitoring, recent research has investigated the use of wearables for health sensing. This has included combining a microcontroller with a variety of sensors, such as photoplethysmograms (PPG) and electrocardiograms (ECG). However, this sensor mounting paradigm unavoidably increases hearables’ price, weight, power consumption, acoustic design complexity, and form factor issues, which pose a significant obstacle to their widespread acceptance.
The feedback and feedforward microphones used by current ANC wearables are used to navigate the ANC feature. Because these mics can record or detect a wide range of bio-signals both within and outside the ear canal, they open up new possibilities for a variety of sensing applications. For instance, feedforward microphones can pick up respirations, while feedback microphones can pick up heartbeats.
Many mobile applications, including respiration monitoring, body activity identification, heart rate monitoring, and ear condition diagnosis, have been spurred by academic research on this passive sensing paradigm. On the other hand, high-pass filters are included in the microphones of consumer-grade ANC headphones to stop saturation caused by vigorous wind noise or body movements.Â
Earbud seal characteristics have a major impact on the quality of the signal during passive listening in the ear canal. Therefore, with market ANC headphones, it is difficult to include health features that depend on passively listening to low-frequency noises (≤ 50 Hz).
tiny physiological signals being measured
APG works around the previously described hardware limitations of ANC headphones by passing a low-power ultrasonic probing signal via the speakers of an ANC headset. On-board feedback microphones pick up the echoes that are caused by this signal. We find that these ultrasound echoes are modulated by the slight skin displacement in the ear canal and the vibrations of the pulse.
We construct a cylindrical resonance model to comprehend the fundamental physics of APG. Since this occurrence occurs on a very tiny scale, the raw pulse signal in the raw ultrasound received is imperceptible. To recover this microphysiological modulation underneath the noise floor, we use coherent detection (we refer to this restored signal as the mixed-down signal; refer to the publication for additional information). With more noticeable dichroic notches, the final APG waveform offers a better perspective of cardiac activity than the PPG waveform (i.e., pressure waveforms that provide rich insights into the central artery system, such as blood pressure).
Practical use of APG sensing
We found in our early testing that APG functions reliably even when there is music playing and faulty earbud seals. But we also observed that the APG signal might be severely disrupted by body motion and could be quite loud. By then, we realized that APG needed to be more robust to compete with almost 80 years of PPG research before it could be considered relevant.
PPGs are quite sophisticated and frequently utilized, although they do have several drawbacks. For instance, two to four diodes are usually used by PPGs sensors to transmit and receive light frequencies in order to sense. However, it is challenging for a single diode to transmit numerous colors at different frequencies because of the super high-frequency (hundreds of Terahertz) nature of the light.Â
However, it is simple to create a low-power, low-cost device that can produce and receive more than 10 audio tones (or frequencies). To enable a higher quality APG signal and motion resilience, we take advantage of channel diversity, a physical phenomenon that describes how wireless signals (like light and audio) at different frequencies have different characters (like attenuation and reflection coefficients) when the signal propagates in a medium.
Next, we show through experimentation that using several frequencies in APG signalling is successful. We simultaneously broadcast three probe signals, each of which has a frequency that ranges uniformly from 30 KHz to 32 KHz. Throughout the experiment, a participant was instructed to shake their head four times to induce interference. One special benefit of APG is coherent detection, which allows for the simultaneous transmission of many frequencies in order to collect a variety of data. This is seen in the image below.
The four head motions are displayed in the 30 kHz phase, while the pulse wave signal is displayed in the 31 kHz magnitude (amplitude). This finding suggests that whereas certain ultrasonic frequencies may be motion-sensitive, others may be responsive to heart activity. As a result, we may utilize the multi-tone APG as a calibration signal to determine the ideal frequency for measuring heart rate and then use that frequency exclusively to obtain a pulse waveform with excellent quality.
Accurate detection of heart rate variability is made possible by the prominent dicrotic notches on the APG pulse waveform, which is more noticeable after determining the optimal frequency for measuring heart rate.
Multiple simultaneous observations, or multi-tone, allow for the creation of array signal processing methods. We show the spectrogram of an ongoing session APG experiments both with and without the use of blind source separation (for further information, refer to the study). Using a Polar ECG chest strap, we also demonstrate the ground truth heart rate measurement in the ongoing experiment.Â
The running cadence (about 3.3 Hz) and two faint lines (about 2 Hz and 4 Hz) that represent the user’s heart rate frequency and its harmonics are visible in the raw APG. Following blind source separation, the heart rate frequencies exhibit a considerable increase in signal-to-noise ratio (SNR), matching the ground truth heart rate frequencies. The heart rate and running cadence derived from the APG and ECG are also displayed. It is evident that the APG precisely monitors the increase in heart rate during the jogging exercise.
Field research and final reflections
We studied user experience (UX) in two rounds with 153 participants. Our findings show that heart rate variability (2.70% median error in inter-beat interval) and heart rate (3.21% median error across individuals in all exercise situations) are consistently measured by APG. Our research indicates that APG is resistant to changes in skin tone, suboptimal seal conditions, and ear canal size, in contrast to PPG, which performs differently across skin tones. The document contains assessments that are more in-depth.
APG operates reliably across a range of user actions, converting any TWS ANC headphones with a straightforward software update into smart sensing headphones. Playing music has no effect on the detecting carrier signal; it is absolutely inaudible. Above all, APG opens up new avenues for low-cost health monitoring and provides fresh insights into biomedical and mobile research.
