Hitachi Solutions for New Work Styles
A Simple Check on People’s State of Mental Health from Their Voice
According to a statement by Japan’s Ministry of Health, Labour and Welfare, the number of people suffering from depression and other mood disorders (emotional disorders) has reached approximately 1.12 million, and the loss per person unable to work due to mental illness has been estimated at 4.22 million yen for each absence. While companies have taken secondary or tertiary prevention measures to deal with poor mental health, they have been slower to engage in primary prevention despite this being seen as the most effective approach. In June 2017, Hitachi Systems, Ltd. launched the mind monitoring service , a cloud service that provides an easily accessible means of mental illness self-assessment and prevention that is powered by MIMOSYS, a technology that measures the mental health condition of a person based on voice. This service is intended as a means for the primary prevention of mental illness (preventing such illness before it manifests). It uses changes in people’s voice for the early detection and prevention of mental illness to provide self-maintenance (helping users to keep themselves healthy), also helping organizations to achieve work style reform.
In Japan, health management has become a societal issue in recent years with major changes in the environment surrounding the medical and healthcare sector due to factors such as the aging population and the rising prevalence of lifestyle diseases and mental illness. The number of people suffering from depression and other mood disorders (emotional disorders) has increased significantly (by 2.5 times) compared to 15 years ago, having reached approximately 1.12 million people according to a statement in December 2015 by Japan's Ministry of Health, Labour and Welfare(1), and is giving rise to economic losses such as people taking time off work.
In the corporate sector, it has been reported that, from among workplaces with more than 1,000 employees, about 90% had instances during the past year of staff resigning or being off work for more than a month at a time due to mental health issues. Although secondary or tertiary prevention measures are often taken to deal with poor mental health, the adoption of primary prevention has been slow despite this being seen as the most effective approach.
The main aims of primary, secondary, and tertiary prevention measures for mental health are as follows.
Depression and other mental illnesses are a major health and safety challenge for people of working age. As it is said that one in five elderly people receiving outpatient treatment for dementia also suffers from depressive disorders, there is an urgent need for action to help with dementia prevention among older generations.
The stress check system was launched in December 2015, obliging all workplaces with 50 or more employees to conduct “stress checks for grasping of the levels of psychological burdens of the laborer.” Unfortunately, because these checks use written questionnaires, there is ambiguity about what respondents are and are not conscious of, and they are subject to reporting biases such as respondents underrating their symptoms.
Hitachi Systems, Ltd. already supplies healthcare services that support health management and disease prevention in the form of a lifelog analysis service that provides information on people's lifestyles and health by analyzing and quantifying aspects of their daily lives, such as the quantity and quality of exercise or sleep, and also the fatigue and stress measurement system that measures fatigue levels using electrocardiograms and pulse waveform measurement (see Figure 1).
The lifelog analysis service uses a special-purpose wrist-watch sensor*1 to collect measurements and the fatigue and stress measurement system uses a special-purpose autonomic nerve sensor*2.
Along with this, Hitachi Systems decided to investigate a cloud service that provides an easy way to assess trends in people's mental health without the need for specialist equipment and seeks to facilitate the taking of action before psychological and physical changes occur by using MIMOSYS (Mind Monitoring System)(2), *3. MIMOSYS is a technology that measures the mental health condition of a person based on voice (involuntary reactions) that was developed by PST Inc. The cloud service, Mind Monitoring Service, was subsequently launched on June 27, 2017 as a primary prevention tool for dealing with mental illness(3).
Figure 1—Hitachi Systems’ Involvement in Healthcare Business
The company supplies a wide range of services that support health promotion and disease prevention.
Dealing with mental illness presents a management challenge to companies and there is considerable activity associated with employee health maintenance and improvement. This is because mental illness is a problem for personnel management and corporate risk management, and it represents one of the key aspects of health management that companies must deal with for themselves.
Loss of productivity can occur both through “absenteeism” and “presentism.” The former refers to time lost through staff not being at work to do their job because they have taken time off or are working short hours, whereas the latter refers to staff who, although present at work, do not perform to an acceptable level due to mental or physical health issues.
The increase in the number of people suffering from mental illness brings costs that include medical bills and compensation for failure to satisfy the employer's obligation of safety, and also leads to a loss of public trust due to the problem of karoshi (workers committing suicide due to overwork) and the associated reputational damage to the company. This means that a failure to take steps to deal with poor mental health can have serious implications for corporate management.
Whereas most actions taken by companies are secondary or tertiary preventions that are prompted by staff presenting with mental illness, the most effective approach is said to be primary prevention whereby indications of a problem are detected and the illness prevented before it manifests.
The requirements for primary prevention are measures for responding to an increase in the number of people presenting with illness, measures for dealing with repeated relapses after returning to work, crisis and risk management, the creation of a vibrant workplace, improvements in workplace morale and job satisfaction, and higher productivity and customer satisfaction. It is important to establish preventive practices so that staff do not become mentally ill by combining these with stress checks (secondary prevention) and programs for supporting a return to work (tertiary prevention).
The benefits of primary prevention go beyond better employee health and higher productivity and creativity, with the potential to also help with staff recruitment and retention and to enhance corporate reputation and facilitate branding. The mind monitoring service uses changes in how people usually speak as a basis for providing primary prevention in the form of the early detection and prevention of mental illness.
Figure 2—Technique for Using Voice to Assess Constantly Changing Mental State
When the brain feels stress, it manifests as an involuntary change in the voice that is outside the person’s control.
Figure 3—Changes in Vocal Cords
The vocal cords loosen when a person is relaxed and the pitch of their voice falls. When stressed, the vocal cords tighten and voice pitch becomes higher.
MIMOSYS is a technique for using people's voice to monitor their continually changing state of mental health.
Speech is produced by the vibration of vocal cords in their throats. The cerebral limbic system in the brain, which is the seat of emotion, is directly connected to the vocal cords by nerves and the brain sends signals to them when it feels stress. The vocal cords are loose when a person feels relaxed, for example, but tense up when they are stressed, causing the person to speak with a higher pitch. The technique uses this to quantify and display a person's state of mental health, as expressed by these involuntary reactions (changes in the vocal cords) (see Figure 2 and Figure 3).
This technique was tested as part of studies into verbal analysis of pathophysiology that used vocal biometric information to identify medical conditions at a social cooperation program of the Graduate School of Medicine, The University of Tokyo(4). The study mainly involved performing acoustic analysis of the characteristics of illnesses that affect the voice by way of the autonomic nervous system (which includes the vagus nerve in the case of emotional disorders, for example), and using this to determine the associated parameters.
This is the analysis of the relative components of the four emotions of joy, anger, sorrow, and calmness, and the intensity of excitement based on the patterns of pitch variation in the voice.
A total of about 20 seconds of speech is required to perform an analysis, with at least six samples being collected each time. A sample consists of the speech from one breath to another with the subject breathing continuously while speaking. The actual words spoken do not matter and may be voluntary speech or reading from a prepared text. The required level of audio quality is 11.025 kHz/16-bit mono, although 8 kHz/8-bit mono has been used in previous analyses.
MIMOSYS has two outputs: vitality and mental activity. Vitality indicates the subject's current state of mental health (at the time their speech was recorded) and is obtained by analyzing one set of voice data samples. Mental activity indicates their long-term state of mental health and is obtained by analyzing voice data from at least the past two weeks, as stipulated in the Diagnostic and Statistical Manual of Mental Disorders IV published by the American Psychiatric Association.
The service quantifies a person's state of mental health by using voice data recorded on a smartphone, fixed-line telephone, mobile phone, or other device to analyze changes in the vocal cords, displaying the result on a personal computer or smartphone in approximately 15 seconds.
Users are able to choose a smartphone app or telephone as their input device, whichever suits them better. The smartphone app can be installed by searching for mind monitoring service on Google*4 Play*4 for Android*4 or on App Store*5 for iOS*6. It works by recording the user reading a piece of text displayed by the app. When using a telephone, the service works by having the user call a toll-free number and recording their response to prompts.
The recorded voice data is sent to a cloud system operated by Hitachi and the results of analysis by the MIMOSYS engine are displayed on the smartphone app in about 15 seconds (up to 40 seconds when using a telephone). When using a telephone, the results are available via a web browser using a previously registered user identity (ID) and password (see Figure 4).
Figure 4—System Environment
The voice data is analyzed by the MIMOSYS engine in the cloud and the result displayed after only 15 seconds.
The service presents its results as numerical values for vitality and mental activity as well as visually using graphs and illustrations (see Figure 5).
As each vitality value is obtained from a single analysis, values for each day are displayed in a graph.
As mental activity is a trend value obtained by the analysis of all voice data over the past two weeks, it is also displayed as a graph of daily values. For the early detection and prevention of mental illness, it is the trend in mental activity rather than vitality that is important.
Figure 5—Types of Analysis Result
Vitality indicates the result of a single analysis and mental activity is a trend value obtained by analyzing all voice data from the past two weeks.
Figure 6—Supervisor Screen
The supervisor screen indicates which users meet the alert conditions. A list of these users can also be displayed to facilitate the monitoring of large numbers of users.
With the subject's consent, the service provides results not only to the person themselves but also to a supervisor (such as a workplace physician or leader) (see Figure 6).
The supervisor has a master file specifying threshold values for people's state of mental health*7 that are used as the basis for raising alerts. The four conditions for an alert are: five or more consecutive days of falling mental activity, mental activity being 20 or more points down over a seven-day period, mental activity being 20 points or less for three or more consecutive days, and two or more days having elapsed since the person last made a voice recording. The last of these is to raise a warning if a user forgets to make a voice recording.
An e-mail is sent automatically whenever a person meets one of the alert conditions. As well as the supervisor being able to receive this e-mail, people who meet alert conditions are highlighted on the supervisor's screen and can be displayed in a list to facilitate the monitoring of a large number of users.
As well as enabling the subject to view their own results and take steps for themselves, these functions help deal with the social problem of mental illness through the early identification of people who have a mental illness or are showing warning signs by enabling the supervisor to act on users' state of mental health.
Once they have subscribed to the service, users have immediate access without any action being required of the supervisor. All the user needs to do is to download the app (if using a smartphone) and then enter their assigned user ID and password.
The service currently obtains its input from either the smartphone app or telephone by having users record their voice when reading a predefined piece of text. In the future, Hitachi intends to further develop the idea from a simple service into a solution by integrating it with smart speakers, care robots, or other applications. The service will also be expanded into a solution with a role in corporate health management that contributes to work style reform and to improvements in engagement, also becoming part of a healthcare solution.
As part of its efforts to encourage the wider adoption of corporate health management, the Ministry of Economy, Trade and Industry has established the Certified Health & Productivity Management Outstanding Organizations Recognition Program. Hitachi hopes that the mind monitoring service will help companies be better recognized as places where it is easy to work and where people want to work.
Details of MIMOSYS, a technology that measures the mental health condition of a person based on voice, were compiled with considerable support and assistance from the people involved, including Shinichi Tokuno, Project Associate Professor, Graduate School of Medicine, The University of Tokyo and PST Inc. The authors would like to take this opportunity to express their deepest thanks.
