EEG Pattern Analysis for Physiological Indicators of Mental Fatigue in Simulated Air Traffic Control Tasks

Objective: This study was designed to identify potential neurophysiologic markers and patterns of mental fatigue among air traffic controllers in their work environment.Background: The monitoring of mental fatigue in air traffic controllers has been of interest as their tasks involve high cognitive workload and are also critical to the safety of the public.Method: High-density electroencephalogram (EEG) was used to record 2-hour long air traffic control studies in eleven participants. Participants were asked to perform realistic tasks in a simulation, to operate a virtual air traffic control system. Fourier Transforms were used to estimate EEG power spectrum, statistical tests were implemented to reveal EEG spatial pattern changes caused by the time-on-task. The concept of mental state transition was introduced to study the development of certain mental states which are related to the mental fatigue. Results: The observation of EEG spectral data over a period of time revealed statistically significant changes spatially localized to central and parietal cortices. Rhythmic EEG activity within theta, alpha, and beta bands indicates transitions among mental states, which appear to be promising indicators for the development of mental fatigue. Mental fatigue indicated by the transition of mental states was estimated to approximately 70 minutes after the time on task. Application: This study can build the foundation to develop promising technologies for real time monitoring of mental fatigue, which will increase public safety and proper human resource planning.


INTRODUCTION
Air traffic control systems are very important for the safety and efficiency of air traffic operations. The major task for an air traffic controller is to safely sequence the flow of aircrafts by preventing them from coming too close to each other. At the same time, air traffic controllers must ensure systematic, quick, and efficient air traffic flow management and are responsible for providing vital information to aircraft pilots, such as navigation guidance, weather reports, and emergency alerts. The tasks for air traffic controllers are cognitively demanding. They need to understand data from a variety of sources quickly, utilize them in positioning aircrafts, and project each of their positions forward into the near future. They are required to work reliably and efficiently with constant focus and effort put forth towards monitoring, controlling, checking, diagnosing, and problem solving in a considerably long duration [Kallus et al., 1996]. These tasks subject controllers to high workloads, which are likely to induce mental fatigue after certain time on tasks. This timeon-task fatigue can be detrimental to the performance of controllers, leading to failures in task completion or lapses in vigilant attention [Bosken et al., 2005]. Due to this reason, the mental fatigue has been of special interest to researchers in multiple disciplines who seek to understand its impact on human work capacity.
In order to determine the amount of time an air traffic controller can effectively remain on task, a method must be developed that can determine the change of mental states and monitor the development of mental fatigue, preferably in real time, to avoid mistakes that will cause airplane crashes and/or human life losses. Mental workloads are traditionally assessed by performance and subjective data (e.g., NASATLX). Recently, physiological measurements (such as ocular movements, heart rate, respiration, and EEG recordings) [Brooking et al., 1996, Han et al., 2005 have also been utilized to monitor mental fatigue. EEG data is considered one of the most reliable methods in detecting mental fatigue [Lal et al., 2003;Murata et al., 2005]. Specifically, EEG spectral powers reflect the number of neurons that discharge synchronously and their distribution is an indicator of the capacity and performance of brain information processing. It is believed that increased alpha and theta band EEG powers are related to cognitive task loads [Kilmesch, 1999] and mental workloads [Smith et al., 2001]. EEG measurements are also sensitive to task difficulty [Gevins et al., 1997]. Some EEG studies [Shen et al., 2007;Caat et al., 2008] focus on developing effective EEG measures in assessing mental fatigue.
Many mental fatigue studies using EEG only explore repetitive simple cognitive tasks that are unrealistic. In the present study, we are interested in identifying indicators of mental fatigue in continuous EEG data from high-density recordings (i.e., 128 channels) where participants perform a continuous air traffic control task in a realistic simulation for two hours. Rather than testing all EEG measures, we implemented our analysis with EEG spectral powers in theta, alpha, and beta bands. The changes in recorded EEG data and mental state transitions indicated by the EEG power spectrum were studied to identify indicators of mental fatigue related to cognitive tasks over long work periods. We conducted this experimental pilot study in eleven participants and will also report their behavioral data related to performance.

Participants
Eleven normal right-handed human subjects (age 25 ± 4.3, all males, right handed) took part in the present study after receiving informed consent according to OU/IRB. Participants were compensated monetarily for their time.

Equipment and Apparatus
EEG data was recorded using our EEG recording system, i.e., Net Amps 300 Amplifier (Electrical Geodesics, Inc. Eugene, OR), connected to a 128-channel electrode cap (HydroCel Geodesic Sensor Net). Scenarios of air traffic controls were simulated by virtual air traffic control software called C-Team V2.0 [Bailey et al., 1999].

Air Traffic Control Task
Each participant was required to go through three experimental sessions while operating the air traffic control system simulated by C-Team V2.0. The task is to navigate airplanes that appear on the screen at the rate of two per minute to respective airports or exits defined on each airplane ( Fig. 1). There are two airports and two exits, and each plane has to be navigated to one of them. The landing of airplanes must obey the orientations of airport runways, which are designed to change every five minutes. The exits remain stationary on the north and west sides of the screen. The south and east side exits are blocked. A restricted area is designed to appear in the middle of the screen, and airplanes must get around it to avoid a crash. Warning signals appear one minute prior to airport runway orientation changes. Proximity warnings are generated when airplanes are too close to other airplanes, boundaries, or restricted areas. A proximity error might turn into a crash error if proper adjustments are not made. Proximity warnings and crash errors are recorded in the replay file, which is used for the performance analysis. Controllers can control airplanes by changing the height, speed, and heading direction of airplanes. Controllers can make inputs to change these parameters in order to safely navigate airplanes to their destinations. Inputs made by controller are also recorded in the replay file.

Procedure
Each participant had to go through one training session and two recording sessions. After the tasks were explained, participants were given 30 minutes to familiarize themselves with C-Team software. They were encouraged to formulate their own strategy to work on tasks during their training session. Each recording session was two hours long. In the first recording, all participants were observed to have a steep learning curve on the strategy planning as these participants were untrained with the air traffic control system. The second recording from all participants was the one which had more stable performance and provided fewer complications from learning. In the present study, we focused our analysis on data from the second recording session, which is believed to produce more reliable results.

Data Acquisition
EEG recordings were sampled at 500 Hz. During EEG recordings, participants were asked to focus on their tasks. Except necessary movements to control keyboard and mouse, participants were instructed to avoid eye blinks, swallowing, or any other movements as much as possible. At the time marker of one hour during the two hour recording, we performed an EEG electrode impendence check. The purpose for this impedance check is to keep the impedance of the net lower than 75 KΩ to ensure the high signal-noise-ratio in EEG data. Participants were instructed to ignore the impedance check and to continue to engage in tasks. This impedance check usually took about 5 to 10 minutes.

Data Processing
Recorded EEG data was applied with a band pass filter of 0.5 Hz-30 Hz. Artifacts and bad channels were marked by visual inspection and replaced by a bad channel replacement tool from EGI Net Station 4.2 (Electrical Geodesics, Inc. Eugene, OR). The data in bad channels were replaced with interpolated data from neighbored channels. The procedure is reasonable since electrical field in the neighbored areas usually has continuous distribution due to the volume conductance effect. In each recording, only few channels were bad and being interpolated. EEG data, after artifact removal and bad channel replacement, was then divided into 10-minute segments. It resulted in 11 sets of 10-minute EEG data (as about 10 minute EEG data during the impedance check were discarded). Every 10-minute segment of EEG data was further epoched into 600 1-second data for the subsequent spectral power analysis as discussed below.
Spectral data calculation. EEG time-frequency representations (TFRs), estimating the time course in power, were computed using a Fourier approach at each channel, applying a sliding tapered window. We analyzed three frequency ranges: 4-8 Hz for theta band, 8-12 Hz for alpha band, and 12-30 Hz for beta band. Within each frequency band, we applied a multitaper approach [Percival and Walden, 1993] using a fixed window length of 0.25 s with 50% overlap. Each of this part was windowed with a Hamming window, where in eight modified periodograms are computed and averaged.
Statistical analysis. In order to reveal spatial pattern changes of EEG with respect to time, EEG segment data were analyzed and compared in each subject. A pair-wise comparison of 11 10-minute segments was performed with ttest at 128 electrode sites for each frequency band (i.e., theta, alpha, and beta). The t-test is to test whether means of two segments are statistically different, also taking considerations of their variances. We selected the statistical significance value at p < 0.05 (corrected with the Bonferroni method). We then plotted out t-values at each electrode on the sensor space with three colors. Green colors mean no statistically significant changes between two compared 10-minute EEG segments. Blue colors indicate the significant increase in the second segment against the first segment, while red colors indicate the significant decrease. Channel Selection. In order to identify EEG changes related to time and further its correlate to mental fatigue due to time on task, a representative channel from spatial electrode maps (see Fig. 2 for an example) was then selected in each participant where the most significant increase in the alpha band is present. Since the pair-wise comparison from statistical analysis only gives the difference in two different segments, the selected channel will be used to investigate absolute dynamic patterns in EEG data.
Power spectral analysis. Power spectra at different time windows (i.e., from different segments) in different frequency bands were inspected with Whisker plots (see an example in Fig. 3). We assessed four types of patterns for EEG spectral power changes along the time axis. (1) Pattern in median of EEG spectral power: we are looking at the 50% mark of 600 samples at each 10-minute time window, which indicates the mean level of activity at each frequency band. The pattern changes in median can indicate that the neural network in human brain, which is responsible for the generation of rhythmic activity, has made a transition from one state to another.
(2) Pattern in variance of EEG spectral power: the pattern of variance provides information regarding the stability of mental states within the time window on handling mental tasks. Its pattern change indicates another aspect of mental state which is not indicated by the median EEG spectral power. In some cases, a variance pattern change without a median pattern change (or few median pattern changes) might present the instability of current mental state and indicate the coming shift to another mental state. (3) Changing pattern of median: the change in the median can be observed by subtracting medians from consecutive segments. The peaks and the dips that are observed will show us how the median has been changing and the rate of its change. The higher peaks indicate a large change in medians, which can indicate the change trends in mental states. (4) Changing pattern of variance: similar to changes in the pattern of median, we can observe changes in pattern of variance to determine the stability change trend in states of the brain.
Mental state analysis. The mental state during a specific neural computation is, dependent not only on the task but the fatigue of the individual, reflecting in the functional condition of the brain. As the EEG system records signals from a population of neurons, changes in states of the human brain can be observed on a macro scale [Pascal et al., 1995]. We utilize the above discussed four patterns to investigate macro brain states. Our hypothesis is that mental fatigue will interfere with neural computation performed by each neuron and thus is observable in macro mental state analysis.
Using values from these four parameters, the time for the transition between mental states can be estimated in each participant. Medians and variances of 10-minute segments are calculated for data which were averaged within specific frequency bands. Change patterns of median and variance were obtained through subsequent subtractions. The time estimate for mental state transitions in our EEG data appears to be a good indicator of the mental fatigue influencing the brain. In some participants, mental state transition was observed in median pattern where a steep slope could be seen. In other participants this kind of transition could depend on observations from plots of median patterns, variance patterns, median change patterns, and/or variance change patterns. We decided the time estimate for mental state transition by combining observations from all plots together.

RESULTS
Nine participant's task performances with C-Team were at reasonable levels. Two participants were excluded from further EEG analysis because their performance was significantly worse than other participants. One other participant was also excluded since this participant did not follow the instruction well and responded to the impedance check at one hour mark, which generated impacts on EEG recordings. Thus, here we presented results from eight participants.
The average number of planes that a participant was handling at a time is 4.0 (SD=0.25). The average number of total proximity warnings each participant received in the 2hour session is 8.3 (SD=6.07). The average number of total crashes that each participant had is 3.0 (SD=4.69). Participants rated their mental workloads after the session with NASA-TLX. The average mental workload rate is 48.8 (SD=17.8). Fig. 2(a) illustrates an example of spatial distribution of alpha EEG spectral power during the 9th 10-minute time window over the 128-electrode space from the participant #4. Each small dot represents an electrode, which is indexed by a number. The electrodes inside of the black circle are on the top of the head, which are the important ones in our EEG study. The electrodes outside the black circle are on the face and recorded EEG data from these channels might be interfered by electrical activity generated by face muscles. Fig.  2(a) indicates distributed alpha spectrum without an obvious emphasis related to time-on-task effects or mental fatigues. Since we are interested in changes happening in two hours, only the difference among paired 10-minute segments will indicate possible correlations to such timed changes. We conducted sequential comparison between the first 10-minute segment and the rest segments in order to explore such changes on the spatial map of electrodes. Fig. 2(b) presents the statistical map from a t-test, which was performed among the first 10-minute and last 10-minute from the participant #4. The blue indicates a significant increase of EEG alpha spectra in the data from the last 10-minute segment against the first 10minute segment, while the red indicates a significant decrease. As shown by all paired t-test results, significant increases in theta, alpha, and beta bands prevail in all participants, and the most significant changes are crowded in the midline area from the frontal area to the anterior parietal area, which are consistent with other studies from the literature about rhythmic changes in EEG related to mental fatigue [Smith et al., 2001]. The electrodes showing the most significant changes in eight participants are marked with red dots in Fig.  2(c), which are all located within the central midline area. These channels are used in the subsequent study of quantitative dynamic EEG rhythmic changes.  Fig. 3 shows the absolute power spectral data of alpha band, in blocks of 10 minutes, from all 8 participants in the form of Whisker plots. Absolute power spectral data within theta and beta bands also indicate similar trends, which are not shown here. Whisker plots graphically depict groups of numerical data with their statistical meanings. In each Whisker plot (representing 10 minutes), there are total 600 samples of short-time Fourier coefficients calculated on 1-second EEG data. The median is the center of the Whisker box, the upper end of box is the upper quartile (Q U ), and the lower end of the box is the lower quartile (Q L ). The ends of the whiskers represent the extreme data points which are not considered as outliers. The values outside (marked by crosses) are called outliers, which represent small portion of data off the main pattern. A consistent trend of increasing and then stabilizing to a constant level can be observed across all participants. It is obvious that this increasing pattern not only appears in median values of 11 10-minute segments, but also in variances of these segments. The variance of each 600 samples can be approximated as the length of Whisker boxes (i.e., the difference between the Q U and Q L ). Furthermore, although general trends of rhythmic alpha EEG activity in all participants are similar, individual variations are also observed. For example, while participants #3 and #7 have early onsets of increases in median and variance values of alpha power spectra, participants #6 and #8 clearly indicate late onsets of such increases.  In order to understand mental state transitions, we plotted out all kinds of maps as discussed in the methods section. Fig.  4(b) shows an example of curves for median values as a function of time from theta (blue curve), alpha (green curve), and beta (red curve) bands in the participant #8. It is clearly indicated that the median value appears stable before 75 minutes and increases sharply after this time point. We estimated 75 minutes as the time for the mental state transition within 2-hour work, which might indicate the development of mental fatigue. Another example Fig. 4(a) is from the participant #1. It is clear that within the time window between 50 minutes and 75 minutes, the median curve has different patterns, which indicates a mental state transition. However, the exact time for this transition is not well defined on the median curve. Furthermore, its variance curve stabilizes slowly after the first half hour and indicates a sudden increase around 65 minutes, which is exactly within the time window estimated from the median curve.

DISCUSSION
In the present study, we analyzed continuous EEG data for potential indicators of mental fatigue due to the time-ontask effect. The experimental task was simulated to mimic those performed by real air traffic controllers. We implemented a power spectrum analysis on our collected continuous EEG data. Our t-test results indicate that there are statistically significant differences existing among EEG data from different time windows at theta, alpha, and beta bands. These dynamic spectral patterns are spatially localized to the midline area over the human brain covering from the frontal area to the anterior parietal area in eight participants who were investigated. Continuous mean and variance changes of power spectra are suggested by our segmented continuous EEG data. From observing changing patterns in means and variances of power spectra at theta, alpha, and beta bands, we found transitions of mental states, in all eight participants happening approximately 70 minutes into the task.
The present study is conducted in a simulated air traffic control task which is realistic and providing a virtual working environment of controllers. We recorded two hour EEG data on each participant and investigated these continuous EEG data. These aspects mimic the realistic working conditions, which are not achieved from previous investigations of event related potentials of mental fatigue using unrealistic cognitive repetitive tasks in relative short durations [Boskem et al., 2005]. The long duration on tasks helps in understanding the development of mental fatigue due to the time-on-task effect.
The observations of transitions between mental states provided us a promising marker in identifying the development of mental fatigue in terms of working time. This can be a potential indicator of mental fatigue which can help in the planning of work shift or task allocation among air traffic controllers. While we missed an independent metric to evaluate the preciseness of 70 minutes on the development of mental fatigue, observed rhythmic pattern changes in theta, alpha, and beta bands, among the channels presented in the central and parietal cortices, is consistent with reports from literatures [Smith et al., 2001;Lal et al., 2003]. The consistency among all eight participants is seen in the small variance of estimated mental transition times (i.e., 7.28 minutes). In our future study, we plan to compare this estimated mental fatigue development with subjective evaluation by participants for a cross validation. It is also worth mentioning that, in the present study, the impedance check might complicate EEG data. We might need use EEG sensor nets which can be used for long-term purposes, such as in sleep monitoring [Steriade and Amzica, 2003].
In summary, we have conducted an EEG study on finding possible neurophysiologic markers for the development of mental fatigue due to time on tasks. We have found the measure related to the transition of mental states as indicated by spatially localized rhythmic brain activities is well correlated to the time-on-task effect and is believed to be a promising indicator of mental fatigue.