Project ALERT

The term ‘hyperscanning’ was coined by Montague et al. in their 2002 paper where functional magnetic resonance imaging (fMRI) was acquired from two subjects simultaneously while they played a children’s game called handy-dandy. While player 1 concealed the object, player 2 had to guess which hand held the object [1]. Hyperscanning studies involve data acquisition from two or more subjects simultaneously and today, it is frequently employed to gain insight on neural underpinnings of inter- and intra-brain interactions on the central and peripheral nervous system [2]. Hyperscanning setups are being explored in various application fields especially to enable the investigation of social synchrony, shared experiences, team chemistry as well as leader-follower dynamics during cooperative or competitive tasks. While quite a few published studies exist for EEG hyperscanning and fNIRS hyperscanning, multimodal hyperscanning studies combining the benefits of EEG and fNIRS are still not routine [3,4,5,6]. In this showcase, we would like to highlight an innovative classroom experiment where a multimodal EEG-fNIRS headset was employed to gauge alertness levels in students in different interactive settings as part of the EFRO funded ALERT project. 

The interdisciplinary project team involved in the ALERT project came together by contributing to various parts of the project. The ANT Neuro and Artinis teams collaborated on the EEG-fNIRS headset design and established an integrated Lab Streaming Layer (LSL) platform to allow for multimodal recordings. Thales and Noldus supported the integration of the hardware in real world simulations for effective outcomes. Thales was responsible to validate the ALERT measurement system and its integration with the HARMONY platform to show results of alertness and workload models in two control room scenarios in collaboration with Twente Safety Region (VRT) and TenneT. Noldus on the other hand used the ALERT data to improve their NoldusHub software to allow this platform to store and display the workload assessment results offline to the research partners at UTwente. The research partners at the University of Twente analyzed the acquired datasets, classified and correctly labelled the EEG data to derive meaningful metrics out of it such that the learnings can be applied in a real-world scenario. 

Monitoring cognitive states and mental fatigue (MF) is imperative in real world settings, especially to monitor mental workload (MWL) in environments that demand complete engagement of participants. For example, the mental workload (MWL) experienced by pilots in flights or flight simulations, alertness levels of students in classroom settings with varying interaction levels can shed light on new approaches that can be adopted to prevent mental fatigue in those professions. 

Hamann et. al. in their 2022 study used both EEG and fNIRS data to record cortical activation in 35 subjects in a simulated flight along with performance data and self-reported MWL and MF ratings and proved the importance of physiological measurements for assessment of cognitive states and adaptive approaches. They concluded that higher task difficulty led to higher subjective MWL ratings, a decline in performance, higher frontal theta activity and reduced frontal deoxygenated hemoglobin (HHb) concentration. They also reported that while fNIRS was sensitive to less difficult tasks, EEG was sensitive to tasks with higher difficulty levels [7]. 

Employing physiological measures such as EEG and fNIRS allows continuous assessment of MWL and can be beneficial in improving the efficacy of teaching outcomes on students. Therefore, the ALERT project presents the possibility of acquiring multimodal datasets in a real-world setting such as the classroom environment to improve our understanding of alertness and engagement.

20 students from the University of Twente (UT) were in one classroom during an in-person lecture and were asked to first self-report their alertness rating using the Karolinska Sleepiness Scale (KSS) in multiple blocks. The live video of the classroom was recorded for a visual feedback of participant states and the lecture was constructed to have passive and interactive blocks to investigate engagement or alertness in these two main experiment conditions. For physiological recordings, the participants wore a multimodal EEG-fNIRS headset. This headset was developed on a 24 channel waveguard™ original cap from ANT Neuro. The cap incorporated the Artinis Brite device in its DualMode with 18 transmitters and 14 receivers, forming 38 channels to measure from frontal and parietal cortical regions. The headset was connected to the eego24™ amplifier which was connected via USB to the eego acquisition device which in this case was a portable tablet, whereas the Brite was connected via Bluetooth to Artinis’s OxySoft software to record the fNIRS data. Finally, the multimodal data stream was reduced to an aggregated data stream that went as an input to the NoldusHub software. 

The main research question that was explored in this classroom experiment was whether there is any ground truth in employing multimodal, largescale hyperscanning experiments in real world environments. In addition, an interesting outcome of the experiment would be to find correlations between sleep and alertness or engagement. Finally, the research team aims to explore whether engagement and alertness can be quantified similarly or can lead to completely different outcomes. 

Mental workload assessment has been made possible with EEG frequency band combinations and ratios, task human engagement is calculated using the engagement index which is derived as beta/ (alpha+theta). MacLean et al’s 2012 paper reports that high vigilance states are characterized by a reduction in the alpha band activity and is correlated to increased activity in the frontal-parietal areas with an increase in beta power, followed by a decrease in theta. Finally, alpha band activity reduction correlates with activities in the parietal brain region which sees a decrease in beta activity, followed by an increase in theta activity during drowsiness and low attention phases [8,9]. A preliminary check of the dataset confirms that engagement index varies for two representative subjects, in the block trials. Figure 2 depicts the engagement index of the students that participated in the ALERT project over the experimental conditions – interactive and passive teaching blocks.