Шишкин Сергей Львович
Ведущий научный сотрудник
кандидат биол. наук
Руководит направлением
Публикации с аффилиацией МЭГ-центра
2024
14.
Yashin, Artem S.; Vasilyev, Anatoly N.; Shevtsova, Yulia G.; Shishkin, Sergei L. (2024). Can Quasi-Movements be Used as a Model of the BCI Based on Attempted Movements? 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Sarawak, Malaysia, October 6-10, 2024. 2028-2033. https://doi.org/10.1109/smc54092.2024.10831475
@conference{Yashin2024d,
title = {Can Quasi-Movements be Used as a Model of the BCI Based on Attempted Movements?},
author = {Artem S. Yashin and Anatoly N. Vasilyev and Yulia G. Shevtsova and Sergei L. Shishkin},
doi = {10.1109/smc54092.2024.10831475},
year = {2024},
date = {2024-10-06},
urldate = {2024-10-06},
booktitle = {2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC)},
pages = {2028-2033},
address = {Sarawak, Malaysia, October 6-10, 2024},
abstract = {Brain-computer interfaces (BCls) based on motor imagery (imagined movements, 1M) are among the most common BCls for the rehabilitation of paralyzed patients. However, it is possible that attempted movements (AM) would be more an effective alternative for 1M. Unlike 1M, AM are difficult to study outside of clinical practice. Nikulin et al. (2008) suggest that quasi-movements (QM) could help model AM in healthy participants without immobilizing interventions. QM result from the amplitude reduction of an overt movement, which leads to the practical absence of electromyography (EMG) response. The performance of QM may have features that may distance QM from AM. Here, we examined the compatibility of QM with a saccade task, which modelled visual interaction with the outside world during the practical use of a BCI. In a study involving 24 volunteers, we used electroencephalography (EEG), EMG, and conducted an extensive survey of the participants. We expected that, compared to 1M, QM in the dual-task condition would be easier and less tiring and would be accompanied by greater event-related desynchronization (ERD) of the sensorimotor rhythms. Our hypotheses were based on the assumption that like AM and unlike 1M, QM is a more external task, and so is more compatible with the saccade task. We reproduced the effect of greater ERD for QM in the dual-task condition but did not find any significant difference between the difficulty or tediousness of QM and 1M. Nevertheless, the survey data gave us important insights into the challenges participants faced when performing QM. Despite EMG values similar to 1M, the feeling of muscle tension experienced by the participants correlated with mean EMG values. The main challenge in performing QM by the participants was to make movements without an amplitude. Performing QM conflicted with the illusion of movement that was supposed to accompany them: without proprioceptive feedback, participants doubt the reality of QM. Our results can be used to improve the procedure of QM training, which should bring them closer to genuine attempts of movements in the eyes of participants.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Brain-computer interfaces (BCls) based on motor imagery (imagined movements, 1M) are among the most common BCls for the rehabilitation of paralyzed patients. However, it is possible that attempted movements (AM) would be more an effective alternative for 1M. Unlike 1M, AM are difficult to study outside of clinical practice. Nikulin et al. (2008) suggest that quasi-movements (QM) could help model AM in healthy participants without immobilizing interventions. QM result from the amplitude reduction of an overt movement, which leads to the practical absence of electromyography (EMG) response. The performance of QM may have features that may distance QM from AM. Here, we examined the compatibility of QM with a saccade task, which modelled visual interaction with the outside world during the practical use of a BCI. In a study involving 24 volunteers, we used electroencephalography (EEG), EMG, and conducted an extensive survey of the participants. We expected that, compared to 1M, QM in the dual-task condition would be easier and less tiring and would be accompanied by greater event-related desynchronization (ERD) of the sensorimotor rhythms. Our hypotheses were based on the assumption that like AM and unlike 1M, QM is a more external task, and so is more compatible with the saccade task. We reproduced the effect of greater ERD for QM in the dual-task condition but did not find any significant difference between the difficulty or tediousness of QM and 1M. Nevertheless, the survey data gave us important insights into the challenges participants faced when performing QM. Despite EMG values similar to 1M, the feeling of muscle tension experienced by the participants correlated with mean EMG values. The main challenge in performing QM by the participants was to make movements without an amplitude. Performing QM conflicted with the illusion of movement that was supposed to accompany them: without proprioceptive feedback, participants doubt the reality of QM. Our results can be used to improve the procedure of QM training, which should bring them closer to genuine attempts of movements in the eyes of participants.
13.
Chetkin, Egor I.; Kozyrsky, Bogdan L.; Shishkin, Sergei L. (2024). Unconditional EEG Synthesis Based on Diffusion Models for Sound Generation. 2024 IEEE International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON). Novosibirsk State University, 30 Sep – 2 Oct, 2024. 416-420. https://doi.org/10.1109/sibircon63777.2024.10758527
@conference{Chetkin2024,
title = {Unconditional EEG Synthesis Based on Diffusion Models for Sound Generation},
author = {Egor I. Chetkin and Bogdan L. Kozyrsky and Sergei L. Shishkin},
doi = {10.1109/sibircon63777.2024.10758527},
year = {2024},
date = {2024-09-30},
urldate = {2024-09-30},
booktitle = {2024 IEEE International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON)},
pages = {416-420},
address = {Novosibirsk State University, 30 Sep – 2 Oct, 2024},
abstract = {Classifiers used in brain-computer interfaces based on the electroencephalography (EEG) typically demonstrate rel-atively low performance, which is a serios obstacle for making them a practical technology. One of the most important limitations that prevents improving EEG classification is the scarcity of the EEG data. Thus, generation of synthetic data could help to enhance classification. Recently, diffusion models were applied time for time series generation and first steps were made in generating synthetic EEG data using them. Here, we introduce MultiChan Wavegrad, a novel diffusion model designed specifically for multichannel EEG data generation. We describe its architecture and preliminary results of its testing using the BCI competition IV 2a dataset with the EEG recorded during motor imagery. The data generated by MultiChanWaveGrad possessed some resemblance to the real EEG data, although did not reproduce the EEG characteristics well enough. Finally, we discuss possible future directions for improving its performance and possibly making it a useful tool for data augmentation, especially for improving training of BCI classifiers.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Classifiers used in brain-computer interfaces based on the electroencephalography (EEG) typically demonstrate rel-atively low performance, which is a serios obstacle for making them a practical technology. One of the most important limitations that prevents improving EEG classification is the scarcity of the EEG data. Thus, generation of synthetic data could help to enhance classification. Recently, diffusion models were applied time for time series generation and first steps were made in generating synthetic EEG data using them. Here, we introduce MultiChan Wavegrad, a novel diffusion model designed specifically for multichannel EEG data generation. We describe its architecture and preliminary results of its testing using the BCI competition IV 2a dataset with the EEG recorded during motor imagery. The data generated by MultiChanWaveGrad possessed some resemblance to the real EEG data, although did not reproduce the EEG characteristics well enough. Finally, we discuss possible future directions for improving its performance and possibly making it a useful tool for data augmentation, especially for improving training of BCI classifiers.
2023
12.
Shevtsova, Yulia G.; Vasilyev, Anatoly N.; Shishkin, Sergei L. (2023). Machine Learning for Gaze-Based Selection: Performance Assessment Without Explicit Labeling. HCI International 2023 – Late Breaking Papers. HCII 2023. Lecture Notes in Computer Science, volume 14054. Springer Nature Switzerland. 311-322. https://doi.org/10.1007/978-3-031-48038-6_19
@conference{Shevtsova2023,
title = {Machine Learning for Gaze-Based Selection: Performance Assessment Without Explicit Labeling},
author = {Yulia G. Shevtsova and Anatoly N. Vasilyev and Sergei L. Shishkin},
doi = {10.1007/978-3-031-48038-6_19},
isbn = {9783031480386},
year = {2023},
date = {2023-11-25},
urldate = {2023-11-25},
booktitle = {HCI International 2023 – Late Breaking Papers. HCII 2023. Lecture Notes in Computer Science, volume 14054},
pages = {311-322},
publisher = {Springer Nature Switzerland},
abstract = {Gaze-based interaction typically requires certain actions to confirm selections, which often makes interaction less convenient. Recently, effective identification of the user’s intention to make a gaze-based selection was demonstrated by Isomoto et al. (2022) using machine learning applied to gaze behavior features. However, a certain bias could appear in that study since the participants were requested to report their intentions during the interaction experiment. Here, we applied several classification algorithms (linear discriminant analysis, RBF and linear support vector machines, and random forest) to gaze features characterizing selections made in a freely played gaze-controlled game, in which moves were made by sequences of gaze-based selections and their gaze-based confirmations, without separate reporting the correctness of the selection. Intention to select was successfully predicted by each of the classifiers using features collected before the selection.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Gaze-based interaction typically requires certain actions to confirm selections, which often makes interaction less convenient. Recently, effective identification of the user’s intention to make a gaze-based selection was demonstrated by Isomoto et al. (2022) using machine learning applied to gaze behavior features. However, a certain bias could appear in that study since the participants were requested to report their intentions during the interaction experiment. Here, we applied several classification algorithms (linear discriminant analysis, RBF and linear support vector machines, and random forest) to gaze features characterizing selections made in a freely played gaze-controlled game, in which moves were made by sequences of gaze-based selections and their gaze-based confirmations, without separate reporting the correctness of the selection. Intention to select was successfully predicted by each of the classifiers using features collected before the selection.
11.
Berdyshev, Daniil A.; Grachev, Artem M.; Shishkin, Sergei L.; Kozyrskiy, Bogdan L. (2023). Meta-Optimization of Initial Weights for More Effective Few- and Zero-Shot Learning in BCI Classification. 2023 IEEE Ural-Siberian Conference on Computational Technologies in Cognitive Science, Genomics and Biomedicine (CSGB). Novosibirsk, Russian Federation, 28-30 September 2023. 263-267. https://doi.org/10.1109/csgb60362.2023.10329624
@conference{Berdyshev2023,
title = {Meta-Optimization of Initial Weights for More Effective Few- and Zero-Shot Learning in BCI Classification},
author = {Daniil A. Berdyshev and Artem M. Grachev and Sergei L. Shishkin and Bogdan L. Kozyrskiy},
doi = {10.1109/csgb60362.2023.10329624},
isbn = {979-8-3503-0797-9},
year = {2023},
date = {2023-09-28},
urldate = {2023-09-28},
booktitle = {2023 IEEE Ural-Siberian Conference on Computational Technologies in Cognitive Science, Genomics and Biomedicine (CSGB)},
pages = {263-267},
address = {Novosibirsk, Russian Federation, 28-30 September 2023},
abstract = {Brain-computer interfaces (BCIs) are heavily reliant on the underlying data classification. Neural network classifiers are often used for this purpose, but their performance is dependent, in turn, on the availability of large training sets, which are difficult to record. Hence, arises the necessity to employ methods capable of operating with limited sample sizes or leveraging experience acquired with different BCI users. Here, we explore the ability of meta-learning algorithms to enable neural network classifiers to leverage experience acquired with EEG data recorded in other BCI users to support few-shot or even zero-shot learning for BCI classifiers. We conducted experiments to assess the quality of EEG data classification using neural networks that were pre-trained on various users who were different from the test user. In these experiments we compared neural networks pre-trained with meta-learning algorithms and with traditional transfer learning, with further fine-tuning on small data amounts and even without fine-tuning. The experiments demonstrated the potential for classification quality improvement through meta-learning in few- and even zero-shot learning scenarios.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Brain-computer interfaces (BCIs) are heavily reliant on the underlying data classification. Neural network classifiers are often used for this purpose, but their performance is dependent, in turn, on the availability of large training sets, which are difficult to record. Hence, arises the necessity to employ methods capable of operating with limited sample sizes or leveraging experience acquired with different BCI users. Here, we explore the ability of meta-learning algorithms to enable neural network classifiers to leverage experience acquired with EEG data recorded in other BCI users to support few-shot or even zero-shot learning for BCI classifiers. We conducted experiments to assess the quality of EEG data classification using neural networks that were pre-trained on various users who were different from the test user. In these experiments we compared neural networks pre-trained with meta-learning algorithms and with traditional transfer learning, with further fine-tuning on small data amounts and even without fine-tuning. The experiments demonstrated the potential for classification quality improvement through meta-learning in few- and even zero-shot learning scenarios.
10.
Chetkin, Egor I.; Shishkin, Sergei L.; Kozyrskiy, Bogdan L. (2023). Bayesian Opportunities for Brain–Computer Interfaces: Enhancement of the Existing Classification Algorithms and Out-of-Domain Detection. Algorithms, 16(9), 429. https://doi.org/10.3390/a16090429
@article{Chetkin2023,
title = {Bayesian Opportunities for Brain–Computer Interfaces: Enhancement of the Existing Classification Algorithms and Out-of-Domain Detection},
author = {Egor I. Chetkin and Sergei L. Shishkin and Bogdan L. Kozyrskiy},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.3390_a16090429.pdf},
doi = {10.3390/a16090429},
issn = {1999-4893},
year = {2023},
date = {2023-09-08},
urldate = {2023-09-08},
journal = {Algorithms},
volume = {16},
number = {9},
pages = {429},
publisher = {MDPI AG},
abstract = {Bayesian neural networks (BNNs) are effective tools for a variety of tasks that allow for the estimation of the uncertainty of the model. As BNNs use prior constraints on parameters, they are better regularized and less prone to overfitting, which is a serious issue for brain–computer interfaces (BCIs), where typically only small training datasets are available. Here, we tested, on the BCI Competition IV 2a motor imagery dataset, if the performance of the widely used, effective neural network classifiers EEGNet and Shallow ConvNet can be improved by turning them into BNNs. Accuracy indeed was higher, at least for a BNN based on Shallow ConvNet with two of three tested prior distributions. We also assessed if BNN-based uncertainty estimation could be used as a tool for out-of-domain (OOD) data detection. The OOD detection worked well only in certain participants; however, we expect that further development of the method may make it work sufficiently well for practical applications.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Bayesian neural networks (BNNs) are effective tools for a variety of tasks that allow for the estimation of the uncertainty of the model. As BNNs use prior constraints on parameters, they are better regularized and less prone to overfitting, which is a serious issue for brain–computer interfaces (BCIs), where typically only small training datasets are available. Here, we tested, on the BCI Competition IV 2a motor imagery dataset, if the performance of the widely used, effective neural network classifiers EEGNet and Shallow ConvNet can be improved by turning them into BNNs. Accuracy indeed was higher, at least for a BNN based on Shallow ConvNet with two of three tested prior distributions. We also assessed if BNN-based uncertainty estimation could be used as a tool for out-of-domain (OOD) data detection. The OOD detection worked well only in certain participants; however, we expect that further development of the method may make it work sufficiently well for practical applications.
9.
Yashin, Artem S.; Shishkin, Sergei L.; Vasilyev, Anatoly N. (2023). Is there a continuum of agentive awareness across physical and mental actions? The case of quasi-movements. Consciousness and Cognition, 112, 103531. https://doi.org/10.1016/j.concog.2023.103531
Abstract | PDF (preprint) | BibTeX
@article{Yashin2023c,
title = {Is there a continuum of agentive awareness across physical and mental actions? The case of quasi-movements},
author = {Artem S. Yashin and Sergei L. Shishkin and Anatoly N. Vasilyev},
url = {https://osf.io/b6hw4},
doi = {10.1016/j.concog.2023.103531},
issn = {1053-8100},
year = {2023},
date = {2023-05-18},
urldate = {2023-05-18},
journal = {Consciousness and Cognition},
volume = {112},
pages = {103531},
publisher = {Elsevier BV},
abstract = {While humans routinely distinguish between physical and mental actions, overt movements (OM) and kinesthetically imagined movements (IM) are often viewed as forming a continuum of activities. Here, we theoretically conceptualized this continuum hypothesis for agentive awareness related to OM and IM and tested it experimentally using quasi-movements (QM), a little studied type of covert actions, which is considered as an inner part of the OM-IM continuum. QM are performed when a movement attempt is minimized down to full extinction of overt movement and muscle activity. We asked participants to perform OM, IM and QM and collected their electromyography data. According to participants’ reports, they experienced QM as OM in terms of intentions and expected sensory feedback, while the verbal descriptors were independent from muscle activation. These results do not fit the OM-QM-IM continuum and suggest qualitative distinction for agentive awareness between IM and QM/OM.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
While humans routinely distinguish between physical and mental actions, overt movements (OM) and kinesthetically imagined movements (IM) are often viewed as forming a continuum of activities. Here, we theoretically conceptualized this continuum hypothesis for agentive awareness related to OM and IM and tested it experimentally using quasi-movements (QM), a little studied type of covert actions, which is considered as an inner part of the OM-IM continuum. QM are performed when a movement attempt is minimized down to full extinction of overt movement and muscle activity. We asked participants to perform OM, IM and QM and collected their electromyography data. According to participants’ reports, they experienced QM as OM in terms of intentions and expected sensory feedback, while the verbal descriptors were independent from muscle activation. These results do not fit the OM-QM-IM continuum and suggest qualitative distinction for agentive awareness between IM and QM/OM.
8.
Vasilyev, Anatoly N.; Yashin, Artem S.; Shishkin, Sergei L. (2023). Quasi-Movements and “Quasi-Quasi-Movements”: Does Residual Muscle Activation Matter? Life, 13(2), 303. https://doi.org/10.3390/life13020303
@article{Vasilyev2023d,
title = {Quasi-Movements and “Quasi-Quasi-Movements”: Does Residual Muscle Activation Matter?},
author = {Anatoly N. Vasilyev and Artem S. Yashin and Sergei L. Shishkin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.3390_life13020303.pdf},
doi = {10.3390/life13020303},
issn = {2075-1729},
year = {2023},
date = {2023-02-00},
urldate = {2023-02-00},
journal = {Life},
volume = {13},
number = {2},
pages = {303},
publisher = {MDPI AG},
abstract = {Quasi-movements (QM) are observed when an individual minimizes a movement to an extent that no related muscle activation is detected. Likewise to imaginary movements (IM) and overt movements, QMs are accompanied by the event-related desynchronization (ERD) of EEG sensorimotor rhythms. Stronger ERD was observed under QMs compared to IMs in some studies. However, the difference could be caused by the remaining muscle activation in QMs that could escape detection. Here, we re-examined the relation between the electromyography (EMG) signal and ERD in QM using sensitive data analysis procedures. More trials with signs of muscle activation were observed in QMs compared with a visual task and IMs. However, the rate of such trials was not correlated with subjective estimates of actual movement. Contralateral ERD did not depend on the EMG but still was stronger in QMs compared with IMs. These results suggest that brain mechanisms are common for QMs in the strict sense and “quasi-quasi-movements” (attempts to perform the same task accompanied by detectable EMG elevation) but differ between them and IMs. QMs could be helpful in research aimed at better understanding motor action and at modeling the use of attempted movements in the brain-computer interfaces with healthy participants.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Quasi-movements (QM) are observed when an individual minimizes a movement to an extent that no related muscle activation is detected. Likewise to imaginary movements (IM) and overt movements, QMs are accompanied by the event-related desynchronization (ERD) of EEG sensorimotor rhythms. Stronger ERD was observed under QMs compared to IMs in some studies. However, the difference could be caused by the remaining muscle activation in QMs that could escape detection. Here, we re-examined the relation between the electromyography (EMG) signal and ERD in QM using sensitive data analysis procedures. More trials with signs of muscle activation were observed in QMs compared with a visual task and IMs. However, the rate of such trials was not correlated with subjective estimates of actual movement. Contralateral ERD did not depend on the EMG but still was stronger in QMs compared with IMs. These results suggest that brain mechanisms are common for QMs in the strict sense and “quasi-quasi-movements” (attempts to perform the same task accompanied by detectable EMG elevation) but differ between them and IMs. QMs could be helpful in research aimed at better understanding motor action and at modeling the use of attempted movements in the brain-computer interfaces with healthy participants.
2022
7.
Yashin, Artem S.; Zhao, Darisy G.; Stolyarova, Anastasiya N.; Moscowsky, Anton D.; Yakovlev, Dmitry S.; Nazhestkin, Ivan A.; Shishkin, Sergei L.; Dubynin, Ignat A. (2022). Subjective Distance Estimates and Sense of Agency in Robotic Wheelchair Control. Applied Sciences, 12(12), 6217. https://doi.org/10.3390/app12126217
@article{Yashin2022,
title = {Subjective Distance Estimates and Sense of Agency in Robotic Wheelchair Control},
author = {Artem S. Yashin and Darisy G. Zhao and Anastasiya N. Stolyarova and Anton D. Moscowsky and Dmitry S. Yakovlev and Ivan A. Nazhestkin and Sergei L. Shishkin and Ignat A. Dubynin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.3390_app12126217.pdf},
doi = {10.3390/app12126217},
issn = {2076-3417},
year = {2022},
date = {2022-06-18},
urldate = {2022-06-18},
journal = {Applied Sciences},
volume = {12},
number = {12},
pages = {6217},
publisher = {MDPI AG},
abstract = {Sense of agency (SoA) refers to an individual’s awareness of their own actions. SoA studies seek to find objective indicators for the feeling of agency. These indicators, being related to the feeling of control, have practical application in vehicle design. However, they have not been investigated for actions related to the agent’s body movement inherent to steering a vehicle. In our study, participants operated a robotic wheelchair under three conditions: active control by a participant, direct control by the experimenter and remote control by the experimenter. In each trial, a participant drove the wheelchair until a sound signal occurred, after which they stopped the wheelchair and estimated the travelled distance. The subjective estimates were significantly greater when participants operated the wheelchair by themselves. This result contrasts with observations under static settings in previous studies. In an additional study on the electroencephalographic response to a sound presented at a random time after movement onset, the observed latencies in the N1 component implied that participants might have a higher sense of control when they drove the wheelchair. The proposed methodology might become useful to indirectly assess the degree of operator control of a vehicle, primarily in the field of rehabilitation technologies.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Sense of agency (SoA) refers to an individual’s awareness of their own actions. SoA studies seek to find objective indicators for the feeling of agency. These indicators, being related to the feeling of control, have practical application in vehicle design. However, they have not been investigated for actions related to the agent’s body movement inherent to steering a vehicle. In our study, participants operated a robotic wheelchair under three conditions: active control by a participant, direct control by the experimenter and remote control by the experimenter. In each trial, a participant drove the wheelchair until a sound signal occurred, after which they stopped the wheelchair and estimated the travelled distance. The subjective estimates were significantly greater when participants operated the wheelchair by themselves. This result contrasts with observations under static settings in previous studies. In an additional study on the electroencephalographic response to a sound presented at a random time after movement onset, the observed latencies in the N1 component implied that participants might have a higher sense of control when they drove the wheelchair. The proposed methodology might become useful to indirectly assess the degree of operator control of a vehicle, primarily in the field of rehabilitation technologies.
6.
Shishkin, Sergei L. (2022). Active Brain-Computer Interfacing for Healthy Users. Frontiers in Neuroscience, 16. https://doi.org/10.3389/fnins.2022.859887
@article{Shishkin2022,
title = {Active Brain-Computer Interfacing for Healthy Users},
author = {Sergei L. Shishkin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.3389_fnins.2022.859887.pdf},
doi = {10.3389/fnins.2022.859887},
issn = {1662-453X},
year = {2022},
date = {2022-04-25},
urldate = {2022-04-25},
journal = {Frontiers in Neuroscience},
volume = {16},
publisher = {Frontiers Media SA},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2021
5.
Dubynin, Ignat A.; Yashin, Artem S.; Velichkovsky, Boris M.; Shishkin, Sergei L. (2021). An experimental paradigm for studying sense of agency in joint human–machine motor actions. Experimental Brain Research, 239(6), 1951-1961. https://doi.org/10.1007/s00221-021-06105-9
@article{Dubynin2021,
title = {An experimental paradigm for studying sense of agency in joint human–machine motor actions},
author = {Ignat A. Dubynin and Artem S. Yashin and Boris M. Velichkovsky and Sergei L. Shishkin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.1007_s00221-021-06105-9.pdf},
doi = {10.1007/s00221-021-06105-9},
issn = {1432-1106},
year = {2021},
date = {2021-04-21},
urldate = {2021-04-21},
journal = {Experimental Brain Research},
volume = {239},
number = {6},
pages = {1951-1961},
publisher = {Springer Science and Business Media LLC},
abstract = {In this paper, we propose an experimental technique for studying the sense of agency (SoA) in joint human–machine actions. This technique is based on the use of an electromechanical finger-lifting device that enables a joint motor action initiated by a participant and completed by the machine. The joint action, later referred to as an “active–passive” action, was implemented as a reaction time task and contrasted with other levels of participant’s involvement, including active movement, passive movement, and observation of a dummy’s movement. In each trial, a feedback sound signal informed the participant whether they had performed the task successfully, i.e. faster than a threshold, which was individually adjusted in the beginning of the experiment. In the active condition, the result depended on the participant, while in other conditions it was preprogrammed for the servo. In context of this task, we studied direct time estimates made by participants and auditory event-related potentials (ERP) in 20 healthy volunteers. The amplitude of the auditory N1 component in the responses to the feedback sound showed no significant effect of activity and success factors, while its latency was shorter in successful trials. Interaction of activity and success factors was significant for subjective time estimates. Surprisingly, the intentional binding effect (subjective compression of time intervals, which is known as a correlate of SoA) only emerged in trials of active condition with negative results. This observation was in contrast with the fact that the active and active–passive movements were both voluntarily initiated by the participant. We believe that studying SoA with the proposed technique may not only add to the understanding of agency but also provide practically relevant results for the development of human–machine systems such as exoskeletons.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In this paper, we propose an experimental technique for studying the sense of agency (SoA) in joint human–machine actions. This technique is based on the use of an electromechanical finger-lifting device that enables a joint motor action initiated by a participant and completed by the machine. The joint action, later referred to as an “active–passive” action, was implemented as a reaction time task and contrasted with other levels of participant’s involvement, including active movement, passive movement, and observation of a dummy’s movement. In each trial, a feedback sound signal informed the participant whether they had performed the task successfully, i.e. faster than a threshold, which was individually adjusted in the beginning of the experiment. In the active condition, the result depended on the participant, while in other conditions it was preprogrammed for the servo. In context of this task, we studied direct time estimates made by participants and auditory event-related potentials (ERP) in 20 healthy volunteers. The amplitude of the auditory N1 component in the responses to the feedback sound showed no significant effect of activity and success factors, while its latency was shorter in successful trials. Interaction of activity and success factors was significant for subjective time estimates. Surprisingly, the intentional binding effect (subjective compression of time intervals, which is known as a correlate of SoA) only emerged in trials of active condition with negative results. This observation was in contrast with the fact that the active and active–passive movements were both voluntarily initiated by the participant. We believe that studying SoA with the proposed technique may not only add to the understanding of agency but also provide practically relevant results for the development of human–machine systems such as exoskeletons.
4.
Zhao, Darisy G; Vasilyev, Anatoly N; Kozyrskiy, Bogdan L; Melnichuk, Eugeny V; Isachenko, Andrey V; Velichkovsky, Boris M; Shishkin, Sergei L (2021). A passive BCI for monitoring the intentionality of the gaze-based moving object selection. Journal of Neural Engineering, 18(2). https://doi.org/10.1088/1741-2552/abda09
@article{Zhao2021,
title = {A passive BCI for monitoring the intentionality of the gaze-based moving object selection},
author = {Darisy G Zhao and Anatoly N Vasilyev and Bogdan L Kozyrskiy and Eugeny V Melnichuk and Andrey V Isachenko and Boris M Velichkovsky and Sergei L Shishkin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.1088_1741-2552_abda09.pdf},
doi = {10.1088/1741-2552/abda09},
issn = {1741-2552},
year = {2021},
date = {2021-03-04},
urldate = {2021-03-04},
journal = {Journal of Neural Engineering},
volume = {18},
number = {2},
publisher = {IOP Publishing},
abstract = {Objective. The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain–machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behavior is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements. Approach. Eighteen participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection. Main results. A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved. Significance. The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g. in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user's involvement during gaze-based interaction.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Objective. The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain–machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behavior is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements. Approach. Eighteen participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection. Main results. A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved. Significance. The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g. in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user's involvement during gaze-based interaction.
3.
Ovchinnikova, Anastasia O.; Vasilyev, Anatoly N.; Zubarev, Ivan P.; Kozyrskiy, Bogdan L.; Shishkin, Sergei L. (2021). MEG-Based Detection of Voluntary Eye Fixations Used to Control a Computer. Frontiers in Neuroscience, 15, 619591. https://doi.org/10.3389/fnins.2021.619591
@article{Ovchinnikova2021,
title = {MEG-Based Detection of Voluntary Eye Fixations Used to Control a Computer},
author = {Anastasia O. Ovchinnikova and Anatoly N. Vasilyev and Ivan P. Zubarev and Bogdan L. Kozyrskiy and Sergei L. Shishkin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.3389_fnins.2021.619591.pdf},
doi = {10.3389/fnins.2021.619591},
issn = {1662-453X},
year = {2021},
date = {2021-02-05},
urldate = {2021-02-05},
journal = {Frontiers in Neuroscience},
volume = {15},
pages = {619591},
publisher = {Frontiers Media SA},
abstract = {Gaze-based input is an efficient way of hand-free human-computer interaction. However, it suffers from the inability of gaze-based interfaces to discriminate voluntary and spontaneous gaze behaviors, which are overtly similar. Here, we demonstrate that voluntary eye fixations can be discriminated from spontaneous ones using short segments of magnetoencephalography (MEG) data measured immediately after the fixation onset. Recently proposed convolutional neural networks (CNNs), linear finite impulse response filters CNN (LF-CNN) and vector autoregressive CNN (VAR-CNN), were applied for binary classification of the MEG signals related to spontaneous and voluntary eye fixations collected in healthy participants (n = 25) who performed a game-like task by fixating on targets voluntarily for 500 ms or longer. Voluntary fixations were identified as those followed by a fixation in a special confirmatory area. Spontaneous vs. voluntary fixation-related single-trial 700 ms MEG segments were non-randomly classified in the majority of participants, with the group average cross-validated ROC AUC of 0.66 ± 0.07 for LF-CNN and 0.67 ± 0.07 for VAR-CNN (M ± SD). When the time interval, from which the MEG data were taken, was extended beyond the onset of the visual feedback, the group average classification performance increased up to 0.91. Analysis of spatial patterns contributing to classification did not reveal signs of significant eye movement impact on the classification results. We conclude that the classification of MEG signals has a certain potential to support gaze-based interfaces by avoiding false responses to spontaneous eye fixations on a single-trial basis. Current results for intention detection prior to gaze-based interface’s feedback, however, are not sufficient for online single-trial eye fixation classification using MEG data alone, and further work is needed to find out if it could be used in practical applications.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Gaze-based input is an efficient way of hand-free human-computer interaction. However, it suffers from the inability of gaze-based interfaces to discriminate voluntary and spontaneous gaze behaviors, which are overtly similar. Here, we demonstrate that voluntary eye fixations can be discriminated from spontaneous ones using short segments of magnetoencephalography (MEG) data measured immediately after the fixation onset. Recently proposed convolutional neural networks (CNNs), linear finite impulse response filters CNN (LF-CNN) and vector autoregressive CNN (VAR-CNN), were applied for binary classification of the MEG signals related to spontaneous and voluntary eye fixations collected in healthy participants (n = 25) who performed a game-like task by fixating on targets voluntarily for 500 ms or longer. Voluntary fixations were identified as those followed by a fixation in a special confirmatory area. Spontaneous vs. voluntary fixation-related single-trial 700 ms MEG segments were non-randomly classified in the majority of participants, with the group average cross-validated ROC AUC of 0.66 ± 0.07 for LF-CNN and 0.67 ± 0.07 for VAR-CNN (M ± SD). When the time interval, from which the MEG data were taken, was extended beyond the onset of the visual feedback, the group average classification performance increased up to 0.91. Analysis of spatial patterns contributing to classification did not reveal signs of significant eye movement impact on the classification results. We conclude that the classification of MEG signals has a certain potential to support gaze-based interfaces by avoiding false responses to spontaneous eye fixations on a single-trial basis. Current results for intention detection prior to gaze-based interface’s feedback, however, are not sufficient for online single-trial eye fixation classification using MEG data alone, and further work is needed to find out if it could be used in practical applications.
2020
2.
Zhao, Darisy G.; Karikov, Nikita D.; Melnichuk, Eugeny V.; Velichkovsky, Boris M.; Shishkin, Sergei L. (2020). Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection. Applied Sciences, 10(24), 8791. https://doi.org/10.3390/app10248791
@article{Zhao2020,
title = {Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection},
author = {Darisy G. Zhao and Nikita D. Karikov and Eugeny V. Melnichuk and Boris M. Velichkovsky and Sergei L. Shishkin},
url = {https://megmoscow.ru/wp-content/uploads/pubs/10.3390_app10248791.pdf},
doi = {10.3390/app10248791},
issn = {2076-3417},
year = {2020},
date = {2020-12-09},
urldate = {2020-12-09},
journal = {Applied Sciences},
volume = {10},
number = {24},
pages = {8791},
publisher = {MDPI AG},
abstract = {Voice- and gaze-based hands-free input are increasingly used in human-machine interaction. Attempts to combine them into a hybrid technology typically employ the voice channel as an information-rich channel. Voice seems to be “overqualified” to serve simply as a substitute of a computer mouse click, to confirm selections made by gaze. It could be expected that the user would feel discomfort if they had to frequently make “clicks” using their voice, or easily get bored, which also could lead to low performance. To test this, we asked 23 healthy participants to select moving objects with smooth pursuit eye movements. Manual confirmation of selection was faster and rated as more convenient than voice-based confirmation. However, the difference was not high, especially when voice was used to pronounce objects’ numbers (speech recognition was not applied): Score of convenience (M ± SD) was 9.2 ± 1.1 for manual and 8.0 ± 2.1 for voice confirmation, and time spent per object was 1269 ± 265 ms and 1626 ± 331 ms, respectively. We conclude that “voice-as-click” can be used to confirm selection in gaze-based interaction with computers as a substitute for the computer mouse click when manual confirmation cannot be used.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Voice- and gaze-based hands-free input are increasingly used in human-machine interaction. Attempts to combine them into a hybrid technology typically employ the voice channel as an information-rich channel. Voice seems to be “overqualified” to serve simply as a substitute of a computer mouse click, to confirm selections made by gaze. It could be expected that the user would feel discomfort if they had to frequently make “clicks” using their voice, or easily get bored, which also could lead to low performance. To test this, we asked 23 healthy participants to select moving objects with smooth pursuit eye movements. Manual confirmation of selection was faster and rated as more convenient than voice-based confirmation. However, the difference was not high, especially when voice was used to pronounce objects’ numbers (speech recognition was not applied): Score of convenience (M ± SD) was 9.2 ± 1.1 for manual and 8.0 ± 2.1 for voice confirmation, and time spent per object was 1269 ± 265 ms and 1626 ± 331 ms, respectively. We conclude that “voice-as-click” can be used to confirm selection in gaze-based interaction with computers as a substitute for the computer mouse click when manual confirmation cannot be used.
2016
1.
Величковский, Б. М.; Нуждин, Ю. О.; Свирин, Е. П.; Строганова, Т. А.; Федорова, А. А.; Шишкин, С. Л. (2016). Управление «силой мысли»: На пути к новым формам взаимодействия человека с техническими устройствами. Вопросы психологии, 62(1), 78-88.
@bachelorthesis{nokey,
title = {Управление «силой мысли»: На пути к новым формам взаимодействия человека с техническими устройствами},
author = {Величковский, Б.М. and Нуждин, Ю.О. and Свирин, Е.П. and Строганова, Т.А. and Федорова, А.А. and Шишкин, С.Л.},
year = {2016},
date = {2016-01-00},
journal = {Вопросы психологии},
volume = {62},
number = {1},
pages = {78-88},
abstract = {Рассмотрены две группы перспективных интерфейсов, создаваемых сегодня для улучшения взаимодействия человека с техническими системами. Интерфейсы первой группы опираются на использование данных, получаемых с помощью методов нейрофизиологии и психофизиологии, прежде всего путем регистрации ЭЭГ/магнитоэнцефалограммы. Интерфейсы второй группы используют данные о микроповедении глаза человека, опираясь на методы айтрекинга, популярные в психологии и эргономике. Продемонстрирована полезность совмещения этих подходов для создания высокоскоростных гибридных интерфейсов, пригодных не только для обеспечения коммуникации лиц с тяжелыми нарушениями речи и моторики, но и для повышения эффективности работы здоровых пользователей операторских профессий. Критическую роль в таком развитии имеет решение ряда фундаментальных проблем психологической науки, таких как поддержка зон совместного внимания в процессах опосредствованного техникой общения и выявление намерений пользователя по характеристикам движений глаз и мозговой активности.},
keywords = {},
pubstate = {published},
tppubtype = {bachelorthesis}
}
Рассмотрены две группы перспективных интерфейсов, создаваемых сегодня для улучшения взаимодействия человека с техническими системами. Интерфейсы первой группы опираются на использование данных, получаемых с помощью методов нейрофизиологии и психофизиологии, прежде всего путем регистрации ЭЭГ/магнитоэнцефалограммы. Интерфейсы второй группы используют данные о микроповедении глаза человека, опираясь на методы айтрекинга, популярные в психологии и эргономике. Продемонстрирована полезность совмещения этих подходов для создания высокоскоростных гибридных интерфейсов, пригодных не только для обеспечения коммуникации лиц с тяжелыми нарушениями речи и моторики, но и для повышения эффективности работы здоровых пользователей операторских профессий. Критическую роль в таком развитии имеет решение ряда фундаментальных проблем психологической науки, таких как поддержка зон совместного внимания в процессах опосредствованного техникой общения и выявление намерений пользователя по характеристикам движений глаз и мозговой активности.