TY - JOUR
T1 - Copilot
T2 - A framework for integrating LLM and BMI to enhance human–robot interaction
AU - Liu, Siyu
AU - Liu, Mengzhen
AU - Ming, Zhiyuan
AU - Huang, Yilun
AU - Ma, Lingfei
AU - Zhang, Deyu
AU - Song, Yifan
AU - Zhang, Jian
AU - Yan, Tianyi
N1 - Publisher Copyright:
© 2026 Published by Elsevier Ltd.
PY - 2026/10
Y1 - 2026/10
N2 - This paper proposes an innovative human–robot interaction (HRI) framework called Copilot, which aims to bridge the gap between human intent and robot intelligence. Currently, existing HRI systems struggle to infer human intentions and rely heavily on predefined rules, a limitation that significantly hinders the advancement of the field. To address this issue, the Copilot framework, for the first time, integrates the environmental understanding capabilities of large language models (LLMs) with the intention recognition advantages of brain-machine interface (BMI). It constructs three core modules: (1) a LLM-based visual evoked potential (LLM-VEP) paradigm module utilizing LLM for scene understanding and dynamic marking; (2) a BMI module employing the blink-triggered multivariate variational mode decomposition with canonical correlation analysis (BT-MVMD-CCA) algorithm; and (3) an intelligent agent flexibly adapting to different task requirements. Through online experimental validation with 12 participants, the system performed optimally when using the EEG-based double blink triggering (EEG-DBT) method: 0% false trigger rate, 94.09% blink detection rate, and 84.00% task completion rate. In offline experiments, the proposed BT-MVMD-CCA algorithm achieved 92.3% classification accuracy and a peak information transfer rate (ITR) of 71.1 bits/min at DTW = 1.5 s. This research not only provides theoretical support for the HRI field, but also offers promising solutions for assistive robotics and manufacturing scenarios.
AB - This paper proposes an innovative human–robot interaction (HRI) framework called Copilot, which aims to bridge the gap between human intent and robot intelligence. Currently, existing HRI systems struggle to infer human intentions and rely heavily on predefined rules, a limitation that significantly hinders the advancement of the field. To address this issue, the Copilot framework, for the first time, integrates the environmental understanding capabilities of large language models (LLMs) with the intention recognition advantages of brain-machine interface (BMI). It constructs three core modules: (1) a LLM-based visual evoked potential (LLM-VEP) paradigm module utilizing LLM for scene understanding and dynamic marking; (2) a BMI module employing the blink-triggered multivariate variational mode decomposition with canonical correlation analysis (BT-MVMD-CCA) algorithm; and (3) an intelligent agent flexibly adapting to different task requirements. Through online experimental validation with 12 participants, the system performed optimally when using the EEG-based double blink triggering (EEG-DBT) method: 0% false trigger rate, 94.09% blink detection rate, and 84.00% task completion rate. In offline experiments, the proposed BT-MVMD-CCA algorithm achieved 92.3% classification accuracy and a peak information transfer rate (ITR) of 71.1 bits/min at DTW = 1.5 s. This research not only provides theoretical support for the HRI field, but also offers promising solutions for assistive robotics and manufacturing scenarios.
KW - Brain-machine interface
KW - Human–robot interaction
KW - Large language model
KW - Manufacturing
UR - https://www.scopus.com/pages/publications/105033335653
U2 - 10.1016/j.rcim.2026.103291
DO - 10.1016/j.rcim.2026.103291
M3 - Article
AN - SCOPUS:105033335653
SN - 0736-5845
VL - 101
JO - Robotics and Computer-Integrated Manufacturing
JF - Robotics and Computer-Integrated Manufacturing
M1 - 103291
ER -