Interpreting multi-agent reinforcement learning decisions via key feature activation

Peizhang Li; Qing Fei; Zhen Chen; Zhongqi Sun; Bo Wang

doi:10.1016/j.neucom.2026.133080

Interpreting multi-agent reinforcement learning decisions via key feature activation

Peizhang Li
, Qing Fei^*
, Zhen Chen
, Zhongqi Sun
, Bo Wang

^*此作品的通讯作者

Beijing Institute of Technology
China Academy of Safety Science and Technology
China State Shipbuilding Corporation

科研成果: 期刊稿件 › 文章 › 同行评审

摘要

Deep Reinforcement Learning (DRL) techniques have demonstrated remarkable performance in autonomous decision-making for unmanned systems, yet their lack of interpretability poses a significant challenge to trust and confidence in real-world applications. Explainable Reinforcement Learning (XRL) is increasingly recognized as a critical technology for addressing this concern. However, existing XRL approaches have some limitations, such as dependence on external data for indirect interpretation and the requirement of additional information encoding for direct interpretation. To overcome these challenges, this work proposes the Key Feature Activation Interpretation (KFAI) network, which transparently vectorizes the agent's states and actions, forming interpretable linear combinations through routing between them. KFAI replaces the opaque decision pathways typical in DRL with an interpretable activation mapping between states and actions, allowing agents to directly output interpretable deterministic policies. Leveraging the KFAI, we develop the multi-agent key-feature reinforcement learning network for multi-agent systems, enabling reliable autonomous decision-making in unmanned swarms. To evaluate the effectiveness of the proposed networks, a comparative study of interpretable modifications for various continuous and discrete MARL algorithms was conducted in simulated environments. The experimental results demonstrate that the methods endowed with interpretability through this modification exhibit superior performance on decision-making tasks when compared to the original baseline networks. Furthermore, we evaluate the interpretability of the proposed method along multiple dimensions, including sparsity and faithfulness, and design a novel visualization technique that demonstrates how multi-agent decision-making processes can be explained in both offline and real-time settings.

源语言	英语
文章编号	133080
期刊	Neurocomputing
卷	677
DOI	https://doi.org/10.1016/j.neucom.2026.133080
出版状态	已出版 - 7 5月 2026
已对外发布	是

访问文件

10.1016/j.neucom.2026.133080

其它文件与链接

链接到 Scopus 的出版物

引用此

@article{76c4cd22f1424853a3ab7d10dacab052,

title = "Interpreting multi-agent reinforcement learning decisions via key feature activation",

abstract = "Deep Reinforcement Learning (DRL) techniques have demonstrated remarkable performance in autonomous decision-making for unmanned systems, yet their lack of interpretability poses a significant challenge to trust and confidence in real-world applications. Explainable Reinforcement Learning (XRL) is increasingly recognized as a critical technology for addressing this concern. However, existing XRL approaches have some limitations, such as dependence on external data for indirect interpretation and the requirement of additional information encoding for direct interpretation. To overcome these challenges, this work proposes the Key Feature Activation Interpretation (KFAI) network, which transparently vectorizes the agent's states and actions, forming interpretable linear combinations through routing between them. KFAI replaces the opaque decision pathways typical in DRL with an interpretable activation mapping between states and actions, allowing agents to directly output interpretable deterministic policies. Leveraging the KFAI, we develop the multi-agent key-feature reinforcement learning network for multi-agent systems, enabling reliable autonomous decision-making in unmanned swarms. To evaluate the effectiveness of the proposed networks, a comparative study of interpretable modifications for various continuous and discrete MARL algorithms was conducted in simulated environments. The experimental results demonstrate that the methods endowed with interpretability through this modification exhibit superior performance on decision-making tasks when compared to the original baseline networks. Furthermore, we evaluate the interpretability of the proposed method along multiple dimensions, including sparsity and faithfulness, and design a novel visualization technique that demonstrates how multi-agent decision-making processes can be explained in both offline and real-time settings.",

keywords = "Deterministic policy, Explainable reinforcement learning, Key feature activation, Multi-agent systems",

author = "Peizhang Li and Qing Fei and Zhen Chen and Zhongqi Sun and Bo Wang",

note = "Publisher Copyright: {\textcopyright} 2026 Elsevier B.V.",

year = "2026",

month = may,

day = "7",

doi = "10.1016/j.neucom.2026.133080",

language = "English",

volume = "677",

journal = "Neurocomputing",

issn = "0925-2312",

publisher = "Elsevier B.V.",

}

TY - JOUR

T1 - Interpreting multi-agent reinforcement learning decisions via key feature activation

AU - Li, Peizhang

AU - Fei, Qing

AU - Chen, Zhen

AU - Sun, Zhongqi

AU - Wang, Bo

PY - 2026/5/7

Y1 - 2026/5/7

N2 - Deep Reinforcement Learning (DRL) techniques have demonstrated remarkable performance in autonomous decision-making for unmanned systems, yet their lack of interpretability poses a significant challenge to trust and confidence in real-world applications. Explainable Reinforcement Learning (XRL) is increasingly recognized as a critical technology for addressing this concern. However, existing XRL approaches have some limitations, such as dependence on external data for indirect interpretation and the requirement of additional information encoding for direct interpretation. To overcome these challenges, this work proposes the Key Feature Activation Interpretation (KFAI) network, which transparently vectorizes the agent's states and actions, forming interpretable linear combinations through routing between them. KFAI replaces the opaque decision pathways typical in DRL with an interpretable activation mapping between states and actions, allowing agents to directly output interpretable deterministic policies. Leveraging the KFAI, we develop the multi-agent key-feature reinforcement learning network for multi-agent systems, enabling reliable autonomous decision-making in unmanned swarms. To evaluate the effectiveness of the proposed networks, a comparative study of interpretable modifications for various continuous and discrete MARL algorithms was conducted in simulated environments. The experimental results demonstrate that the methods endowed with interpretability through this modification exhibit superior performance on decision-making tasks when compared to the original baseline networks. Furthermore, we evaluate the interpretability of the proposed method along multiple dimensions, including sparsity and faithfulness, and design a novel visualization technique that demonstrates how multi-agent decision-making processes can be explained in both offline and real-time settings.

AB - Deep Reinforcement Learning (DRL) techniques have demonstrated remarkable performance in autonomous decision-making for unmanned systems, yet their lack of interpretability poses a significant challenge to trust and confidence in real-world applications. Explainable Reinforcement Learning (XRL) is increasingly recognized as a critical technology for addressing this concern. However, existing XRL approaches have some limitations, such as dependence on external data for indirect interpretation and the requirement of additional information encoding for direct interpretation. To overcome these challenges, this work proposes the Key Feature Activation Interpretation (KFAI) network, which transparently vectorizes the agent's states and actions, forming interpretable linear combinations through routing between them. KFAI replaces the opaque decision pathways typical in DRL with an interpretable activation mapping between states and actions, allowing agents to directly output interpretable deterministic policies. Leveraging the KFAI, we develop the multi-agent key-feature reinforcement learning network for multi-agent systems, enabling reliable autonomous decision-making in unmanned swarms. To evaluate the effectiveness of the proposed networks, a comparative study of interpretable modifications for various continuous and discrete MARL algorithms was conducted in simulated environments. The experimental results demonstrate that the methods endowed with interpretability through this modification exhibit superior performance on decision-making tasks when compared to the original baseline networks. Furthermore, we evaluate the interpretability of the proposed method along multiple dimensions, including sparsity and faithfulness, and design a novel visualization technique that demonstrates how multi-agent decision-making processes can be explained in both offline and real-time settings.

KW - Deterministic policy

KW - Explainable reinforcement learning

KW - Key feature activation

KW - Multi-agent systems

UR - https://www.scopus.com/pages/publications/105030451651

U2 - 10.1016/j.neucom.2026.133080

DO - 10.1016/j.neucom.2026.133080

M3 - Article

AN - SCOPUS:105030451651

SN - 0925-2312

VL - 677

JO - Neurocomputing

JF - Neurocomputing

M1 - 133080

ER -

Interpreting multi-agent reinforcement learning decisions via key feature activation

摘要

访问文件

其它文件与链接

学术指纹

引用此