Multi-Agent Risks from Advanced AI

Lewis Hammond; Alan Chan; Jesse Clifton; Jason Hoelscher-Obermaier; Akbir Khan; Euan McLean; Chandler Smith; Wolfram Barfuss; Jakob Foerster; Tomáš Gavenčiak; The Anh Han; Edward Hughes; Vojtěch Kovařík; Jan Kulveit; Joel Z. Leibo; Caspar Oesterheld; Christian Schroeder de Witt; Nisarg Shah; Michael Wellman; Paolo Bova; Theodor Cimpeanu; Carson Ezell; Quentin Feuillade-Montixi; Matija Franklin; Esben Kran; Igor Krawczuk; Max Lamparth; Niklas Lauffer; Alexander Meinke; Sumeet Motwani; Anka Reuel; Vincent Conitzer; Michael Dennis; Iason Gabriel; Adam Gleave; Gillian Hadfield; Nika Haghtalab; Atoosa Kasirzadeh; Sébastien Krier; Kate Larson; Joel Lehman; David C. Parkes; Georgios Piliouras; Iyad Rahwan

doi:10.48550/arXiv.2502.14143

Multi-Agent Risks from Advanced AI

Lewis Hammond, Alan Chan, Jesse Clifton, Jason Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenčiak, The Anh Han, Edward Hughes, Vojtěch Kovařík, Jan Kulveit, Joel Z. Leibo, Caspar Oesterheld, Christian Schroeder de Witt, Nisarg Shah, Michael Wellman, Paolo BovaTheodor Cimpeanu, Carson Ezell, Quentin Feuillade-Montixi, Matija Franklin, Esben Kran, Igor Krawczuk, Max Lamparth, Niklas Lauffer, Alexander Meinke, Sumeet Motwani, Anka Reuel, Vincent Conitzer, Michael Dennis, Iason Gabriel, Adam Gleave, Gillian Hadfield, Nika Haghtalab, Atoosa Kasirzadeh, Sébastien Krier, Kate Larson, Joel Lehman, David C. Parkes, Georgios Piliouras, Iyad Rahwan

Research output: Working paper › Preprint

26 Downloads (Pure)

Abstract

The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI.

Original language	English
Publisher	arXiv
Pages	1-96
Number of pages	96
DOIs	https://doi.org/10.48550/arXiv.2502.14143
Publication status	Published - 19 Feb 2025

Bibliographical note

Cooperative AI Foundation, Technical Report #1

Access to Document

10.48550/arXiv.2502.14143Licence: CC BY

Full Text V1Final published version, 3.87 MBLicence: CC BY

Cite this

Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., Han, T. A., Hughes, E., Kovařík, V., Kulveit, J., Leibo, J. Z., Oesterheld, C., de Witt, C. S., Shah, N., Wellman, M., ... Rahwan, I. (2025). Multi-Agent Risks from Advanced AI. (pp. 1-96). arXiv. https://doi.org/10.48550/arXiv.2502.14143

@techreport{277067ee85c64dce9b84a9d4edae49f7,

title = "Multi-Agent Risks from Advanced AI",

abstract = "The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI.",

author = "Lewis Hammond and Alan Chan and Jesse Clifton and Jason Hoelscher-Obermaier and Akbir Khan and Euan McLean and Chandler Smith and Wolfram Barfuss and Jakob Foerster and Tom{\'a}{\v s} Gaven{\v c}iak and Han, \{The Anh\} and Edward Hughes and Vojt{\v e}ch Kova{\v r}{\'i}k and Jan Kulveit and Leibo, \{Joel Z.\} and Caspar Oesterheld and \{de Witt\}, \{Christian Schroeder\} and Nisarg Shah and Michael Wellman and Paolo Bova and Theodor Cimpeanu and Carson Ezell and Quentin Feuillade-Montixi and Matija Franklin and Esben Kran and Igor Krawczuk and Max Lamparth and Niklas Lauffer and Alexander Meinke and Sumeet Motwani and Anka Reuel and Vincent Conitzer and Michael Dennis and Iason Gabriel and Adam Gleave and Gillian Hadfield and Nika Haghtalab and Atoosa Kasirzadeh and S{\'e}bastien Krier and Kate Larson and Joel Lehman and Parkes, \{David C.\} and Georgios Piliouras and Iyad Rahwan",

note = "Cooperative AI Foundation, Technical Report \#1",

year = "2025",

month = feb,

day = "19",

doi = "10.48550/arXiv.2502.14143",

language = "English",

pages = "1--96",

publisher = "arXiv",

type = "WorkingPaper",

institution = "arXiv",

}

Hammond, L, Chan, A, Clifton, J, Hoelscher-Obermaier, J, Khan, A, McLean, E, Smith, C, Barfuss, W, Foerster, J, Gavenčiak, T, Han, TA, Hughes, E, Kovařík, V, Kulveit, J, Leibo, JZ, Oesterheld, C, de Witt, CS, Shah, N, Wellman, M, Bova, P, Cimpeanu, T, Ezell, C, Feuillade-Montixi, Q, Franklin, M, Kran, E, Krawczuk, I, Lamparth, M, Lauffer, N, Meinke, A, Motwani, S, Reuel, A, Conitzer, V, Dennis, M, Gabriel, I, Gleave, A, Hadfield, G, Haghtalab, N, Kasirzadeh, A, Krier, S, Larson, K, Lehman, J, Parkes, DC, Piliouras, G & Rahwan, I 2025 'Multi-Agent Risks from Advanced AI' arXiv, pp. 1-96. https://doi.org/10.48550/arXiv.2502.14143

TY - UNPB

T1 - Multi-Agent Risks from Advanced AI

AU - Hammond, Lewis

AU - Chan, Alan

AU - Clifton, Jesse

AU - Hoelscher-Obermaier, Jason

AU - Khan, Akbir

AU - McLean, Euan

AU - Smith, Chandler

AU - Barfuss, Wolfram

AU - Foerster, Jakob

AU - Gavenčiak, Tomáš

AU - Han, The Anh

AU - Hughes, Edward

AU - Kovařík, Vojtěch

AU - Kulveit, Jan

AU - Leibo, Joel Z.

AU - Oesterheld, Caspar

AU - de Witt, Christian Schroeder

AU - Shah, Nisarg

AU - Wellman, Michael

AU - Bova, Paolo

AU - Cimpeanu, Theodor

AU - Ezell, Carson

AU - Feuillade-Montixi, Quentin

AU - Franklin, Matija

AU - Kran, Esben

AU - Krawczuk, Igor

AU - Lamparth, Max

AU - Lauffer, Niklas

AU - Meinke, Alexander

AU - Motwani, Sumeet

AU - Reuel, Anka

AU - Conitzer, Vincent

AU - Dennis, Michael

AU - Gabriel, Iason

AU - Gleave, Adam

AU - Hadfield, Gillian

AU - Haghtalab, Nika

AU - Kasirzadeh, Atoosa

AU - Krier, Sébastien

AU - Larson, Kate

AU - Lehman, Joel

AU - Parkes, David C.

AU - Piliouras, Georgios

AU - Rahwan, Iyad

N1 - Cooperative AI Foundation, Technical Report #1

PY - 2025/2/19

Y1 - 2025/2/19

N2 - The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI.

AB - The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI.

UR - https://arxiv.org/abs/2502.14143v1

U2 - 10.48550/arXiv.2502.14143

DO - 10.48550/arXiv.2502.14143

M3 - Preprint

SP - 1

EP - 96

BT - Multi-Agent Risks from Advanced AI

PB - arXiv

ER -

Multi-Agent Risks from Advanced AI

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

EPSRC standard grant "Evolutionary Game Theory Under Uncertainty"

Cite this

Multi-Agent Risks from Advanced AI

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Projects

EPSRC standard grant "Evolutionary Game Theory Under Uncertainty"

Cite this