Nonlinear effects of higher-order interactions on pattern formation and species persistence in cyclic ecosystems

¹ School of Statistics and Mathematics, Yunnan University of Finance and Economics Kunming, Yunnan 650221, China
² Department of Applied Mathematics, College of Applied Sciences, Kyung Hee University Yongin 17104, Republic of Korea
³ G-LAMP NEXUS Institute, Kyung Hee University - Yongin 17104, Republic of Korea

Received: 13 February 2026
Accepted: 26 May 2026

Abstract

Recent studies have highlighted the critical role of higher-order interactions (HOIs) in shaping the dynamics and biodiversity of complex ecosystems, particularly those modeled by the rock-paper-scissors (RPS) game. Although RPS systems have provided a classic framework for understanding cyclic competition and have been extended to include HOIs, the specific effects of these interactions on biodiversity in increasingly complex systems remain uncertain. In this paper, we examine how HOIs affect species dynamics in the spatial RPS system, linking HOIs to the local proportion of natural enemies. Using Monte Carlo simulations, we investigate how HOIs and mobility shape species evolution at both macroscopic and microscopic levels. Our simulations demonstrate that increasing the regulatory weight of HOIs reduces vacancies and sharpens the spiral patterns formed by the three species, which are sensitive to mobility. Basin entropy analysis further indicates that the regulatory weight of HOIs alters system unpredictability, which increases with stronger HOI effects. Additionally, extinction probability —a classic measure of biodiversity robustness— shows that HOIs can reduce extinction risk in high-mobility regimes by adjusting their weight and intensity. Our findings underscore the essential role of HOIs in sustaining biodiversity and suggest that our approaches can be extended to analyze multi-species systems with environmental heterogeneity.