Multi-agent reinforcement learning and graph attention network: an end-to-end solution for UAV cluster conflict resolution

In previous article, we sorted out the algorithm panorama of UAV conflict resolution. Among them, reinforcement learning (especially MARL) is labeled as the “most realistic option” for swarms of 50+ drones. This article will focus on this route, starting from the foundation of single-agent RL, going into the core challenges of multi-agent scenarios, analyzing mainstream algorithms such as MADDPG, QMIX, COMA, and MAPPO, and focusing on how GAT (Graph Attention Network) provides MARL with scalable topology awareness capabilities, and ultimately achieves an end-to-end conflict resolution strategy.

---

1. From single agent to multi-agent: Why is MARL so difficult?

1.1 Single-agent RL review

Let’s start with the familiar single-agent RL. The single-agent MDP is described by the quadruple :

• State value function:
• Action value function:
• Optimal strategy:

The core assumption of single-agent RL: The environment is stable - no matter how many episodes you train, the dynamics of the environment always remain unchanged.

1.2 Three essential difficulties of multi-agent

Multi-agent scenarios break this assumption and bring about three fundamental difficulties:

① Environmental non-stationarity (Non-Stationarity)

When agent is learning policy , the policies of other agents are also changing. This means:$$ \mathcal{P}_i(s’\mid s, a_1,\dots,a_n) \neq \mathcal{P}_i(s’\mid s, a_1,\dots,a_n, a_1’,\dots,a_n’)

r_t = f(\mathbf{s}_t, \mathbf{a}t, \mathbf{s}{t+1})

\nabla_{\theta_i} J(\theta_i) = \mathbb{E}{\mathbf{s} \sim \mathcal{D}}\left[ \nabla{\theta_i} \log \pi_i(a_i \mid o_i) \cdot Q_i^\pi(\mathbf{s}, a_1, \dots, a_n) \Big|_{a_i = \pi_i(o_i)} \right]

Where is the Action-Observation Trajectory of agent , is a Monotonic Mixing Network, satisfying:

The monotonicity constraint guarantees a key property: During decentralized execution, each agent’s independent greedy maximization of is equivalent to the global maximization of .

class QMIXMixingNetwork(nn.Module):
    """
    单调混合网络：将各智能体的 Q_i 混合为全局 Q_tot
    关键约束：所有权值非负（保证单调性）
    """
    def __init__(self, n_agents, embed_dim=64):
        super().__init__()
        # Hyper-network 生成混合网络的权值
        self.hyper_w1 = nn.Sequential(
            nn.Linear(n_agents, embed_dim),
            nn.ReLU(),
            nn.Linear(embed_dim, n_agents * embed_dim),  # 输出 (n_agents × embed_dim) 权值
        )
        self.hyper_b1 = nn.Linear(n_agents, embed_dim)
        self.hyper_w2 = nn.Sequential(
            nn.Linear(embed_dim, embed_dim),
            nn.ReLU(),
            nn.Linear(embed_dim, embed_dim)
        )
        self.hyper_b2 = nn.Linear(embed_dim, 1)
    
    def forward(self, q_values, state):
        """
        q_values: (batch, n_agents) 各智能体的 Q 值
        state: (batch, state_dim) 全局状态（用于生成 hyper-network 输入）
        """
        batch_size = q_values.size(0)
        
        # 第一层：W₁ * Q + b₁
        w1 = torch.abs(self.hyper_w1(state))          # (batch, n_agents * embed_dim)
        w1 = w1.view(batch_size, q_values.size(1), -1)  # (batch, n_agents, embed_dim)
        b1 = self.hyper_b1(state).unsqueeze(1)       # (batch, 1, embed_dim)
        
        q_hidden = torch.relu(torch.bmm(q_values.unsqueeze(1), w1) + b1)  # (batch, 1, embed_dim)
        
        # 第二层：W₂ * h + b₂
        w2 = torch.abs(self.hyper_w2(q_hidden.squeeze(1)))  # (batch, embed_dim, embed_dim)
        b2 = self.hyper_b2(q_hidden.squeeze(1)).unsqueeze(1)  # (batch, 1, 1)
        
        q_tot = torch.bmm(q_hidden, w2.unsqueeze(1)) + b2  # (batch, 1, 1)
        return q_tot.squeeze(-1)  # (batch,)


class QMIXAgent:
    """QMIX 算法"""
    def __init__(self, obs_dim, action_dim, n_agents, agent_id):
        self.agent_id = agent_id
        self.action_dim = action_dim
        
        # 每个智能体的 RNN（处理动作-观测历史）
        self.rnn = nn.GRUCell(obs_dim + action_dim, obs_dim)
        # Q 网络
        self.q_net = nn.Sequential(
            nn.Linear(obs_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 64),
            nn.ReLU(),
            nn.Linear(64, action_dim)
        )
        
        self.target_rnn = nn.GRUCell(obs_dim + action_dim, obs_dim)
        self.target_q_net = nn.Sequential(
            nn.Linear(obs_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 64),
            nn.ReLU(),
            nn.Linear(64, action_dim)
        )
        self.hard_update()
    
    def hard_update(self):
        self.target_rnn.load_state_dict(self.rnn.state_dict())
        self.target_q_net.load_state_dict(self.q_net.state_dict())
    
    def get_q_values(self, hidden, obs, last_action):
        """给定 (hidden, obs, last_action) 输出 Q(s,a)"""
        rnn_input = torch.cat([obs, last_action], dim=1)
        new_hidden = self.rnn(rnn_input, hidden)
        q_values = self.q_net(new_hidden)
        return q_values, new_hidden
    
    def select_action_epsilon_greedy(self, q_values, epsilon):
        """ε-贪心策略"""
        if random.random() < epsilon:
            return random.randint(0, self.action_dim - 1)
        return q_values.argmax(dim=1).item()


def train_qmix():
    """QMIX 训练循环（伪代码）"""
    n_agents = 8
    n_episodes = 50000
    
    agents = [QMIXAgent(obs_dim=12, action_dim=5, n_agents=n_agents, agent_id=i)
              for i in range(n_agents)]
    mixer = QMIXMixingNetwork(n_agents)
    
    optimizers = [optim.Adam(agent.q_net.parameters(), lr=2e-4) for agent in agents]
    mixer_optimizer = optim.Adam(mixer.parameters(), lr=2e-4)
    
    replay = ReplayBuffer(capacity=100000)
    
    for ep in range(n_episodes):
        # 环境交互
        states = env.reset()  # (n_agents, obs_dim)
        hidden = [torch.zeros(1, 12) for _ in range(n_agents)]
        last_actions = [torch.zeros(1, 5) for _ in range(n_agents)]
        episode_reward = 0
        
        while not done:
            actions = []
            for i, agent in enumerate(agents):
                q_vals, hidden[i] = agent.get_q_values(hidden[i],
                    torch.FloatTensor(states[i]).unsqueeze(0),
                    last_actions[i])
                a = agent.select_action_epsilon_greedy(q_vals.squeeze(0), epsilon=0.1)
                actions.append(a)
                last_actions[i] = torch.zeros(1, 5)
                last_actions[i][0, a] = 1.0
            
            next_states, rewards, done = env.step(actions)
            replay.push(states, last_actions, rewards, next_states, done)
            states = next_states
            episode_reward += sum(rewards)
        
        # 学习
        if len(replay) > 1024:
            batch = replay.sample(32)
            # QMIX 损失计算 ...
            # 单调混合 + 中心化训练 ...

2.4 MAPPO: The victory of policy gradient in highly parallel scenarios

MAPPO (Multi-Agent PPO) extends the PPO algorithm to multi-agent scenarios and has performed well in UAV cluster tasks in recent years (multiple top conference papers from 2022 to 2024).

Core advantages of PPO: Trust Region constraints ensure training stability and avoid the hyperparameter disaster of the DDPG series.

PPO -clip target:

Where is the probability ratio, is GAE (Generalized Advantage Estimation).Typical configuration of MAPPO in UAV conflict resolution:

Parameters	Recommended values	Description
Clip ratio	0.2	PPO default
Horizon	128–256	Number of rollout steps per epoch
PPO epochs	2–4	Number of repeated updates per batch
GAE	0.95	Bias-variance balance of dominance estimate
Hidden layer dimension	64–128	Sufficient for UAV scenarios
Normalization	OBS + Reward Normalization	Key! Great impact on multi-agent convergence

---

3. GAT: Let MARL learn “who to follow”

3.1 Why does MARL need a graph structure?

In a UAV cluster, not all agents are equally important. Take conflict resolution as an example:

• UAV about to collide with me → High Concern
• UAVs far out of sight → can be ignored
• Approaching moving obstacles → requires dynamic attention

However, traditional MARL (such as MADDPG, QMIX) treats all neighbors equally: either fully connected ( communication), or fixed topology (such as ring, nearest neighbor).

The introduction of GAT solves two core problems:

1. Adaptive Neighbor Weight: Learn which neighbors are more important to the current decision through the attention mechanism
2. Scalability: Does not increase with the number of drones and supports dynamic topology

3.2 GAT core principles

GAT performs Neighbor Aggregation on the features of node at each layer, and the weight is dynamically calculated by the attention mechanism:$$ \alpha_{ij} = \frac{\exp\left(\text{LeakyReLU}\left(\mathbf{a}^\top[\mathbf{W}\mathbf{h}_i \Vert \mathbf{W}\mathbf{h}j]\right)\right)} {\sum{k \in \mathcal{N}_i} \exp\left(\text{LeakyReLU}\left(\mathbf{a}^\top[\mathbf{W}\mathbf{h}_i \Vert \mathbf{W}\mathbf{h}_k]\right)\right)}

\mathbf{h}i’ = \sigma\left(\sum{j \in \mathcal{N}i} \alpha{ij} \mathbf{W}\mathbf{h}_j\right)

\pi_{safe}(s) = \text{Proj}{\mathcal{A}{safe}(s)} \pi(s)