Agents in many multi-agent systems must balance several, possibly conflicting, objectives, and the trade-offs an agent can achieve depend on how the other agents behave. We study this setting through the Multi-Objective Markov Game (MOMG), a Markov game in which each agent receives a vector of rewards, and adopt the Pareto-Nash Equilibrium (PNE) as the solution concept: at a PNE, no agent can unilaterally improve one of its objectives without sacrificing another. We show that PNEs always exist and coincide with the Nash equilibria of the linearly scalarized games with strictly positive weights, so standard single-objective Markov-game solvers suffice to compute them. We further define the Weak Pareto-Nash Equilibrium (WPNE), which excludes only unilateral deviations that strictly improve all objectives at once. The associated gap (WPNG) is zero exactly at WPNEs and never exceeds the Nash gap of any linearly scalarized game; hence an approximate Nash equilibrium of a scalarized game is also an approximate WPNE. When the preference profile is known in advance, learning in an MOMG therefore reduces to learning in a single scalarized Markov game, for which we provide a sample-efficient algorithm. When preferences are revealed only after data collection, we design a two-phase algorithm that first explores without any preference information; with high probability, every Nash equilibrium of the learned scalarized game is then an approximate WPNE for every queried preference profile, and the WPNG of any policy can be estimated accurately from the learned model. Experiments on tabular games and four multi-objective multi-agent environments illustrate the theory and show the effect of preference-free exploration.

翻译：暂无翻译