In Internet of Things (IoT) status update systems, where information is sampled and subsequently transmitted from a source to a destination node, the imperative necessity lies in maintaining the timeliness of information and updating the system with optimal frequency. Optimizing information freshness in resource-limited status update systems often involves Constrained Markov Decision Process (CMDP) problems with update rate constraints. Solving CMDP problems, especially with multiple constraints, is a challenging task. To address this, we present a token-based approach that transforms CMDP into an unconstrained MDP, simplifying the solution process. We apply this approach to systems with one and two update rate constraints for optimizing Age of Incorrect Information (AoII) and Age of Information (AoI) metrics, respectively, and explore the analytical and numerical aspects. Additionally, we introduce an iterative triangle bisection method for solving the CMDP problems with two constraints, comparing its results with the token-based MDP approach. Our findings show that the token-based approach yields superior performance over baseline policies, converging to the optimal policy as the maximum number of tokens increases.

翻译：在物联网状态更新系统中，信息从源节点采样并传输至目的节点，其核心需求在于保持信息的时效性并以最优频率更新系统。在资源受限的状态更新系统中优化信息新鲜度通常涉及带有更新率约束的约束马尔可夫决策过程问题。求解CMDP问题，尤其是具有多重约束的情况，是一项具有挑战性的任务。为此，我们提出一种基于令牌的方法，将CMDP转化为无约束MDP，从而简化求解过程。我们将该方法分别应用于具有单约束和双约束的系统，以优化错误信息年龄和信息年龄指标，并探讨其解析与数值特性。此外，我们引入一种迭代三角形二分法来求解具有双重约束的CMDP问题，并将其结果与基于令牌的MDP方法进行比较。研究结果表明，基于令牌的方法相比基线策略具有更优性能，且随着最大令牌数量的增加收敛至最优策略。