In Internet of Things (IoT) status update systems, where information is sampled and subsequently transmitted from a source to a destination node, the imperative necessity lies in maintaining the timeliness of information and updating the system with optimal frequency. Optimizing information freshness in resource-limited status update systems often involves Constrained Markov Decision Process (CMDP) problems with update rate constraints. Solving CMDP problems, especially with multiple constraints, is a challenging task. To address this, we present a token-based approach that transforms CMDP into an unconstrained MDP, simplifying the solution process. We apply this approach to systems with one and two update rate constraints for optimizing Age of Incorrect Information (AoII) and Age of Information (AoI) metrics, respectively, and explore the analytical and numerical aspects. Additionally, we introduce an iterative triangle bisection method for solving the CMDP problems with two constraints, comparing its results with the token-based MDP approach. Our findings show that the token-based approach yields superior performance over baseline policies, converging to the optimal policy as the maximum number of tokens increases.

翻译：在物联网（IoT）状态更新系统中，信息被采样并从源节点传输到目的节点，维护信息的时效性并以最优频率更新系统具有迫切必要性。在资源受限的状态更新系统中优化信息新鲜度通常涉及带有更新率约束的约束马尔可夫决策过程（CMDP）问题。求解CMDP问题（特别是多约束情形）是一项具有挑战性的任务。为此，我们提出一种令牌方法，将CMDP转化为无约束MDP，从而简化求解过程。我们将该方法分别应用于具有一个和两个更新率约束的系统，以优化错误信息年龄（AoII）和信息年龄（AoI）指标，并探讨了分析与数值方面的特性。此外，我们引入了一种用于求解双约束CMDP问题的迭代三角二分法，并将其结果与基于令牌的MDP方法进行了比较。实验结果表明，与基线策略相比，令牌方法性能更优，且随着最大令牌数量的增加，该方法能收敛至最优策略。