Liu and Zhang

PCL for RMAB with General Observation Models

PCL-Indexability and Whittle Index for Restless Bandits with General Observation Models

Keqin Liu and Chengzhong Zhang National Center for Applied Mathematics, Nanjing, China, 210093. kqliu@nju.edu.cn

In this paper, we consider a general observation model for restless multi-armed bandit problems. The operation of the player needs to be based on certain feedback mechanism that is error-prone due to resource constraints or environmental or intrinsic noises. By establishing a general probabilistic model for dynamics of feedback/observation, we formulate the problem as a restless bandit with a countable belief state space starting from an arbitrary initial belief (a priori information). We apply the achievable region method with partial conservation law (PCL) to the infinite-state problem and analyze its indexability and priority index (Whittle index). Finally, we propose an approximation process to transform the problem into which the AG algorithm of Niño-Mora and Bertsimas for finite-state problems can be applied to. Numerical experiments show that our algorithm has an excellent performance.

restless bandit, POMDP, countable state space, partial conservation law, Whittle index This paper was first submitted on July 3rd, 2024.

1 Introduction

Multi-armed bandit (MAB) is a classic operations research problem that involves a learner making choices among actions with uncertain/random rewards in order to maximize the expected return in long-run. The MAB problem is often associated with the important exploration-exploitation tradeoff and was initially proposed by Robbins (1952). Starting from the classic stochastic scheduling problem, the ongoing development of various MAB models makes it applicable to a wide range of practical fields, including clinical trials, recommendation systems, cognitive communications, and financial investments (Gittins 1979, Berry and Fristedt 1985, Press 2009, Farias and Madan 2011, Hoffman et al. 2011, Shen et al. 2015).

In the classic Bayesian MAB problem, the bandit machine has a total of $N$ arms and one player pulling an arm in each time slot. After a player selected one of these arms and activated it, a random reward will be accrued depending on the activated arm and its current state. In this process, all states are visible, and only the state of the arm being activated changes according to a Markov chain. The goal of the MAB problem is to search a policy that maximizes the long-term cumulative reward. Gittins (1979) proved that the classic MAB problem can be solved optimally by an index policy, referred to as Gittins index, and the player only needs to activate the arm with the largest index at each moment. The index policy significantly reduces the complexity of the problem from being exponential with the number of arms to being linear. After that, Whittle (1988) extended the MAB problem to the restless MAB (RMAB) model, in which $K$ arms can be activated at each moment, and the arms not chosen may also undergo state transitions over time. However, for the general RMAB problem with a finite state space, Papadimitriou and Tsitsiklis (1994) showed that the computational complexity of finding the optimal policy is already PSPACE-hard. Fortunately, Whittle (1988) generalized Gittins index to Whittle index that provides a possible solution to RMAB by considering Lagrangian relaxation and duality. Accordingly, Whittle index is optimal under a relaxed constraint that the time-average number of activated arms is $K$. For the original problem, the Whittle index policy has been proven to be asymptotically optimal per-arm wise as the number of arms goes to infinity under certain conditions (Weber and Weiss 1990, 1991). Nonetheless, not every RMAB is indexable, i.e., Whittle index may not exist. And the establishment of indexability is itself a difficult problem. Even if indexability is proven for a particular RMAB, analytical solutions of the Whittle index function can still be hard to obtain. For some important categories of RMAB models, Whittle indexability and its strong performance have been demonstrated in the literature, such as the dual-speed bandit problem and partially observable RMAB (Glazebrook et al. 2002, Ahmad et al. 2009, Liu and Zhao 2010, Gittins et al. 2011, Liu 2021, Liu et al. 2024).

In this paper, we extend the partially observable RMAB considered in Liu et al. (2010, 2024) to general observation models. The previous work only considers special classes of observation errors and noises within the framework of partially observation Markov decision processes (POMDP) (Zhao and Sadler 2007, Liu et al. 2010, 2024). But general algorithms for POMDP often suffer from curse of dimensionality and becomes cumbersome when the value functions for dynamic programming are too complex (Smallwood and Sondik 1973, Sondik 1978). In order to deal with general observation models, we need an alternative methodology to design efficient algorithms for systems with large sets of parameters. In the interdisciplinary field of operations research and stochastic optimization, an analytical method called “Achievable Region” emerged in the 1990s. Interestingly, this method transforms a time-series optimization problem into linear programming (LP) problems that can be solved efficiently (Coffman and Mitrani 1980, Federgruen and Groenevelt 1988a, b, Gelenbe and Mitrani 2010). The challenge of using this method is mainly on proving the feasibility of such transformation (Bertsimas 1995). Referred to as the general conservation law (GCL), Bertsimas and Niño-Mora (1996) proposed the required structure of performance measures for the achievable region to be a polyhedron called extended polymatroid. Based on GCL, they extended Klimov’s algorithm to several stochastic scheduling problems, including the classic MAB problem (Klimov 1975, Thomas 1991). By relaxing the GCL restrictions, Niño-Mora (2001) proposed the partial conservation law (PCL) and specified conditions for an RMAB to satisfy PCL and subsequently lead to the numerical calculation of Whittle index. Later on, Niño-Mora (2002) offered an economic explanation of PCL-indexability and its relation to Whittle indexability where Whittle index can be interpreted as the optimal marginal cost rate. A more comprehensive summary of this work can be found in Niño-Mora (2007). However, all such work considers finite state spaces except that Frostig and Weiss (1999) extended the GCL framework for classic MAB to countable state spaces. In this paper, we extend the PCL framework to the RMAB with a general parametrization of observation model and an infinite state space. The main challenge of this extension is due to the high-dimensional probability state space (belief states) with complex dynamics for transitions over time. By establishing the weak duality between LP formulations, we build the PCL framework for analyzing our RMAB and subsequently design an efficient algorithm for calculating Whittle index when PCL-indexability is satisfied. Finally, we demonstrate the superior performance of our algorithm by numerical simulations.

2 Main Results

2.1 Model Formulation

In this section, we begin with the formulation of the RMAB problem with general observation models. Assume the system has $N$ arms and the state space of the $n$th arm is $\mathcal{M}_{n}$. Therefore the entire system state space is $\mathcal{M}=\prod_{n=1}^{N}\mathcal{M}_{n}$. At each moment, the actual state of each arm undergos a state transition based on its own Markov probability transition matrix, and we will select $K$ arms to activate. For those activated arms, we can observe their states (with errors) and accrue some reward dependent on the observed states and the actual states. For those arms that are not activated, we can neither observe their states nor obtain reward from them, and their states still transit over time. Let $S_{n}(t),O_{n}(t)\in\mathbb{Z}^{+}$ be respectively the true and observed states of arm $n$ in slot $t$ and $A(t)$ the set of activated arms in slot $t$. First consider a single-arm process (i.e. $N=1$ and $\mathcal{M}=\mathcal{M}_{n}$) and drop the subscript $n$ (sometimes the time index $t$ is also dropped if referring to the same time index) for convenience. Suppose the state transition matrix, error matrix and reward matrix are $P=\{p_{ij}\}_{\mathcal{M}\times\mathcal{M}}$, $E=\{\varepsilon_{ij}\}_{\mathcal{M}\times\mathcal{M}}$εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT } start_POSTSUBSCRIPT caligraphic_M × caligraphic_M end_POSTSUBSCRIPT and $R=\{r_{ij}\}_{\mathcal{M}\times\mathcal{M}}$ respectively. For error matrix, $\varepsilon_{ij}$εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT represents the probability that the observed state is $j$ when the true state is $i$, that is, $\varepsilon_{ij}=P(O=j|S=i)$εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_P ( italic_O = italic_j | italic_S = italic_i ), while $r_{ij}$ represents the reward obtained when the true state is $i$ and the observed state is $j$ (different observations may lead to different sub-actions and thus different rewards following). Denoted by ${\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}_a=\{({\mathrm{<ruby><rb>\omega </rb><rt>おめが</rt></ruby>}}_1,\mathrm{\cdots },{\mathrm{<ruby><rb>\omega </rb><rt>おめが</rt></ruby>}}_M)|{\sum }_{i=1}^M{\mathrm{<ruby><rb>\omega </rb><rt>おめが</rt></ruby>}}_i=1,{\mathrm{<ruby><rb>\omega </rb><rt>おめが</rt></ruby>}}_i\ge 0,1\le i\le M:=|ℳ|\}$Ωおめが𝑎conditional-setsubscript𝜔1⋯subscript𝜔𝑀formulae-sequencesuperscriptsubscript𝑖1𝑀subscript𝜔𝑖1formulae-sequencesubscript𝜔𝑖01𝑖𝑀assignℳ{\Omega}_{a}=\{(\omega_{1},\cdots,\omega_{M})|\sum_{i=1}^{M}\omega_{i}=1,% \omega_{i}\geq 0,1\leq i\leq M:=|\mathcal{M}|\}roman_Ωおめが start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT = { ( italic_ωおめが start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_ωおめが start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT ) | ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = 1 , italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ≥ 0 , 1 ≤ italic_i ≤ italic_M := | caligraphic_M | } the belief space, for each $𝝎(t)=({\mathrm{<ruby><rb>\omega </rb><rt>おめが</rt></ruby>}}_1(t),\mathrm{\cdots },{\mathrm{<ruby><rb>\omega </rb><rt>おめが</rt></ruby>}}_M(t))\in {\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}_a$Ωおめが𝑎\boldsymbol{\omega}(t)=(\omega_{1}(t),\cdots,\omega_{M}(t))\in{\Omega_{a}}bold_italic_ωおめが ( italic_t ) = ( italic_ωおめが start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_t ) , ⋯ , italic_ωおめが start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT ( italic_t ) ) ∈ roman_Ωおめが start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT, $\omega_{i}(t)$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_t ) represents the conditional probability that the true state is $i$ in slot $t$ (based on past observations). If the current belief state is $(\omega_{1},\cdots,\omega_{M})$ωおめが start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_ωおめが start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT ), the current expected reward is $\sum_{i=1}^{M}\omega_{i}(\sum_{j=1}^{M}\varepsilon_{ij}r_{ij})$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ). In addition, we may sometimes receive additional feedback related to the true state and the immediate reward obtained that helps us better estimate the actual arm state (e.g., ACK/NAK in a communication channel to indicate whether the data was successfully transmitted (Liu et al. 2024)). Assume that there are $L$ feedback states (positive integers) in total that encompass all possible observations, and denote the feedback state at time $t$ by $F(t)~{}(L\leq M^{2})$. Define $\rho_{ij}=P(F(t)=j|S(t)=i)$ρろー start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_P ( italic_F ( italic_t ) = italic_j | italic_S ( italic_t ) = italic_i ). By Bayes rule, we have

$$P(S(t+1)=j|F(t)=i)=\frac{P(S(t+1)=j,F(t)=i)}{P(F(t)=i)}=\frac{\sum_{k=1}^{M}p_{kj}\rho_{ki}\omega_{k}}{\sum_{k=1}^{M}\rho_{ki}\omega_{k}}.$$ρろー start_POSTSUBSCRIPT italic_k italic_i end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_ρろー start_POSTSUBSCRIPT italic_k italic_i end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG . (1)

Thus the rule of belief update is

$$\boldsymbol{\omega}(t+1)=\begin{cases}(\frac{\sum_{i=1}^{M}p_{i1}\rho_{ij}\omega_{i}}{\sum_{i=1}^{M}\rho_{ij}\omega_{i}},\cdots,\frac{\sum_{i=1}^{M}p_{iM}\rho_{ij}\omega_{i}}{\sum_{i=1}^{M}\rho_{ij}\omega_{i}}),&\text{if active and }F(t)=j\\ \boldsymbol{\omega}(t)P,&\text{if passive}\end{cases}.$$ωおめが ( italic_t + 1 ) = { start_ROW start_CELL ( divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT italic_ρろー start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_ρろー start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG , ⋯ , divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_M end_POSTSUBSCRIPT italic_ρろー start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_ρろー start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) , end_CELL start_CELL if active and italic_F ( italic_t ) = italic_j end_CELL end_ROW start_ROW start_CELL bold_italic_ωおめが ( italic_t ) italic_P , end_CELL start_CELL if passive end_CELL end_ROW . (2)

In a simpler scenario where there is no additional feedback but only the observed states, the update rule becomes

$$\boldsymbol{\omega}(t+1)=\begin{cases}(\frac{\sum_{i=1}^{M}p_{i1}\varepsilon_{ij}\omega_{i}}{\sum_{i=1}^{M}\varepsilon_{ij}\omega_{i}},\cdots,\frac{\sum_{i=1}^{M}p_{iM}\varepsilon_{ij}\omega_{i}}{\sum_{i=1}^{M}\varepsilon_{ij}\omega_{i}}),&\text{if active and }O(t)=j\\ \boldsymbol{\omega}(t)P,&\text{if passive}\end{cases}.$$ωおめが ( italic_t + 1 ) = { start_ROW start_CELL ( divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG , ⋯ , divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_M end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) , end_CELL start_CELL if active and italic_O ( italic_t ) = italic_j end_CELL end_ROW start_ROW start_CELL bold_italic_ωおめが ( italic_t ) italic_P , end_CELL start_CELL if passive end_CELL end_ROW . (3)

In the extreme case when we cannot observe any state after activating an arm, we simply treat the obtained reward (might be $0$) as the feedback. By normalizing the immediate reward $r(t)$ to take only positive integer values, we have

$$\rho_{ij}= P(r(t)=j|S(t)=i)$$ρろー start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT =

$$= \frac{P(r(t)=j,S(t)=i)}{P(S(t)=i)}$$

$$= \frac{\sum_{k=1}^{M}P(r(t)=j,O(t)=k,S(t)=i)}{\omega_{i}}$$ ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG

$$= \frac{\sum_{k=1}^{M}P(r(t)=j|O(t)=k,S(t)=i)P(O(t)=k|S(t)=i)\omega_{i}}{\omega_{i}}$$ ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG

$$= \sum_{k=1}^{M}\mathbbm{1}(r_{ik}=j)\varepsilon_{ik}$$ εいぷしろん start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT

or equivalently $\rho_{ir}=\sum_{j=1}^{M}\mathbbm{1}(r_{ij}=r)\varepsilon_{ij}$ρろー start_POSTSUBSCRIPT italic_i italic_r end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT blackboard_1 ( italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r ) italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT. Thus we have

$$\sum_{i=1}^{M}p_{i1}\rho_{ir}\omega_{i}={} \sum_{i=1}^{M}p_{i1}(\sum_{j=1}^{M}\mathbbm{1}(r_{ij}=r)\varepsilon_{ij}\omega_{i})$$ρろー start_POSTSUBSCRIPT italic_i italic_r end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )

$$={} \sum_{i=1}^{M}\sum_{j=1}^{M}\mathbbm{1}(r_{ij}=r)p_{i1}\varepsilon_{ij}\omega_{i}$$ εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT

$$={} \sum_{r_{ij}=r}p_{i1}\varepsilon_{ij}\omega_{i}.$$ εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT . (4)

Similarly,

$$\sum_{i=1}^{M}\rho_{ir}\omega_{i}=\sum_{r_{ij}=r}\varepsilon_{ij}\omega_{i}.$$ρろー start_POSTSUBSCRIPT italic_i italic_r end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT . (5)

Combining (2.1) and (5), the belief state update rule is

$$\boldsymbol{\omega}(t+1)=\begin{cases}(\frac{\sum_{r_{ij}=r}p_{i1}\varepsilon_{ij}\omega_{i}}{\sum_{r_{ij}=r}\varepsilon_{ij}\omega_{i}},\cdots,\frac{\sum_{r_{ij}=r}p_{iM}\varepsilon_{ij}\omega_{i}}{\sum_{r_{ij}=r}\varepsilon_{ij}\omega_{i}}),&\text{if active and }r(t)=r\\ \boldsymbol{\omega}(t)P,&\text{if passive}\end{cases}.$$ωおめが ( italic_t + 1 ) = { start_ROW start_CELL ( divide start_ARG ∑ start_POSTSUBSCRIPT italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG , ⋯ , divide start_ARG ∑ start_POSTSUBSCRIPT italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_M end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) , end_CELL start_CELL if active and italic_r ( italic_t ) = italic_r end_CELL end_ROW start_ROW start_CELL bold_italic_ωおめが ( italic_t ) italic_P , end_CELL start_CELL if passive end_CELL end_ROW . (6)

A more practical situation is that we can jointly obtain information from the observed state and the obtained reward. In this case, we can similarly give the belief update rule:

$$\boldsymbol{\omega}(t+1)=\begin{cases}(\frac{\sum_{i=1}^{M}p_{i1}\varepsilon_{ij}\omega_{i}\mathbbm{1}(r_{ij}=r)}{\sum_{i=1}^{M}\varepsilon_{ij}\omega_{i}\mathbbm{1}(r_{ij}=r)},\cdots,\frac{\sum_{i=1}^{M}p_{iM}\varepsilon_{ij}\omega_{i}\mathbbm{1}(r_{ij}=r)}{\sum_{i=1}^{M}\varepsilon_{ij}\omega_{i}\mathbbm{1}(r_{ij}=r)}),&\text{if active and }O(t)=j,\\ &r(t)=r~{}(\exists i~{}s.t.~{}r_{ij}=r)\\ \boldsymbol{\omega}(t)P,&\text{if passive}\end{cases}.$$ωおめが ( italic_t + 1 ) = { start_ROW start_CELL ( divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT blackboard_1 ( italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT blackboard_1 ( italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r ) end_ARG , ⋯ , divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_M end_POSTSUBSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT blackboard_1 ( italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_εいぷしろん start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT blackboard_1 ( italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r ) end_ARG ) , end_CELL start_CELL if active and italic_O ( italic_t ) = italic_j , end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL italic_r ( italic_t ) = italic_r ( ∃ italic_i italic_s . italic_t . italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = italic_r ) end_CELL end_ROW start_ROW start_CELL bold_italic_ωおめが ( italic_t ) italic_P , end_CELL start_CELL if passive end_CELL end_ROW . (7)

For the restless multi-armed bandit model, the goal is to find a policy $\pi$πぱい which maps the belief states of all arms into an active set $A(t)$ in slot $t$ that maximize the long-term expected discounted reward. In other words, if we denote the reward obtained by the $n$-th arm in slot $t$ by $r_{n}(t)$, then our objective is

$$\max_{\pi}\mathbb{E}_{\pi} \left[\sum_{t=1}^{+\infty}\beta^{t-1}\sum_{n=1}^{N}r_{n}(t)\bigg{|}\boldsymbol{\omega}_{1}(1),\cdots,\boldsymbol{\omega}_{n}(1)\right],$$πぱい end_POSTSUBSCRIPT blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT βべーた start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT italic_r start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_t ) | bold_italic_ωおめが start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( 1 ) , ⋯ , bold_italic_ωおめが start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( 1 ) ] , (8)

$$s.t.~{}|A(t)|=K,t\geq 1,$$ (9)

where $0\leq\beta<1$βべーた < 1 is the discount factor and $\boldsymbol{\omega}_{n}(t)$ωおめが start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_t ) is the belief state (vector) of arm $n$ in slot $t$. In this problem, the diversity of states, choices, errors makes the problem highly complex. In RMAB problems, searching for an easily computable priority index policy is the mainstream. The core idea is to assign an index (a real number) to the current state of each arm, and then activate those arms with top $K$ large indexes. The goal of this paper is to theoretically characterize the conditions when such an index policy exists and provide a detailed algorithm for efficiently computing the index function (if it exists).

2.2 Whittle Index

Whittle (1988) relaxed the constraint on the exact number of arms activated in each slot, requiring only that the expected number of arms activated per slot on average (in the sense of discounted time) is $K$, i.e.

$$\mathbb{E}_{\pi}\left[\sum_{t=1}^{+\infty}\beta^{t-1}\sum_{n=1}^{N}\mathbbm{1}(n\in A(t))\bigg{|}\boldsymbol{\Omega}(1)\right]=\frac{K}{1-\beta}$$πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT blackboard_1 ( italic_n ∈ italic_A ( italic_t ) ) | bold_Ωおめが ( 1 ) ] = divide start_ARG italic_K end_ARG start_ARG 1 - italic_βべーた end_ARG (10)

or

$$\mathbb{E}_{\pi}\left[\sum_{t=1}^{+\infty}\beta^{t-1}\sum_{n=1}^{N}\mathbbm{1}(n\notin A(t))\bigg{|}\boldsymbol{\Omega}(1)\right]=\frac{N-K}{1-\beta},$$πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT blackboard_1 ( italic_n ∉ italic_A ( italic_t ) ) | bold_Ωおめが ( 1 ) ] = divide start_ARG italic_N - italic_K end_ARG start_ARG 1 - italic_βべーた end_ARG , (11)

where $\boldsymbol{\Omega}(t)=(\boldsymbol{\omega}_{1}(t),\cdots,\boldsymbol{\omega}_{N}(t))$Ωおめが ( italic_t ) = ( bold_italic_ωおめが start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_t ) , ⋯ , bold_italic_ωおめが start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ( italic_t ) ) with $\boldsymbol{\omega}_{n}(t)$ωおめが start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_t ) as the belief state (vector) for arm $n$ at time $t$. Thus the Lagrange optimization problem can be written as

$$\max_{\pi}\mathbb{E}_{\pi}\left[\sum_{t=1}^{+\infty}\beta^{t-1}\sum_{n=1}^{N}(\mathbbm{1}(n\in A(t))r_{n}(t)+\lambda\mathbbm{1}(n\notin A(t)))\bigg{|}\boldsymbol{\Omega}(1)\right].$$πぱい end_POSTSUBSCRIPT blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT ( blackboard_1 ( italic_n ∈ italic_A ( italic_t ) ) italic_r start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_t ) + italic_λらむだ blackboard_1 ( italic_n ∉ italic_A ( italic_t ) ) ) | bold_Ωおめが ( 1 ) ] . (12)

The above problem can be decomposed into $N$ independent subproblems, that is, for any $1\leq n\leq N$,

$$\max_{\pi}\mathbb{E}_{\pi}\left[\sum_{t=1}^{+\infty}\beta^{t-1}(\mathbbm{1}(n\in A(t))r_{n}(t)+\lambda\mathbbm{1}(n\notin A(t)))\bigg{|}\boldsymbol{\omega}_{n}(1)\right].$$πぱい end_POSTSUBSCRIPT blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ( blackboard_1 ( italic_n ∈ italic_A ( italic_t ) ) italic_r start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_t ) + italic_λらむだ blackboard_1 ( italic_n ∉ italic_A ( italic_t ) ) ) | bold_italic_ωおめが start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( 1 ) ] . (13)

The explanation of the optimization problem above is that for each arm, when it is not selected (made passive), we will receive a subsidy $\lambda$λらむだ. Since the problems above are independent, we just need to consider the single-arm case. For notation simplicity, we drop the arm index $n$ from now on. For a given arm, the optimal policy for the relaxed optimization problem divides the arm state space into two subset (here the arm state space is the arm belief state space): active set $\mathcal{A}(\lambda)$λらむだ ) and passive set $\mathcal{P}(\lambda)$λらむだ ). Specifically, $\mathcal{P}(\lambda)$λらむだ ) contains all belief states in which the optimal choice is passive when the subsidy is $\lambda$λらむだ. In particular, for a certain state $s$, if both active and passive actiosn are optimal, we include it in $\mathcal{P}(\lambda)$λらむだ ), and $\mathcal{A}(\lambda)$λらむだ ) is just the complement of $\mathcal{P}(\lambda)$λらむだ ) in the entire state space. Under the concept of passive set, Whittle indexability can be stated as follows:

Definition 2.1 (Whittle Indexability)

An arm is indexable if the passive set $\mathcal{P}(\lambda)$λらむだ ) increases from $\emptyset$ to the whole state space as the subsidy $\lambda$λらむだ increases from $-\infty$ to $+\infty$. The RMAB problem is indexable if every decoupled problem is indexable.

Indexability states that, once an arm is made passive with subsidy $\lambda$λらむだ, it should also be passive with any $\lambda^{\prime}$λらむだ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT larger than $\lambda$λらむだ. If the problem satisfies indexability, its Whittle index is defined as follows:

Definition 2.2 (Whittle Index)

If an arm is indexable, the Whittle index $W(s)$ of a state $s$ is the infimum subsidy $\lambda$λらむだ that keeps $s$ in the passive set $\mathcal{P}(\lambda)$λらむだ ). That is,

$$W(s):=\inf\{\lambda:s\in\mathcal{P}(\lambda)\}.$$λらむだ : italic_s ∈ caligraphic_P ( italic_λらむだ ) } .

By continuity, $W(s)$ is the infimum subsidy that makes it equivalent to be active or passive at state $s$. Under the definitions of indexability and Whittle index, the Whittle index policy for the original multi-armed bandit problem is simply to activate the $K$ arms whose states offer the largest Whittle indices. In fact, for the classic MAB problem where passive arms do not change states, the Whittle indexability is always satisfied and the Whittle index is reduced to the Gittins index.

2.3 Belief State Space

Different from the perfect observation case, the update of the belief state is nonlinear for the general observation model. This yields much difficulty for us to use value functions and dynamic programming methods to analyze the problem. Given an initial belief state, it appears that the size of the belief state space will grow exponentially over time as all possible realizations of observations/rewards/feedback are incessantly traversed and updated. Fortunately, a large number of numerical experiments have shown that in the sense of Euclidean norm approximation, the state space exhibits convergence after a process of iterative calculations (see Sec. 3 for details). Note that the initial belief state is $\boldsymbol{\omega}$ωおめが, $\{\mathcal{B}_{l}\}_{0\leq l\leq L}$ is a list of operators defined by feedback states and state update rules, that is, for $1\leq l\leq L$, $\mathcal{B}_{l}\boldsymbol{\omega}$ωおめが is the updated state caused by the $l$-th feedback state under the active action, and $\mathcal{B}_{0}$ is the state update operator under the passive action. In this paper, $\mathcal{B}_{0}\boldsymbol{\omega}=\boldsymbol{\omega}P$ωおめが = bold_italic_ωおめが italic_P. Define the $T$-step state space recursively as follows:

Definition 2.3 ($T$-step state space)

Define

Ωおめがconditional𝑇𝝎casesconditional-setsubscriptℬsubscript𝑙1⋯subscriptℬsubscript𝑙𝑇𝝎formulae-sequence0subscript𝑙1⋯subscript𝑙𝑇𝐿Ωおめが𝑇conditional1𝝎𝑇2conditional-setsubscriptℬ𝑙𝝎0𝑙𝐿𝝎𝑇1\Omega(T|\boldsymbol{\omega})=\begin{cases}\{\mathcal{B}_{l_{1}}\cdots\mathcal% {B}_{l_{T}}\boldsymbol{\omega}:0\leq l_{1},\cdots,l_{T}\leq L\}\cup\Omega(T-1|% \boldsymbol{\omega}),&T\geq 2\\ \{\mathcal{B}_{l}\boldsymbol{\omega}:0\leq l\leq L\}\cup\{\boldsymbol{\omega}% \},&T=1\end{cases}.roman_Ωおめが ( italic_T | bold_italic_ωおめが ) = { start_ROW start_CELL { caligraphic_B start_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ⋯ caligraphic_B start_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_T end_POSTSUBSCRIPT end_POSTSUBSCRIPT bold_italic_ωおめが : 0 ≤ italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_l start_POSTSUBSCRIPT italic_T end_POSTSUBSCRIPT ≤ italic_L } ∪ roman_Ωおめが ( italic_T - 1 | bold_italic_ωおめが ) , end_CELL start_CELL italic_T ≥ 2 end_CELL end_ROW start_ROW start_CELL { caligraphic_B start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT bold_italic_ωおめが : 0 ≤ italic_l ≤ italic_L } ∪ { bold_italic_ωおめが } , end_CELL start_CELL italic_T = 1 end_CELL end_ROW .

We call $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}(T|𝛚)$Ωおめがconditional𝑇𝛚\Omega(T|\boldsymbol{\omega})roman_Ωおめが ( italic_T | bold_italic_ωおめが ) the $T$-step state space under the initial belief state $\boldsymbol{\omega}$ωおめが.

Under the definition of the $T$-step state space, $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}(+\mathrm{\infty }|{𝝎}_{\mathrm{𝟎}})$Ωおめがconditionalsubscript𝝎0\Omega(+\infty|\boldsymbol{\omega_{0}})roman_Ωおめが ( + ∞ | bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) is the belief space obtained by traversing all possible state update rules starting from the initial state $\boldsymbol{\omega_{0}}$ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT. Clearly, $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}(+\mathrm{\infty }|{𝝎}_{\mathrm{𝟎}})$Ωおめがconditionalsubscript𝝎0\Omega(+\infty|\boldsymbol{\omega_{0}})roman_Ωおめが ( + ∞ | bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) is countable and we denote it by $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0\Omega(\boldsymbol{\omega_{0}})roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ). Under a policy $\pi$πぱい, let $p_{ij}(\pi):=P(\boldsymbol{\omega}(t+1)=\boldsymbol{\omega_{j}}|\boldsymbol{\omega}(t)=\boldsymbol{\omega_{i}})$πぱい ) := italic_P ( bold_italic_ωおめが ( italic_t + 1 ) = bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT | bold_italic_ωおめが ( italic_t ) = bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ) be the conditional probability of transition from belief state $\boldsymbol{\omega_{i}}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT to $\boldsymbol{\omega_{j}}$ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT. Due to different actions of activation and passivity affecting the state update, decomposition of $p_{ij}(\pi)$πぱい ) should be carried out. Let $a(t)$ be the indicator function of whether the arm is activated in slot $t$, then we define the transition probability of the belief state as follows:

$$p_{ij}^{0}=P(\boldsymbol{\omega}(t+1)=\boldsymbol{\omega}_{j}|\boldsymbol{\omega}(t)=\boldsymbol{\omega}_{i},a(t)=0),$$ωおめが ( italic_t + 1 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | bold_italic_ωおめが ( italic_t ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_a ( italic_t ) = 0 ) , (14)

$$p_{ij}^{1}=P(\boldsymbol{\omega}(t+1)=\boldsymbol{\omega}_{j}|\boldsymbol{\omega}(t)=\boldsymbol{\omega}_{i},a(t)=1).$$ωおめが ( italic_t + 1 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | bold_italic_ωおめが ( italic_t ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_a ( italic_t ) = 1 ) . (15)

By (14)-(15) and the total probability theorem, we have

$$p_{ij}(\pi) =\sum\limits_{a=0}^{1}P(a(t)=a|\boldsymbol{\omega}(t)=\boldsymbol{\omega}_{i})p_{ij}^{a}.$$πぱい ) ωおめが ( italic_t ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_a end_POSTSUPERSCRIPT . (16)

The elements in the probability transition matrix under passive and active actions have the following more specific expressions:

$$p_{ij}^{0}=\begin{cases}1,&\text{if $\boldsymbol{\omega}_{j}=\boldsymbol{\omega}_{i}P$}\\ 0,&\text{otherwise}\end{cases}$$ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_P end_CELL end_ROW start_ROW start_CELL 0 , end_CELL start_CELL otherwise end_CELL end_ROW (17)

and

$$p_{ij}^{1} =P(\boldsymbol{\omega}(t+1)=\boldsymbol{\omega}_{j}|\text{active and }\boldsymbol{\omega}(t)=\boldsymbol{\omega}_{i})$$ ωおめが ( italic_t + 1 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | active and bold_italic_ωおめが ( italic_t ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) (18)

$$=\sum\limits_{\{l:\mathcal{B}_{l}\boldsymbol{\omega}_{i}=\boldsymbol{\omega}_{j},1\leq l\leq L\}}P(F(t)=l|\text{active and }\boldsymbol{\omega}(t)=\boldsymbol{\omega}_{i}).$$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT , 1 ≤ italic_l ≤ italic_L } end_POSTSUBSCRIPT italic_P ( italic_F ( italic_t ) = italic_l | active and bold_italic_ωおめが ( italic_t ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) .

In the following sections, these two probability transition matrices will play an important role in the calculation of Whittle indices.

2.4 Two-Arm Problem and Achievable Region

With the rapid development at the junction of operations research, stochastic optimization and reinforcement learning, quite a few effective methods for finite-state problems have been found, among which achievable region with conservation laws is a great example. Bertsimas and Niño-Mora (1996) and Niño-Mora (2001) adopted the analytical framework of generalized conservation laws and partial conservation laws for the classic MAB and RMAB problems with finite state spaces, respectively, and provided efficient index algorithms for the corresponding problems. In the rest subsections, we will apply the PCL framework to the analysis of our RMAB problem with an infinite state space, and theoretically build the foundation for the construction of an efficient index policy.

Consider the single-arm process with $M$ states discussed in the previous section. For an initial belief state $\boldsymbol{\omega_{0}}$ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT, let $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0\Omega(\boldsymbol{\omega_{0}})roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) be the countable belief state space generated by iteratively updating $\boldsymbol{\omega_{0}}$ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT through state transitions. During the process of making the arm active or passive, since all belief states fall within $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0\Omega(\boldsymbol{\omega_{0}})roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ), the entire time period can be completely partitioned by the time segments in which each state is located. In this scenario, define the performance indicator variables for each belief state as follows:

$$I_{\boldsymbol{\omega}}^{1}(t)=\begin{cases}1,&\text{if the arm is active in slot $t$ and its belief state is $\boldsymbol{\omega}$}\\ 0,&\text{otherwise}\end{cases}$$ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_t ) = { start_ROW start_CELL 1 , end_CELL start_CELL if the arm is active in slot italic_t and its belief state is bold_italic_ωおめが end_CELL end_ROW start_ROW start_CELL 0 , end_CELL start_CELL otherwise end_CELL end_ROW

and

$$I_{\boldsymbol{\omega}}^{0}(t)=\begin{cases}1,&\text{if the arm is passive in slot $t$ and its belief state is $\boldsymbol{\omega}$}\\ 0,&\text{otherwise}\end{cases}.$$ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ( italic_t ) = { start_ROW start_CELL 1 , end_CELL start_CELL if the arm is passive in slot italic_t and its belief state is bold_italic_ωおめが end_CELL end_ROW start_ROW start_CELL 0 , end_CELL start_CELL otherwise end_CELL end_ROW .

Furthermore, define the performance measures of belief state $\boldsymbol{\omega}$ωおめが as follows:

$$x_{\omega}^{a}(\pi)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{+\infty}\beta^{t}I_{\boldsymbol{\omega}}^{a}(t)\right],\quad a=0,1$$ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_a end_POSTSUPERSCRIPT ( italic_πぱい ) = blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT italic_I start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_a end_POSTSUPERSCRIPT ( italic_t ) ] , italic_a = 0 , 1 (19)

$$x_{\omega}^{a}(\pi,\boldsymbol{\omega}_{i})=\mathbb{E}_{\pi}\left[\sum_{t=0}^{+\infty}\beta^{t}I_{\boldsymbol{\omega}}^{a}(t)\bigg{|}\boldsymbol{\omega}(0)=\boldsymbol{\omega}_{i}\right],\quad a=0,1$$ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_a end_POSTSUPERSCRIPT ( italic_πぱい , bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT italic_I start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_a end_POSTSUPERSCRIPT ( italic_t ) | bold_italic_ωおめが ( 0 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ] , italic_a = 0 , 1 (20)

where $\boldsymbol{\omega}(t)$ωおめが ( italic_t ) is the belief state of the arm in slot $t$. In Whittle’s relaxation, the Lagrangian multiplier $\lambda$λらむだ can be regarded as a subsidy when the arm is made passive. Thus the optimization objective of the partially observable RMAB problem with subsidy can be written as follows:

$$\max_{\pi} \sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_{0}})}R_{\boldsymbol{\omega}}x_{\boldsymbol{\omega}}^{1}+\lambda\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_{0}})}x_{\boldsymbol{\omega}}^{0}$$πぱい end_POSTSUBSCRIPT Ωおめがsubscript𝝎0subscript𝑅𝝎superscriptsubscript𝑥𝝎1𝜆subscript𝝎Ωおめがsubscript𝝎0superscriptsubscript𝑥𝝎0\displaystyle\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_{0}}% )}R_{\boldsymbol{\omega}}x_{\boldsymbol{\omega}}^{1}+\lambda\sum\limits_{% \boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_{0}})}x_{\boldsymbol{\omega}}^% {0}∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT + italic_λらむだ ∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT (21)

subject to $x_{{𝝎}_i}^1+x_{{𝝎}_{𝒊}}^0=e_{{𝝎}_i}+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}({\displaystyle \sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}}{p}_{ji}^1{x}_{{𝝎}_j}^1+{\displaystyle \sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}}{p}_{ji}^0{x}_{{𝝎}_j}^0),\forall {𝝎}_{𝒊}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑗𝑖1superscriptsubscript𝑥subscript𝝎𝑗1subscriptsubscript𝝎𝒋Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑗𝑖0superscriptsubscript𝑥subscript𝝎𝑗0for-allsubscript𝝎𝒊Ωおめがsubscript𝝎0\displaystyle x_{\boldsymbol{\omega}_{i}}^{1}+x_{\boldsymbol{\omega_{i}}}^{0}=% e_{\boldsymbol{\omega}_{i}}+\beta(\sum\limits_{\boldsymbol{\omega_{j}}\in% \Omega(\boldsymbol{\omega_{0}})}p_{ji}^{1}x_{\boldsymbol{\omega}_{j}}^{1}+\sum% \limits_{\boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_{ji}^{0}x% _{\boldsymbol{\omega}_{j}}^{0}),\quad\forall\boldsymbol{\omega_{i}}\in\Omega(% \boldsymbol{\omega_{0}})italic_x start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT + italic_x start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT = italic_e start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_βべーた ( ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_j italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT + ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_j italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ) , ∀ bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT )

$x_{{𝝎}_i}^1,x_{{𝝎}_i}^0\ge 0,\forall {𝝎}_{𝒊}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0\displaystyle x_{\boldsymbol{\omega}_{i}}^{1},x_{\boldsymbol{\omega}_{i}}^{0}% \geq 0,\quad\forall\boldsymbol{\omega_{i}}\in\Omega(\boldsymbol{\omega_{0}})italic_x start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT , italic_x start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ≥ 0 , ∀ bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT )

where $R_{\boldsymbol{\omega}}=\sum\limits_{i=1}^{M}\omega_{i}R_{i},\boldsymbol{\omega}=(\omega_{1},\cdots,\omega_{M})$ωおめが end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_R start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_ωおめが = ( italic_ωおめが start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_ωおめが start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT ), $p_{ij}^{a}$ are given by (17)-(18), $e_{\boldsymbol{\omega}_{i}}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT is the indicator whether the initial belief state is $\boldsymbol{\omega}_{i}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, $r_{i}$ is the expected immediate reward when $S(t)=i$ with the active action. Obviously, $R_{\boldsymbol{\omega}}$ωおめが end_POSTSUBSCRIPT is bounded above and we can assume $|R_{𝝎}|\le C,\forall 𝝎\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0|R_{\boldsymbol{\omega}}|\leq C,\forall\boldsymbol{\omega}\in\Omega(% \boldsymbol{\omega_{0}})| italic_R start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT | ≤ italic_C , ∀ bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ). For the equality constraints of this optimization problem, it can be understood that the left side of the equation is the direct performance measure for state $\boldsymbol{\omega}_{i}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, while the right side of the equation is another way to represent it. That is, each occurrence of state $\boldsymbol{\omega}_{i}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT in the system is transitioned by another state $\boldsymbol{\omega}_{j}$ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT. Therefore, the performance measure for state $\boldsymbol{\omega}_{i}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT can be represented by a combination of performance measures of other states. This model with subsidy can be explained more intuitively through a two-arm system. In this system, the first arm is the original arm while the second arm (auxiliary arm) has only one state 0. In every slot we choose one of the two arms to activate. When the auxiliary arm is activated, we obtain a fixed reward $\lambda$λらむだ. Our goal is to find a policy to maximize the long-term expected discounted reward by deciding which arm to choose in each slot. The objective function can thus be written as

$$\underset{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}{\max }\sum _{𝝎\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}{R}_{𝝎}{x}_{𝝎}^1(\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>})+\mathrm{<ruby><rb>\lambda </rb><rt>らむだ</rt></ruby>}{x}_0^1(\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}),$$Ωおめがsubscript𝝎0subscript𝑅𝝎superscriptsubscript𝑥𝝎1𝜋𝜆superscriptsubscript𝑥01𝜋\max\limits_{\pi}\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_% {0}})}R_{\boldsymbol{\omega}}x_{\boldsymbol{\omega}}^{1}(\pi)+\lambda x_{0}^{1% }(\pi),roman_max start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_πぱい ) + italic_λらむだ italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_πぱい ) ,

where $x_{0}^{1}(\pi)$πぱい ) is the performance measure of state 0 being activated under policy $\pi$πぱい. Let $X$ be the set of all elements ${(x_0^1(\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}),x_{𝝎}^1(\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}))}_{𝝎\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}$Ωおめがsubscript𝝎0(x_{0}^{1}(\pi),x_{\boldsymbol{\omega}}^{1}(\pi))_{\boldsymbol{\omega}\in% \Omega(\boldsymbol{\omega_{0}})}( italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_πぱい ) , italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_πぱい ) ) start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT as $\pi$πぱい traverses the admissible (feasible) policy set $\mathrm{<ruby><rb>\Pi </rb><rt>ぱい</rt></ruby>}$Πぱい\Piroman_Πぱい. We call $X$ the achievable region. Under this definition, the optimization objective function can be written as follows:

$$\underset{(x_0^1,x_{𝝎}^1)\in X}{\max }\sum _{𝝎\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}{R}_{𝝎}{x}_{𝝎}^1+\mathrm{<ruby><rb>\lambda </rb><rt>らむだ</rt></ruby>}{x}_0^1.$$Ωおめがsubscript𝝎0subscript𝑅𝝎superscriptsubscript𝑥𝝎1𝜆superscriptsubscript𝑥01\quad\max_{(x_{0}^{1},x_{\boldsymbol{\omega}}^{1})\in X}\sum\limits_{% \boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_{0}})}R_{\boldsymbol{\omega}}x% _{\boldsymbol{\omega}}^{1}+\lambda x_{0}^{1}.roman_max start_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT , italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ) ∈ italic_X end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT + italic_λらむだ italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT . (OPT)

The core of solving this optimization problem is to mathematically characterize the achievable region $X$. The so-called conservation law refers to the fact that for any $(x_{0}^{1},x_{\boldsymbol{\omega}}^{1})\in X$ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ) ∈ italic_X, these components may satisfy certain equality or inequality constraints. For the above model, a trivial equality constraint is $x_0^1+\sum _{𝝎\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}{x}_{𝝎}^1=\frac{1}{1-\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}}$Ωおめがsubscript𝝎0superscriptsubscript𝑥𝝎111𝛽x_{0}^{1}+\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega_{0}})}x% _{\boldsymbol{\omega}}^{1}=\frac{1}{1-\beta}italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT + ∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT = divide start_ARG 1 end_ARG start_ARG 1 - italic_βべーた end_ARG. This is because the RMAB system has exactly one state in the active phase at each moment (time-conservation). In the following subsections, we will further explore the rich mathematical structure of performance measures for the two-arm system with countable states under the concept of achievable region.

2.5 Partial Conservation Law

In this section, we elaborate the relationship between performance measures for each state in the two-arm system. To measure the performance of the original arm under policy $\pi$πぱい, we define the following variables:

$$T_{\boldsymbol{\omega}_{i}}^{\pi} :=\mathbb{E}_{\pi}\left[\sum\limits_{t=0}^{+\infty}\beta^{t}\mathbbm{1}(a(t)=1)\bigg{|}\boldsymbol{\omega}(0)=\boldsymbol{\omega}_{i}\right],\quad\boldsymbol{\omega_{i}}\in\Omega(\boldsymbol{\omega_{0}}),$$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT Ωおめがsubscript𝝎0\displaystyle:=\mathbb{E}_{\pi}\left[\sum\limits_{t=0}^{+\infty}\beta^{t}% \mathbbm{1}(a(t)=1)\bigg{|}\boldsymbol{\omega}(0)=\boldsymbol{\omega}_{i}% \right],\quad\boldsymbol{\omega_{i}}\in\Omega(\boldsymbol{\omega_{0}}),:= blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT blackboard_1 ( italic_a ( italic_t ) = 1 ) | bold_italic_ωおめが ( 0 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ] , bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) , (22)

$$R_{\boldsymbol{\omega}_{i}}^{\pi} :=\mathbb{E}_{\pi}\left[\sum\limits_{t=0}^{+\infty}\beta^{t}R_{\boldsymbol{\omega}(t)}\mathbbm{1}(a(t)=1)\bigg{|}\boldsymbol{\omega}(0)=\boldsymbol{\omega}_{i}\right],\quad\boldsymbol{\omega_{i}}\in\Omega(\boldsymbol{\omega_{0}}).$$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT Ωおめがsubscript𝝎0\displaystyle:=\mathbb{E}_{\pi}\left[\sum\limits_{t=0}^{+\infty}\beta^{t}R_{% \boldsymbol{\omega}(t)}\mathbbm{1}(a(t)=1)\bigg{|}\boldsymbol{\omega}(0)=% \boldsymbol{\omega}_{i}\right],\quad\boldsymbol{\omega_{i}}\in\Omega(% \boldsymbol{\omega_{0}}).:= blackboard_E start_POSTSUBSCRIPT italic_πぱい end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが ( italic_t ) end_POSTSUBSCRIPT blackboard_1 ( italic_a ( italic_t ) = 1 ) | bold_italic_ωおめが ( 0 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ] , bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) . (23)

Here $T_{\boldsymbol{\omega}_{i}}^{\pi}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT describes the expected total discounted time for the original arm to be activated in the two-arm system under policy $\pi$πぱい. We require that the policy $\pi$πぱい only depends on the original arm. Similarly, $R_{{𝝎}_i}^{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}$ΩおめがR_{\boldsymbol{\omega}_{i}}^{\Omega}italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT represents the expected total discounted reward obtained from the original arm under policy $\pi$πぱい. Therefore, $T_{\boldsymbol{\omega}_{i}}^{\pi}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT and $R_{\boldsymbol{\omega}_{i}}^{\pi}$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT also have the following expressions:

$$T_{\boldsymbol{\omega}_{i}}^{\pi} =\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega}_{0})}x_{\boldsymbol{\omega}}^{1}(\pi,\boldsymbol{\omega}_{i}),$$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT Ωおめがsubscript𝝎0superscriptsubscript𝑥𝝎1𝜋subscript𝝎𝑖\displaystyle=\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega}_{0% })}x_{\boldsymbol{\omega}}^{1}(\pi,\boldsymbol{\omega}_{i}),= ∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_πぱい , bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) , (24)

$$R_{\boldsymbol{\omega}_{i}}^{\pi} =\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega}_{0})}R_{\boldsymbol{\omega}}x_{\boldsymbol{\omega}}^{1}(\pi,\boldsymbol{\omega}_{i}).$$ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT Ωおめがsubscript𝝎0subscript𝑅𝝎superscriptsubscript𝑥𝝎1𝜋subscript𝝎𝑖\displaystyle=\sum\limits_{\boldsymbol{\omega}\in\Omega(\boldsymbol{\omega}_{0% })}R_{\boldsymbol{\omega}}x_{\boldsymbol{\omega}}^{1}(\pi,\boldsymbol{\omega}_% {i}).= ∑ start_POSTSUBSCRIPT bold_italic_ωおめが ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT bold_italic_ωおめが end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT ( italic_πぱい , bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) . (25)

Clearly, the optimal policy partitions the belief state space $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0\Omega(\boldsymbol{\omega_{0}})roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) of the original arm into two sets where the optimal actions should be active and passive respectively. For any $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}\subset \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$ΩおめがΩおめがsubscript𝝎0\Omega\subset\Omega(\boldsymbol{\omega_{0}})roman_Ωおめが ⊂ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ), we call ${\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}_{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}$Ωおめが\pi_{\Omega}italic_πぱい start_POSTSUBSCRIPT roman_Ωおめが end_POSTSUBSCRIPT an $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}$Ωおめが\Omegaroman_Ωおめが-priority policy if the original arm is activated when its current belief state falls into $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}$Ωおめが\Omegaroman_Ωおめが and made passive when the state falls into ${\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^c$Ωおめが𝑐\Omega^{c}roman_Ωおめが start_POSTSUPERSCRIPT italic_c end_POSTSUPERSCRIPT. For $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}$Ωおめが\Omegaroman_Ωおめが-priority policy, we have

$T_{{𝝎}_i}^{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}$Ωおめが\displaystyle T_{\boldsymbol{\omega}_{i}}^{\Omega}italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT $={𝔼}_{{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}_{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}}\left[{\displaystyle \sum _{t=0}^{+\mathrm{\infty }}}{\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}}^t\mathrm{𝟙}(𝝎(t)\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>})|𝝎(0)={𝝎}_i\right],{𝝎}_{𝒊}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}}),$Ωおめがdelimited-[]conditionalsuperscriptsubscript𝑡0superscript𝛽𝑡1𝝎𝑡Ωおめが𝝎0subscript𝝎𝑖subscript𝝎𝒊Ωおめがsubscript𝝎0\displaystyle=\mathbb{E}_{\pi_{\Omega}}\left[\sum\limits_{t=0}^{+\infty}\beta^% {t}\mathbbm{1}(\boldsymbol{\omega}(t)\in\Omega)\bigg{|}\boldsymbol{\omega}(0)=% \boldsymbol{\omega}_{i}\right],\quad\boldsymbol{\omega_{i}}\in\Omega(% \boldsymbol{\omega_{0}}),= blackboard_E start_POSTSUBSCRIPT italic_πぱい start_POSTSUBSCRIPT roman_Ωおめが end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT blackboard_1 ( bold_italic_ωおめが ( italic_t ) ∈ roman_Ωおめが ) | bold_italic_ωおめが ( 0 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ] , bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) , (26)

$R_{{𝝎}_i}^{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}$Ωおめが\displaystyle R_{\boldsymbol{\omega}_{i}}^{\Omega}italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT $={𝔼}_{{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}_{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}}\left[{\displaystyle \sum _{t=0}^{+\mathrm{\infty }}}{\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}}^t{R}_{𝝎(t)}\mathrm{𝟙}(𝝎(t)\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>})|𝝎(0)={𝝎}_i\right],{𝝎}_{𝒊}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}}).$Ωおめがdelimited-[]conditionalsuperscriptsubscript𝑡0superscript𝛽𝑡subscript𝑅𝝎𝑡1𝝎𝑡Ωおめが𝝎0subscript𝝎𝑖subscript𝝎𝒊Ωおめがsubscript𝝎0\displaystyle=\mathbb{E}_{\pi_{\Omega}}\left[\sum\limits_{t=0}^{+\infty}\beta^% {t}R_{\boldsymbol{\omega}(t)}\mathbbm{1}(\boldsymbol{\omega}(t)\in\Omega)\bigg% {|}\boldsymbol{\omega}(0)=\boldsymbol{\omega}_{i}\right],\quad\boldsymbol{% \omega_{i}}\in\Omega(\boldsymbol{\omega_{0}}).= blackboard_E start_POSTSUBSCRIPT italic_πぱい start_POSTSUBSCRIPT roman_Ωおめが end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT + ∞ end_POSTSUPERSCRIPT italic_βべーた start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが ( italic_t ) end_POSTSUBSCRIPT blackboard_1 ( bold_italic_ωおめが ( italic_t ) ∈ roman_Ωおめが ) | bold_italic_ωおめが ( 0 ) = bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ] , bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) . (27)

From the above definition, we can easily give the following dynamic programming equations for these two variables:

Ωおめがcases1𝛽subscriptsubscript𝝎𝒋Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑇subscript𝝎𝑖Ωおめがsubscript𝝎𝑖Ωおめが𝛽subscriptsubscript𝝎𝒋Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑇subscript𝝎𝑖Ωおめがsubscript𝝎𝑖ΩおめがT_{\boldsymbol{\omega}_{i}}^{\Omega}=\begin{cases}1+\beta\sum\limits_{% \boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_{ij}^{1}T_{% \boldsymbol{\omega}_{i}}^{\Omega},&\boldsymbol{\omega}_{i}\in\Omega\\ \beta\sum\limits_{\boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_% {ij}^{0}T_{\boldsymbol{\omega}_{i}}^{\Omega}&\boldsymbol{\omega}_{i}\notin% \Omega\end{cases}italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT = { start_ROW start_CELL 1 + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT , end_CELL start_CELL bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが end_CELL end_ROW start_ROW start_CELL italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT end_CELL start_CELL bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∉ roman_Ωおめが end_CELL end_ROW (28)

and

Ωおめがcasessubscript𝑅subscript𝝎𝑖𝛽subscriptsubscript𝝎𝒋Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑅subscript𝝎𝑖Ωおめがsubscript𝝎𝑖Ωおめが𝛽subscriptsubscript𝝎𝒋Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑅subscript𝝎𝑖Ωおめがsubscript𝝎𝑖ΩおめがR_{\boldsymbol{\omega}_{i}}^{\Omega}=\begin{cases}R_{\boldsymbol{\omega}_{i}}+% \beta\sum\limits_{\boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_% {ij}^{1}R_{\boldsymbol{\omega}_{i}}^{\Omega},&\boldsymbol{\omega}_{i}\in\Omega% \\ \beta\sum\limits_{\boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_% {ij}^{0}R_{\boldsymbol{\omega}_{i}}^{\Omega}&\boldsymbol{\omega}_{i}\notin% \Omega\end{cases}.italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT = { start_ROW start_CELL italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT , end_CELL start_CELL bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∈ roman_Ωおめが end_CELL end_ROW start_ROW start_CELL italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT end_CELL start_CELL bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∉ roman_Ωおめが end_CELL end_ROW . (29)

If $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})$Ωおめがsubscript𝝎0\Omega(\boldsymbol{\omega_{0}})roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) is a finite state space, we can directly solve for $T_{\boldsymbol{\omega_{i}}}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT and $R_{\boldsymbol{\omega_{i}}}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT from the two equation sets above. However, a direct solution is not available for countable state spaces. To further investigate the properties of these variables, we define $V_{\boldsymbol{\omega_{i}}}^{\pi}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT as the expected total discounted reward starting from state $\boldsymbol{\omega_{i}}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT under policy $\pi$πぱい. We have

$$V_{\boldsymbol{\omega_{i}}}^{\pi}=R_{\boldsymbol{\omega_{i}}}^{\pi}+\lambda\left(\frac{1}{1-\beta}-T_{\boldsymbol{\omega_{i}}}^{\pi}\right).$$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT = italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT + italic_λらむだ ( divide start_ARG 1 end_ARG start_ARG 1 - italic_βべーた end_ARG - italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT ) . (30)

Let $V_{\boldsymbol{\omega_{i}},0}^{\pi}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT , 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT and $V_{\boldsymbol{\omega_{i}},1}^{\pi}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT , 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT be the expected total discounted rewards with the initial state $\boldsymbol{\omega_{i}}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT when taking the passive and active actions respectively, then

$V_{{𝝎}_{𝒊},0}^{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}=\mathrm{<ruby><rb>\lambda </rb><rt>らむだ</rt></ruby>}+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}{\displaystyle \sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}}{p}_{ij}^0{V}_{{𝝎}_{𝒋}}^{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}},$Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑉subscript𝝎𝒋𝜋\displaystyle V_{\boldsymbol{\omega_{i}},0}^{\pi}=\lambda+\beta\sum\limits_{% \boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_{ij}^{0}V_{% \boldsymbol{\omega_{j}}}^{\pi},italic_V start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT , 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT = italic_λらむだ + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT italic_V start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT , (31)

$V_{{𝝎}_{𝒊},1}^{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}=R_{{𝝎}_{𝒊}}+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}{\displaystyle \sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}}{p}_{ij}^1{V}_{{𝝎}_{𝒋}}^{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}.$Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑉subscript𝝎𝒋𝜋\displaystyle V_{\boldsymbol{\omega_{i}},1}^{\pi}=R_{\boldsymbol{\omega_{i}}}+% \beta\sum\limits_{\boldsymbol{\omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}p_% {ij}^{1}V_{\boldsymbol{\omega_{j}}}^{\pi}.italic_V start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT , 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT = italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT italic_V start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT . (32)

If, under policy $\pi$πぱい, the expected total discounted reward obtained by active and passive actions are the same for the initial state $\boldsymbol{\omega_{i}}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT, i.e., $V_{\boldsymbol{\omega_{i},0}}^{\pi}=V_{\boldsymbol{\omega_{i},1}}^{\pi}$ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT bold_, bold_0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT = italic_V start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT bold_, bold_1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT, then we can solve for $\lambda$λらむだ:

$$\mathrm{<ruby><rb>\lambda </rb><rt>らむだ</rt></ruby>}=\frac{R_{{𝝎}_{𝒊}}+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}\sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}(p_{ij}^1-p_{ij}^0)R_{{𝝎}_{𝒋}}^{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}}{1+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}\sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}(p_{ij}^1-p_{ij}^0)T_{{𝝎}_{𝒋}}^{\mathrm{<ruby><rb>\pi </rb><rt>ぱい</rt></ruby>}}}.$$Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑅subscript𝝎𝒋𝜋1𝛽subscriptsubscript𝝎𝒋Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑇subscript𝝎𝒋𝜋\lambda=\frac{R_{\boldsymbol{\omega_{i}}}+\beta\sum\limits_{\boldsymbol{\omega% _{j}}\in\Omega(\boldsymbol{\omega_{0}})}(p_{ij}^{1}-p_{ij}^{0})R_{\boldsymbol{% \omega_{j}}}^{\pi}}{1+\beta\sum\limits_{\boldsymbol{\omega_{j}}\in\Omega(% \boldsymbol{\omega_{0}})}(p_{ij}^{1}-p_{ij}^{0})T_{\boldsymbol{\omega_{j}}}^{% \pi}}.italic_λらむだ = divide start_ARG italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT ( italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT - italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ) italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT end_ARG start_ARG 1 + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT ( italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT - italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ) italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_πぱい end_POSTSUPERSCRIPT end_ARG . (33)

Now define

$A_{{𝝎}_i}^{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}$Ωおめが\displaystyle A_{\boldsymbol{\omega}_{i}}^{\Omega}italic_A start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT $:=1+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}{\displaystyle \sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}}(p_{ij}^1-p_{ij}^0)T_{{𝝎}_j}^{{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^c},$Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑇subscript𝝎𝑗superscriptΩおめが𝑐\displaystyle:=1+\beta\sum\limits_{\boldsymbol{\omega_{j}}\in\Omega(% \boldsymbol{\omega_{0}})}(p_{ij}^{1}-p_{ij}^{0})T_{\boldsymbol{\omega}_{j}}^{% \Omega^{c}},:= 1 + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT ( italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT - italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ) italic_T start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが start_POSTSUPERSCRIPT italic_c end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT , (34)

$W_{{𝝎}_i}^{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}$Ωおめが\displaystyle W_{\boldsymbol{\omega}_{i}}^{\Omega}italic_W start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが end_POSTSUPERSCRIPT $:=R_{{𝝎}_i}+\mathrm{<ruby><rb>\beta </rb><rt>べーた</rt></ruby>}{\displaystyle \sum _{{𝝎}_{𝒋}\in \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}({𝝎}_{\mathrm{𝟎}})}}(p_{ij}^1-p_{ij}^0)R_{{𝝎}_j}^{{\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^c}.$Ωおめがsubscript𝝎0superscriptsubscript𝑝𝑖𝑗1superscriptsubscript𝑝𝑖𝑗0superscriptsubscript𝑅subscript𝝎𝑗superscriptΩおめが𝑐\displaystyle:=R_{\boldsymbol{\omega}_{i}}+\beta\sum\limits_{\boldsymbol{% \omega_{j}}\in\Omega(\boldsymbol{\omega_{0}})}(p_{ij}^{1}-p_{ij}^{0})R_{% \boldsymbol{\omega}_{j}}^{\Omega^{c}}.:= italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_βべーた ∑ start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT bold_italic_j end_POSTSUBSCRIPT ∈ roman_Ωおめが ( bold_italic_ωおめが start_POSTSUBSCRIPT bold_0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT ( italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT - italic_p start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT ) italic_R start_POSTSUBSCRIPT bold_italic_ωおめが start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_Ωおめが start_POSTSUPERSCRIPT italic_c end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT . (35)

For the auxiliary arm, we have $A_{0}^{\{0\}}=1$ and $W_{0}^{\{0\}}=\lambda$λらむだ. Generally, we can extend these variables to the multi-arm case. Assume that there are two arms and their state spaces are $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}(𝝎)$Ωおめが𝝎\Omega(\boldsymbol{\omega})roman_Ωおめが ( bold_italic_ωおめが ) and ${\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^{\prime }({𝝎}^{\prime })$Ωおめが′superscript𝝎′\Omega^{\prime}(\boldsymbol{\omega}^{\prime})roman_Ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( bold_italic_ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) with $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}(𝝎)\cap {\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^{\prime }({𝝎}^{\prime })=\mathrm{\varnothing }$Ωおめが𝝎superscriptΩおめが′superscript𝝎′\Omega(\boldsymbol{\omega})\cap\Omega^{\prime}(\boldsymbol{\omega}^{\prime})=\emptysetroman_Ωおめが ( bold_italic_ωおめが ) ∩ roman_Ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( bold_italic_ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) = ∅. Then for $\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}\subset \mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}(𝝎)$ΩおめがΩおめが𝝎\Omega\subset\Omega(\boldsymbol{\omega})roman_Ωおめが ⊂ roman_Ωおめが ( bold_italic_ωおめが ), ${\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^{\prime }\subset {\mathrm{<ruby><rb>\Omega </rb><rt>おめが</rt></ruby>}}^{\prime }({𝝎}^{\prime })$Ωおめが′superscriptΩおめが′superscript𝝎′\Omega^{\prime}\subset\Omega^{\prime}(\boldsymbol{\omega}^{\prime})roman_Ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ⊂ roman_Ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( bold_italic_ωおめが start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) and