LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality - POLARIS - Performance analysis and Optimization of LARge Infrastructure and Systems Access content directly
Journal Articles Mathematics of Operations Research Year : 2023

LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality

Abstract

We provide a framework to analyse control policies for the restless Markovian bandit model, under both finite and infinite time horizon. We show that when the population of arms goes to infinity, the value of the optimal control policy converges to the solution of a linear program (LP). We provide necessary and sufficient conditions for a generic control policy to be: i) asymptotically optimal; ii) asymptotically optimal with square root convergence rate; iii) asymptotically optimal with exponential rate. We then construct the LP-index policy that is asymptotically optimal with square root convergence rate on all models, and with exponential rate if the model is non-degenerate in finite horizon, and satisfies a uniform global attractor property in infinite horizon. We next define the LP-update policy, which is essentially a repeated LP-index policy that solves a new linear program at each decision epoch. We provide numerical experiments to compare the efficiency of LP-based policies. We compare the performance of the LP-index policy and the LP-update policy with other heuristics. Our result demonstrates that the LP-update policy outperforms the LP-index policy in general, and can have a significant advantage when the transition matrices are wrongly estimated.
Fichier principal
Vignette du fichier
LP_finite_infinite.pdf (806.54 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03262307 , version 1 (18-06-2021)
hal-03262307 , version 2 (11-05-2022)
hal-03262307 , version 3 (13-09-2022)
hal-03262307 , version 4 (21-12-2023)

Licence

Attribution

Identifiers

Cite

Nicolas Gast, Bruno Gaujal, Chen Yan. LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality. Mathematics of Operations Research, 2023, pp.1-29. ⟨10.1287/moor.2022.0101⟩. ⟨hal-03262307v4⟩
245 View
434 Download

Altmetric

Share

Gmail Facebook X LinkedIn More