Online Learning Based Joint Gateway Selection and User Scheduling in Non-Stationary Air-Ground Networks

Air-ground networks have emerged as a promising paradigm for enhancing mobile user coverage and quality of service. In such networks, the system needs to select appropriate gateways as relays between ground and air, and meanwhile schedule user transmissions. Due to the heterogeneity of radio access technologies, the joint gateway selection and user scheduling (GSUS) becomes a crucial yet challenging problem for maximizing network resource utilization. However, when terrestrial users access the continuously moving aerial access point, the ground-to-air channel state becomes dynamic and non-stationary, which reduces the effectiveness of optimization-based techniques and conventional Reinforcement Learning (RL) methods for solving the GSUS problem. In this paper, we design an intelligent GSUS (iGSUS) scheme by incorporating representation learning into the RL framework to tackle the non-stationarity. Specifically, we use Dynamic Parameter Markov Decision Process to decompose the non-stationary MDP into a sequence of stationary MDPs. These MDPs are encoded with latent parameters by representation learning, enabling the RL algorithm to efficiently learn and exploit appropriate GSUS policies in an online learning manner. Simulation results show that the proposed iGSUS scheme is significantly better than several benchmarks in utility, average network throughput and packet loss rate, showcasing its adaptability in non-stationary air-ground networks.