Q-CE: Self-organized Cognitive Engine based on Q-learning

One of the main challenges in Cognitive Radio Networks (CRNs) is that the dynamic radio environment affects the Quality of Service (QoS) requirements of Secondary Users (SUs). Another challenge is how to predict Primary Users (PUs) activities over licensed channels, to avoid interfering with PUs. So, there is a need to implement a Cognitive Engine (CE) as a self-organized entity in a CR that overcomes those challenges. In our previous studies, an algorithm called Adaptive Discrete Particle Swarm Optimization (ADPSO) combined with Case-Based Reasoning (CBR) has been proposed for CE. ADPSO selects the optimal configurations when an unknown environment is countered while CBR allows usage of previous knowledge in an environment that has been previously observed. CBR however depends on a single observation of the previous state and gives inaccurate results where an individual states' performance changes. Another problem is how to find the best action when the environment is changing dynamically. In this paper, we propose a self-organized Q-Learning-based CE (Q-CE) which: 1) autonomously adapts the link configuration; 2) applies the previous action under similar environments and 3) where the environment changes, Q-CE learns from radio environment behavior and PU activities the best action to apply, in order to achieve QoS requirements and avoid interfering with PUs. The proposed CE combines following methods: ADPSO for link configuration; CBR for fast reasoning under similar environment; and Q-Learning to learn the environmental behavior. The results show improvements of about 67% in the achieved throughput, about 50% in signaling overhead when compared with the previous solutions that use only ADPSO and CBR.