Proximal Policy Optimization for Efficient Channel Allocation with Quality of Service (QoS) in Cognitive Radio Networks

Mahanty Rashmita; Kalyana Chakravarthy Chilukuri; N Chaitanya Kumar; T. Vidhyavathi; Regidi Suneetha; V Sita Rama Prasad; Badugu Samatha

doi:10.52756/ijerr.2024.v46.026

Authors

Mahanty Rashmita Department of Basic Sciences and Humanities, Vignan’s Institute of Engineering for Women, Visakhapatnam, Andhra Pradesh, India https://orcid.org/0000-0001-9247-8295
Kalyana Chakravarthy Chilukuri Department of Computer Science and Engineering, MVGR College of Engineering(A), Vizianagaram, Andhra Pradesh, India https://orcid.org/0000-0002-8137-7074
N Chaitanya Kumar Department of Information Technology, Anil Neerukonda Institute of Technology and Sciences(A), Visakhapatnam, Andhra Pradesh, India https://orcid.org/0009-0006-0535-7174
T. Vidhyavathi Department of Electronics and Communication Engineering, Anil Neerukonda Institute of Technology and Sciences(A), Visakhapatnam, Andhra Pradesh, India https://orcid.org/0000-0003-3609-729X
Regidi Suneetha Department of Electronics and Communication Engineering, Sanketika Vidya Parishad Engineering College, Visakhapatnam, Andhra Pradesh, India https://orcid.org/0000-0003-2204-5480
V Sita Rama Prasad Department of Computer science and Engineering, Vignan's Institute of Engineering for Women, Visakhapatnam, Andhra Pradesh, India https://orcid.org/0009-0002-4924-8712
Badugu Samatha Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Green Fields, Vaddeswaram, Andhra Pradesh, India https://orcid.org/0000-0003-1353-2797

DOI:

https://doi.org/10.52756/ijerr.2024.v46.026

Keywords:

Reinforcement learning, channel allocation, QoS, Q-Learning, actor-critic, PPO

Abstract

A multi-variable relationship exists in Cognitive Radio Networks (CRNs) where factors such as Energy efficiency, Throughput, Delay and Signal Noise Ratio (SINR) are related. The SINR shows the quality of the signal and is defined as the total power of a specific signal over the total power of an inter signal plus noise. This work proposes an effective energy and delay-efficient channel allocation strategy for CRNs (Cognitive Radio Networks) using Q-Learning and actor-criticism algorithms that maximize rewards. We also propose a Proximal Policy Optimization (PPO) algorithm that uses clipping of surrogate objectives to prevent large policy changes and ensure that the other parameters remain stable over time. We study the tradeoff between rewards, energy efficiency and other parameters and compare the algorithms with respect to the same. Results show that the proposed PPO method, while using optimally increased energy consumption, significantly reduces the delay, improves the thought and reduces the packet loss ratio for efficient channel allocation. This is positive with our findings shown in the results section and by comparing the proposed method with other algorithms to identify improved throughput and channel utilization. As the simulation results indicate that the PPO algorithm has very high throughput and significantly minimizes the delay and packet loss, it is suitable for application in all sorts of services such as video, imaging or M2M. The results are also compared with two of the existing channel allocation schemes and they confirm that the proposed algorithm performs better in terms of throughput discussed in one scheme and channel efficiency in the other.

References

Azaly, N., Badran, E., Kheirallah, H., & Farag, H. (2020). Centralized Dynamic Channel Reservation Mechanism via SDN for CR Networks Spectrum Allocation. IEEE Access, 8, 192493-192505. https://doi.org/10.1109/ACCESS.2020.3032666

Chakraborty, T., & Misra, I. S. (2020). A novel three-phase target channel allocation scheme for multi-user Cognitive Radio Networks. Computer Communications, 154, 18–39. https://doi.org/10.1016/j.comcom.2020.02.026

Dey, S., & Misra, I. (2018). A Novel Content Aware Channel Allocation Scheme for Video Applications over CRN. Wireless Personal Communications, 100. https://doi.org/10.1007/s11277-018-5650-4

El-Toukhy, A., Tantawy, M., & Tarrad, I. (2016). QoS-driven channel allocation schemes based on secondary users' priority in cognitive radio networks. International Journal of Wireless and Mobile Computing, 11(1), 91. https://doi.org/10.1504/IJWMC.2016.080182

Hossen, M. A., & Yoo, S.J. (2019). Q-Learning Based Multi-Objective Clustering Algorithm for Cognitive Radio Ad Hoc Networks. IEEE Access, 7, 181959–181971. https://doi.org/10.1109/access.2019.2959313

Jang, S.J., Han, C.H., Lee, K.E., & Yoo, S.J. (2019). Reinforcement learning-based dynamic band and channel selection in cognitive radio ad-hoc networks. EURASIP Journal on Wireless Communications and Networking, 2019(1). https://doi.org/10.1186/s13638-019-1433-1

Kumar, A., Dutta, S., & Pranav, P. (2023). Prevention of VM Timing side-channel attack in a cloud environment using randomized timing approach in AES – 128. Int. J. Exp. Res. Rev., 31(Spl Volume), 131-140. https://doi.org/10.52756/10.52756/ijerr.2023.v31spl.013

Madhuri, T. N. P., Rao, M. S., Santosh, P. S., Tejaswi, P., & Devendra, S. (2022). Data Communication Protocol using Elliptic Curve Cryptography for Wireless Body Area Network. 2022 6th International Conference on Computing Methodologies and Communication (ICCMC), pp.133–139. https://doi.org/10.1109/iccmc53470.2022.9753898

Malik, S. A. (2020). Efficient channel allocation using matching theory for QoS provisioning in radio networks. Sensors, 20(7), 1872. https://doi.org/10.3390/s20071872

Nayak, D., Muduli, A., Hussain, M., Mirza, A., Gummadipudi, J., & Kumar, N. (2020). Channel allocation in cognitive radio networks using energy detection technique. Materials Today: Proceedings, 33. https://doi.org/10.1016/j.matpr.2020.06.491

Nayani, A. S. K., Sekhar, C., Rao, M. S., & Rao, K. V. (2021). Enhancing image resolution and denoising using autoencoder. In Lecture Notes on Data Engineering and Communications Technologies, pp. 649–659. https://doi.org/10.1007/978-981-15-8335-3_50

Pal, R., & Dahiya, S. (2022). Optimal Channel Selection algorithm for CRahNs. Physical Communication, 54, 101853. https://doi.org/10.1016/j.phycom.2022.101853

Pal, R., Gupta, N., Prakash, A., Tripathi, R., & Rodrigues, J. J. P. C. (2020). Deep reinforcement learning based optimal channel selection for cognitive radio vehicular ad?hoc network. IET Communications, 14(19), 3464–3471. Portico. https://doi.org/10.1049/iet-com.2020.0451

Pavan, M.N., Kumar, S., Nayak, G., & Narender, M. (2024). Deep Reinforcement Learning based channel allocation (DRLCA) in Cognitive Radio Networks. Journal of Electrical Systems.

Srivastava, A., Pal, R., Prakash, A., Tripathi, R., Gupta, N., & Alkhayyat, A. (2024). Optimal Channel Selection and Switching Using Q-Learning in Cognitive Radio Ad Hoc Networks. IEEE Transactions on Consumer Electronics, 70(3), 6314–6326. https://doi.org/10.1109/tce.2024.3413333

Tlouyamma, J., & Velempini, M. (2021). Channel Selection Algorithm Optimized for Improved Performance in Cognitive Radio Networks. Wireless Personal Communications.

Varshney, P., Singh, R. P., & Jain, R. K. (2024). Performance Analysis of Millimeter-Wave Propagation Characteristics for Various Channel Models in the Indoor Environment. International Journal of Experimental Research and Review, 44, 102–114. https://doi.org/10.52756/IJERR.2024.v44spl.009

Wang, Y.H., & Liao, S.L. (2018). Dynamic channel allocation scheme in CRN applied fuzzy-inference system. Journal of Computers (Taiwan). 29, 141-155. http://dx.doi.org/10.3966/199115992018062903013.

Xu, Q., Li, S., Gaber, J., & Han, Y. (2024). Modelling Analysis of Channel Assembling in CRNs Based on Priority Scheduling Strategy with Reserved Queue. Electronics, 13(15), 3051. https://doi.org/10.3390/electronics13153051

Ye, Z., Wang, Y., & Wan, P. (2020). Joint Channel Allocation and Power Control Based on Long Short-Term Memory Deep Q Network in Cognitive Radio Networks. Complexity, 2020, 1-11. http://dx.doi.org/10.1155/2020/1628023

Zhang, M., Zhu, X., Jiang, H., Bian, T., & Yang, Y. (2023). A dynamic channel allocation protocol based on data traffic characterization for cognitive-radio wireless sensor networks. SSRN. http://dx.doi.org/10.2139/ssrn.4457362

Zhao, G., Li, Y., Xu, C., Han, Z., Xing, Y., & Yu, S. (2019). Joint Power Control and Channel Allocation for Interference Mitigation Based on Reinforcement Learning. IEEE Access, 7, 177254–177265. https://doi.org/10.1109/access.2019.2937438