Constructing and Deploying AI Native Physical layer using Wireless Autoencoders for Enhanced RF and Optical Wireless Communication

Abstract

Conventional wireless communication systems rely on fixed, block-based signal processing pipelines that are manually optimized and lack the ability to adapt in real time to dynamic channel conditions, hardware impairments, or varying modulation requirements. These limitations make them increasingly inefficient in modern communication environments characterized by mobility, interference, and diverse quality-of-service demands. As wireless standards like IEEE 802.11 and beyond continue to evolve, there is a growing need for intelligent transceivers that can adapt their complexity, structure, and behavior on demand. This dissertation addresses that need by introducing a deep learning-based framework that replaces static transceiver components with dynamic neural networks capable of learning, adapting, and co-optimizing over the air.This dissertation presents a unified deep learning framework for designing intelligent, modulation-adaptive wireless communication systems. Motivated by the limitations of conventional block-based architectures in coping with dynamic channel conditions and diverse modulation schemes, this work proposes a novel end-to-end system architecture composed of dynamic neural networks (DNNs) at both the transmitter and receiver. The study begins by developing multiple decoder architectures capable of runtime structural adaptation, including modulation-aware network switching, dynamic weight reloading, and early exit inference. Each decoder strategy is tailored for QPSK, 16-QAM, and 64-QAM, enabling complexity-efficient decoding while improving block error rate (BLER) under varying SNR conditions. The dissertation then introduces a dynamic neural encoder based on modulation-aware network switching. A single architecture houses specialized subnetworks for each modulation order, and conventional channel state information (CSI) is used to select the appropriate path at runtime. This encoder is trained using constellation points generated by a standard IEEE 802.11n stack via a vector signal generator, with performance evaluated in terms of error vector magnitude (EVM). The encoder and decoder are subsequently integrated into a full end-to-end autoencoder-based communication system. Fine-tuning is applied to jointly optimize the encoder and decoder using both simulated and real over-the-air data collected via a USRP X310 software-defined radio. The final system is evaluated comprehensively across multiple metrics—BLER, EVM, floating-point operations (FLOPs), and energy consumption—demonstrating significant gains over conventional systems. Among the decoder strategies, dynamic weight reloading achieves the best overall performance, while early exit provides the most efficient inference for low-order modulations. This work demonstrates the feasibility and effectiveness of dynamic deep learning architectures for next-generation physical layer design. By leveraging runtime adaptability, structural modularity, and joint optimization, the proposed system offers improved robustness, efficiency, and scalability. The findings lay the groundwork for intelligent, resource-aware transceivers capable of operating effectively across a wide range of wireless environments.