Observation of Kosterlitz-Thouless Metal-to-Insulator Transition in Quantum Anomalous Hall Insulators

Interlayer exchange coupling (IEC) between two magnetic layers sandwiched by a nonmagnetic spacer layer plays a critical role in shaping the magnetic properties of such heterostructures. The quantum anomalous Hall (QAH) effect has been realized in a structure composed of two magnetically doped topological insulator (TI) layers separated by an undoped TI layer. The quantized Hall conductance observed in this sandwich heterostructure originates from the combined contribution of the top and bottom surface states. In this work, we employ molecular beam epitaxy to synthesize a series of magnetic TI sandwiches with varying thicknesses of the middle undoped TI layer. The well-quantized QAH effect is observed in all these samples and its critical behavior is modulated by the IEC between the top and bottom magnetic TI layers. Near the plateau phase transition (PPT), we find that thinner QAH samples exhibit a two-dimensional critical metal behavior with nearly temperature-independent longitudinal resistance, whereas thicker QAH samples behave as a three-dimensional insulator with reduced longitudinal resistance at higher temperatures. The IEC-induced critical-metal-to-insulator transition in the QAH PPT regime can be understood through a two-channel Chalker-Coddington network model by tuning inter-channel tunneling. The agreement between experiment and theory strongly supports the QAH PPT within the Kosterlitz-Thouless framework, where the critical metal and disordered insulator phases exist in bound and unbound states of vortex-antivortex pairs, respectively.