Heat transfer and melt flow of keyhole, transition and conduction modes in laser beam oscillating welding

•Hybrid heat source considered the vapor heat influence and laser-material interaction was developed.•Molten pool evolutions and keyhole dynamics during keyhole, transition and conduction modes were analyzed.•Reflections of laser rays in the keyhole and the energy absorptions were predicted.•Penetration spiking defect influenced by keyhole stability was studied.