Towards vendor-agnostic real-time optical network design with extended Kalman state estimation and recurrent neural network machine learning [Invited]

The network operator's call for open, disaggregated optical networks to accelerate innovation and reduce cost, make progress in the standardization of interfaces, and raise telemetry capabilities in optical network systems has created an opportunity to adopt a new paradigm for optical network design. This new paradigm is driven by direct measurement and continuous learning from the actual optical hardware deployed in the field. We report an approach towards practical, vendor-agnostic, real-time optical network design and network management using a combination of two learning models. We generalize our physics-based optical model parameter estimation algorithm using the extended Kalman state estimation theory and, for the first time, to the best of our knowledge, present results using real optical network field data. An observed 0.3 dB standard deviation of the difference between typical predicted and measured signal quality appears mostly attributable to transponder performance variance. We further propose using the physics-based optical model parameter values as inputs to a second learning model with a recurrent neural network such as a gated recurrent unit (GRU) to allocate the appropriate required optical margin relative to the typical signal quality predicted by the physics-based optical model. A proof of concept shows that for a dataset of 3000 optical connections with a wide variety of amplified spontaneous emission noise and nonlinear noise limited conditions, a 10-hidden-unit 2-layer GRU was sufficient to realize a margin prediction error standard deviation below 0.2 dB. This approach of measurement data-driven automated network design will simplify deployment and enable efficient operation of open optical networks.