Deep transform and metric learning network: Wedding deep dictionary learning and neural network

•Reformulate Dictionary Learning (DL) as Transform and Metric learning.•The above reformulation is implemented as a tandem of a linear layer and an RNN.•This is the first work which bridges deep DL and the combination of linear and RNNs.•The proposed method demonstrates efficiency, scalability and discrimination power.•The combinations of CNN architectures achieve better accuracy and stronger robustness.