I implemented an interpretable LSTM from this paper https://arxiv.org/pdf/1905.12034.pdf. It works by segmenting the context for each input variable, and then applying an attention layer to interpret each variables effect.
I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.