While I wouldn't say that the problem has been "well studied", the research community has been paying interest, and some progress has been made, most notably- as pointed out by benjaminjackman- LIME[1]. Roughly speaking, LIME learns a locally approximate model which can be interpreted. It will work will any black box model, not just neural networks.
For someone who wants an in-depth look at sci-fi interfaces from a designer's perspective, https://scifiinterfaces.wordpress.com/ and the corresponding book [1] are pretty good resources. The book was even featured on YC's Winter (Summer?) reading list.
[1] https://www.reddit.com/r/privacy/comments/76c605/airborn_os_...