Demystifying Explainable AI (XAI)

What is explainable AI and how can and should it be used? Are explanations the “holy grail” to solve the black-box problem or should there be other non-technical solutions?