Deep learning models give state-of-the-art results on diverse problems, but their lack of interpretability is a major problem. Consider a model trained to predict which DNA mutations cause disease: if the model performs well, it has likely identified patterns that biologists would like to understand. However, this is difficult if the model is a black box. We present algorithms that provide detailed explanations for individual predictions made by a deep learning model and discover recurring patterns across the entire dataset. Our algorithms address significant limitations of existing interpretability methods. We show examples from genomics where the use of deep learning in conjunction with our interpretability algorithms leads to novel biological insights.