Interpretability of deep neural networks has focused on the analysis of individual samples. This can distract our attention from patterns originating at the distribution of the dataset itself. We broaden the scope of interpretability analysis, from individual images to entire datasets, and found that some high-performing classifiers use less than half the information contained in any given sample. While the learned features are more intuitive to visualize for image-centric neural networks, in time-series it is much more complicated as there is no direct interpretation of the filters and inputs as compared to image modality. In this talk we are presenting two approaches to analyze the behavior of networks with respect to used input signal, which pave the way to go beyond simple layer stacking and towards a more principled design of neural networks.