Following other fields of science, Deep Learning models are gaining attention within the statistical physics community as a powerful and efficient way for analysing experimental and synthetic time series, and for quantifying properties thereof. Applying such models is nevertheless a path full of pitfalls, not only due to their inherent complexity, but also to a lack of understanding of some of their idiosyncrasies. We here discuss some of these pitfalls in the context of time series classification, covering from the selection of the best model hyperparameters, how the models have to be trained, to the way data have to be pre-processed. While not providing one-fits-all answers, the statistical physics practitioner will here find what questions ought to be posed, and a first guide about how to tackle them.