I found that even Introduction to Statistical Learning made a few too many assumptions when I tried to work through it. I recently finished Jim Hefferon's Linear Algebra [1] and now I'm working through Introduction to Applications of Linear Algebra: Vectors, Matrices, and Least Squares [2] (along with a python companion [3]). The two texts have overlaps but I've found them more helpful than redundant; it's nice to hear different angles on the same topic. I'm planning to focus on statistics next with Blitzstein and Hwang's Introduction to Probability [4] before returning to ISLR.