IMO, David Spivak's book[0] is much less technical than anything coming from Bartosz Milewski. In fact, I think Spivak's book is simpler than that of Lawvere. If you know a little naive set theory, Spivak's book is a breeze.
Learn to read and do math proofs. Then learn some abstract algebra and general topology so that you can understand examples you'll meet in category theory. Then, bam, you're ready for category theory.
The other way is to dive straight into category theory. Doable, but difficult. You'll be learning how to parse math arguments and argumentation style specific to category theory and examples from other branches of math along with the theory(from those branches AND category theory itself). Tough shyt if you ask me.
To get started with learning how mathematicians present and defend their arguments, there's free and really good Book of Proof by Richard Hammack[0].
[0] Talking about the one "for scientists".