Cannabis was a big factor for me. My sperm morphology improved after I ceased using it. Anecdotal, but there are a few studies that support the idea (cannabis in general doesn't have a lot of studies because it is a Schedule 1).
If you're a heavy user, it can't hurt to stop while you're trying, and it may help. (I don't see it mentioned often in these discussions.)