Its pretty old so maybe just useful for inspiration but the old Amiga/Atari game Dragon Lord / Dragon's Breath (Europe) has a very interesting custom magic system in which you can create unique spells using combinations of various alchemical reagants.
FWIW I think it's a very difficult thing to create such a system, without making it a tedious exercise in testing ingredient combinations.