the books did have a lot more of that, most of it was cut for the show
much more time warging
tons more prophecies
marwyn the mage
lots more with glass candles and the cult of starry wisdom
euron greyjoy is magical and eldritch in his villainy and is foreshadowed to cause the 2nd long night be calling down a meteor strike
there's merfolk called squishers and they are real and eat people. Biter is a half-breed with one such creature
the white walkers are more ghosty and ethereal and less zombie guys
) he just started training to become a Maester, which he does in the show but iirc they kind of rush the whole thing, their more like scientists/philosophers/alchemists than actual wizards but he is his apprenticing with a Maester who knows ravenspeach which is implied to be kind of magical, and has come into contact with Glass Candles which are basically non-evil lotr Palantirs which I don't think were in the show at all? so he is brushing up against magic a bit