I timed my DSi; it varies a lot by game but the largest I tried (Pokemon Black 2) loaded in around 25 seconds on the first run, then loads in under 10 seconds on subsequent runs.

I recently started messing with my DSi again after a few years away from it, and you're definitely not alone in being confused by it all - I really have no idea what exactly I did to it. To refresh everything I ended up just wiping the SD card, then putting Twilightmenu on it. I can still boot into unlaunch so I must have modified the firmware / internal storage before, but I don't know if hiyaCFW is something that lives on there on the SD card.

Speaking of which, that's the same setup as my SD card, which is a 32gb Transcend. I can't seem to find my SD card reader to double check but I'm pretty sure of using 32k cluster size.

Might be worth mentioning, I have another DSi with a wildly different system boot time for no apparent reason. I set it up the exact same, but it takes 15+ seconds to boot vs the <5 of my main DSi, even when I swap SD cards. The only difference between them is that the one that takes a long time to boot is Japanese. Games take about the same time to boot so it doesn't seem directly related to your issue but who knows.