Is it just me, or is Mage Hand useless? Like, impressively useless.
I'm new to DnD (and DnD-like games) so I could be in the wrong here, but every time I think of something that Mage Hand should be able to do, it just doesn't.
Want to pick up an unreachable item and carry it to me? No.
Want to loot an otherwise unreachable body? No.
Want to pickpocket someone without being caught? No. (Okay, I get that this one would be pretty broken, even in normal DnD this is sometimes disallowed)
Want to fly a few feet up and light an overhead brazier? No.
Want to do literally anything useful? No.
Want to squeeze through a small hole to see a room you've already looted? Sure!
I'm at the point where instead of trying to think creatively about how to use it, I just immediately write it off because it probably can't do whatever I'm thinking of. I am genuinely surprised when I find out Mage Hand can do something, and that's not a good thing.
The only idea I had that actually worked was using it to stealth the early phase spider section by just throwing the gem at the end backwards, then moving the hand in the opposite direction to draw aggro. That's literally the only "useful" thing I've done with it, and I've still not found a use for the gem.
So I ask, what have you done with Mage Hand that's actually useful?
You're right that you can come up with pretty good ways to challenge players with certain spells. The problem is that it can be pretty difficult to do on the fly. Assuming the party goes in a direction you haven't really prepped for, they're are a lot of abilities that can make it trivial if you forget about them.
There's a really big, tedious, ongoing discussion on exactly how overrated 5e D&D is and what type of game it wants to be, but it's fair to say the system has a lot of small things that trivialize challenges. Goodberry means you never have to worry about food ever again. Fly means physical distance is not much of a problem. Pass without trace means stealth will almost always work. Leomunds tiny but means sleeping is almost always safe.
All of these examples can be fixed. Goblins can stack a bunch of rocks on leomunds hut for example. The problem is that it gets repetitive and forced to counter everything all the time.
I agree though that the developers have done a really good job trying to handle all the complexity of turning a tabletop RPG into a video game.
Ok, second part is to take a step back, and instead of trying to counter the players, just make the world a living one. Have the enemies be smart. Think in terms of factions and resources instead of individuals and encounters.
Even the wildest monsters can be smart predators, not because they're smart, but because they're predators.
Another question you're bringing is wilderness survival. With this, you're trying to make dnd 5e something it absolutely isn't: a survival game. Survival in dnd is abstract and easy up to lvl5. After that, it's not supposed to be a difficult, which is why you have spells like create food and water.
But these spells are not free. A party with this spell and leomund's tiny hut means two spellcasters are down one spell when the day begin. At level5 that means only 1 of the important lvl3 spells. And if they can't cast it, they can't rest, because I doubt they will have provisions and camp if they rely on this. And if they don't have comfortable enough rest or no ration, they can't benefit from a long rest.
Time is usually the resource you need to constrain. For each day that passes, the vilain should have something going. This way the players will have to manage their rest properly.
Back to the resources of the enemies. A big mistake many seem to do is to give the monopoly of magic to the players. But it is ridiculous to do it. Enemies should have available magic one way or another. No an infinite supply of it, but they would have some. How many sorcerers, warlocks and clerics in a colony of goblins? Certainly quite some. Below lvl5 is apprentice level. Lvl5 to 9 is expert level. You can have a bunch of apprentices and a handful of experts in a colony. And then dispel magic, counterspell and everything is also available to you, the dm.
Before tier3, you have easy solutions for each and any trick a spellcaster can come up with. If they use their spells for utility, they will very quickly run out of them, and they won't have them for combat.
Usually people who prefer PF2 will have a bias against 5e balance. It's a bias because both games have different philosophies about it. It's perfectly valid to hate one and love the other, or the opposite. PF2 is more about the encounter and the tactic. 5e is more about strategy and finding ways around the obstacles.