Yes, there has to be a minimal level of trust between the server and devices/users. You're level of security defines what point the computer decides "yep, that's good enough, I can trust this is the real user accessing me." A true, perfectly secure system has no access, it's a black box that nothing can interact with, because it can only trust itself.

At some point you have to trust yourself not to mess up too bad, you are the weak point in security, since I'm assuming you're the only one who's accessing the system right now.

I personally use plaintext password files, with appropriately managed permissions (only the owner can see or read the file.) As long as the user login is secure, and root/admin access is secure, I feel comfortable that no one but me can access the credentials. To manage remote access to the system, I use hardware (YubiKey) to store my SSH keys, with a PIN code lock that wipes the keys if entered incorrectly 5 times. I don't have any government agencies coming after me (as far as i know) so no one has a practical way to extract the keys if the device were stolen off me, and the PIN retry limit prevents brute forcing. I trust myself to manage these hardware keys appropriately.

*Edit: to add to the "appropriately managed" bit there, each sub system (home automation, file server, media hosting, etc) should be properly containerized or isolated (using different user accounts) so that if one service is compromised, the others are still somewhat protected. *

Physical access to your server is endgame. If an attacker can physically mess with your system, you've lost, and that can only be fixed externally with home security improvements. A skilled attacker doesn't need your ssh tokens to gain access if they can plug a keyboard into the server itself. I've also seen a demo of a neat little kit the feds use to seamlessly move a computer power cord from the wall to a portable battery pack, so they can simply walk out, with the device still powered up, and do what they want to it back at home base (it's used mainly for raids on various computer fraudsters, but still, it exists, and can be used on you). I trust myself not to do stupid stuff that gets me targeted by a hacker group, or raided by the FBI.

Something less targeted, like a burglary (not focused on your server), can be protected against by disk encryption. I don't use any disk encryption, but I probably should. I like the idea other user's wrote down here, of using a TPM module to store disk encryption keys, so it can detect if the hardware or OS changed between boots and deny decryption. I'd also take it one step further and encrypt the data/password files with an encryption method that requires someone to log in and type a password, that way if someone were to steal the device and power it up elsewhere, the passwords are still safe until you OK it, essentially authorizing unexpected reboots, at the cost of having to log into the server every time it starts (not fun if you're doing maintenance.) if you do this bit right, you don't have to trust yourself to do anything but remember the password.

Sorry about the length there, but security and access management is a complicated topic, so it requires a lot of talking. Hopefully it helped!