I'd start with internal services first before opening yourself up to the outside, starting with Jellyfin possibly. Docker is worth learning as it's agnostic to the system you are using and helps with prototyping/testing quickly - something invaluable as you learn and make mistakes in your journey. From maintaining the fun stuff, such as your media service, you'll learn the other stuff, like ssh keys, Reverse proxy &c: Iterative improvement is the game when it comes to your homelab and yourself.

First: Good for you, enjoy the journey! Second: Just as others already pointed out, Mastodon is not really a beginner project. You want to understand what you are doing, not just make everything work no matter what. Some reasons why I'd not start with Mastodon:

• Complex deployment stack (for beginners)
• Needs regular maintenance
• Security considerations (if you haven't managed/hardened a server before)
• Long term project

So instead: Have a look at awesome-selfhosted for ideas. A personal dashboard, photo gallery or a PiHole/AdGuard is a good start.

About Docker; it's a bit more than just dependency separation. It's a kind of virtualization, but without each container running it's own kernel. Advantage is: Docker images run (with some configuration) relatively lightweight out of the box. So there's no need to install the applications natively. While I'm a great fan of Docker, you'd probably learn more installing things natively in the beginning. Or maybe do both, it's up to you. However, if you decide to use Docker, be sure to understand what's going on under the hood. That's where the fun begins. Everyone can pull and start images, but not everyone knows how to customize or build them themselves.

No matter what you decide to do, have fun. And if you've any questions, there's plenty of documentation online or just ask. The selfhosting community is very welcoming towards new members ;)

Can't answer NixOS bits as I've not used it, but a few points to shoot off:

• Starting with a Mastodon instance should not be the first step. Setting up an nginx webserver that says hello world! should.
• There's not much to learn about using SSH as much as setting it up. Key-only auth is a must. Changing the port to something high is a convenience - I hate my logs getting spammed by all the bots. Some are more than annoying enough. I have also enabled rate limiting on my SSH port for that reason alone, using iptables. I really need to have a look at nftables as that is faster and better.
• Domain is only really necessary if you want to reach your stuff externally. Mastodon would count as such usecase. Whether you go proper paid or free - up to you, but do keep in mind the recent Mali situation into account. Domains are cheap anyway.
  • As a sidenote - you can set up your own authoritative DNS, your own VPN and then use whatever domain you desire. You can make your internal DNS serve your internal website on google.com if you wish :) It would, of course, only work on devices using your own DNS as the resolver. You would also most likely want to roll your own CA for that, but I'm getting distracted here :)

In short - go for it! The worst that can happen is you will learn invaluable skills and have to start over.

You don't require docker to self host. You have a lot to learn, so trimming down the amount of things your doing is the right idea. Ignore it for now.

You don't have to buy a domain, you can use your IP directly, or use a free dynamic dns service.

Mastadon may be too complicated for a first host for a beginner. I would recommend trying something simpler first, but it's your hobby so do it how you want to.

As others have suggested I would stick to internal only until you're comfortable. Dealing with automated port scanning from all over the globe is a nightmare if you aren't running a tight ship, and a collasal headache to deal with if they get in. I started with pihole, jellyfin, and some other simple docker services. Check out linuxserver.io if you go the docker route, they have good prebuilt compose files for you to use and the community has good info on their forums. You don't need a domain but it's preferred if you want to host something public-facing. Make sure you have fail2ban configured if you open stuff up to the web, it makes dealing with the automated attacks easier.

if you still haven't done it, here's a good video that explains how to setup ssh keys https://youtube.com/watch?v=ZKZbPZCnHRE&feature=share7