[-] jax@lemmy.cloudhub.social 3 points 1 week ago* (last edited 1 week ago)

Yeah for sure! I like to post about both the positive and negative experiences. I find things like that to be a valuable learning tool.

From a security perspective, it’s important to understand the systems you’ve implemented and test that they are working as expected. I think in that example if I had tested user sign-up sooner I could have caught the configuration issue.

It's also important to have good observability into your system, both metrics and logs. Metrics to help detect if something weird is happening (increased resource usage could point to ransomware or crypto mining) and logging to track down what happened and see what systems are impacted.

From a technical controls standpoint, it's good practice to segregate your applications from other systems and control planes like IPMI and switching/routing admin interfaces. It's also good to try to limit holes in your firewall. In this cluster, I have Cloudflare Tunnels setup so that I don't have to open ports to access web servers, and I get access to their WAF tooling. You could do something similar with a VPS running WireGuard, CrowdSec, and a reverse proxy.

[-] jax@lemmy.cloudhub.social 3 points 1 week ago* (last edited 1 week ago)

Not at all! I agree, and COVID didn't help at all. I do want to try and be accurate though :p

[-] jax@lemmy.cloudhub.social 4 points 1 week ago* (last edited 1 week ago)

Its possible that I estimated the timeline wrong 😅

I’ve added a note to the blog, thanks!

[-] jax@lemmy.cloudhub.social 5 points 1 week ago

Glad I could provide some insight! It’s not something I see talked about too much even on Reddit. Let me know if you have any questions or things I could flesh out more in the article!

I’m still relatively new to ActivityPub and Federated systems in general, though I’ve had my Lemmy and Mastodon instances for 8+ months now I don’t use them as much as I was expecting, sadly. Running your own instance can be very isolating and any content you put directly on your instance probably won’t gain much traction (at least on Mastodon, Lemmy seems to fair a bit better).

It’s one of a handful of blogs that I’ve run over the last couple of years, the other one that’s still online is HomeLab.Blog. I actually meant to run a federated blog platform like WriteFreely, but they don’t have a production docket image, and I saw that Ghost is planning on adding ActivityPub support.

This article might be more appropriate on that blog and an article about my experience with Federated systems might be more on-topic on this one. Oops.

43
Teachings from years of HomeLab (blog.cloudhub.social)

A slightly less technical post - these are some things I've learned from having a HomeLab for over a decade.

[-] jax@lemmy.cloudhub.social 4 points 1 week ago

Yeah, this seems like old news - cookies can be stolen, and FIDO doesn't change that unless you are prompting the hardware token for validation with every request (which isn't feasible for most things, though might be a good idea for sensitive actions).

[-] jax@lemmy.cloudhub.social 2 points 1 week ago

What's everyone's workflows with these systems? Do you catalog both physical and digital documents? Where do you store the documents?

[-] jax@lemmy.cloudhub.social 2 points 2 weeks ago

Yeah, I used to host a Matrix instance - could do that for this one too.

The issue is more about setting up the Kubernetes manifests and templating them. I usually use the chart's built-in postgres and redis config, though using an operator would make it more scalable for sure.

I'm using Authentik for auth, but I do also like Keycloak.

[-] jax@lemmy.cloudhub.social 2 points 2 weeks ago

That's actually super helpful! I haven't done much custom Helm chart-ing, and was kinda lost where to start. That really helps break the process down, and the tip about skipping state to start is very wise.

18
Fediverse Apps on Kubernetes? (lemmy.cloudhub.social)

cross-posted from: https://lemmy.cloudhub.social/post/347779

I am running a Kubernetes cluster for this domain, and I'm looking at more services to run (right now I have Mastodon and Lemmy).

I was considering WriteFreely and PixelFed, but they don't seem to have an easy solution for running on Kubernetes (WriteFreely doesn't even have a production-ready docker image).

Is anyone else running federated services in their lab? Do you run any of them on Kubernetes?

1
Fediverse Apps on Kubernetes? (lemmy.cloudhub.social)

I am running a Kubernetes cluster for this domain, and I'm looking at more services to run (right now I have Mastodon and Lemmy).

I was considering WriteFreely and PixelFed, but they don't seem to have an easy solution for running on Kubernetes (WriteFreely doesn't even have a production-ready docker image).

Is anyone else running federated services in their lab? Do you run any of them on Kubernetes?

1
submitted 2 weeks ago* (last edited 2 weeks ago) by jax@lemmy.cloudhub.social to c/homelab@lemmy.cloudhub.social

Reposting some of my older popular blog posts. This one is probably out of date, I doubt the configuration examples still work as they did back in 2020.

1
Internal SMTP relay? (lemmy.cloudhub.social)

I have a need for an internal SMTP relay inside a kubernetes cluster. What is everyone using for docker/kubernetes SMTP relays these days?

Goal is to have all internal services route emails through this relay and it in turn sends the emails out via SendGrid, should be a fairly easy task, just not something I've done for a few years.

1
submitted 9 months ago* (last edited 9 months ago) by jax@lemmy.cloudhub.social to c/homelab@lemmy.cloudhub.social

Hey all! We're back after a couple of weeks of downtime on Lemmy due to some DB migration issues + Kubernetes liveness timeouts, and general lack of time to troubleshoot. For the latest status, you can view the status page for the cluster here: https://cloudhub-social.github.io/Status/

We are also well overdue for a What's in Your Homelab for the month of August, so we'll use this post for that as well!

[-] jax@lemmy.cloudhub.social 3 points 10 months ago

Just because it’s not public facing doesn’t mean that it’s not an issue. It might be less of an issue, but it is still a massive vulnerability.

All it takes is one misconfiguration or other vulnerable system to use this as a jumping off point to burrow into other systems. Especially if this system has elevated access to sensitive locations within your network.

[-] jax@lemmy.cloudhub.social 1 points 10 months ago

… how the hell have I never heard of this?

1

Since it's been about a month since the last post, it's time for another one!

"What's in your homelab?" (July 2023)!

This could be anything from hardware to software to things your running in the cloud (#cloudlab).

Hardware and diagram pics are always welcome!

[-] jax@lemmy.cloudhub.social 1 points 10 months ago

I’m on Kubernetes, but it’d be the same via Docker - a volume mount (iirc at the same place it stores local data if you don’t use S3, should be in the docs)

[-] jax@lemmy.cloudhub.social 3 points 10 months ago

My instance has a couple users currently lol

0

When the following is true:

  • User attempts to create an account
  • Instance has "require registration application" enabled
  • Instance's email is not working/unavailable

the application seems to get lost, the user never receives an email (even after email functionality is restored), nor can that email/username be used going forward to re-submit the account creation request.

Additionally, since the user never verifies their email, the instance admin never gets a registration application.

It's not currently an issue for me, however, would it be possible to delete these ghost users? If you lookup the profile/username in the database, you can view it via the web UI, but the only options appear to be either blocking the user or banning them. It might be good to be able to completely delete the accounts, no?

1
submitted 11 months ago* (last edited 11 months ago) by jax@lemmy.cloudhub.social to c/sysadmin@lemmy.ml

cross-posted from: https://lemmy.cloudhub.social/post/14149

What's everyone using for status monitoring and/or status pages either in their lab or at work?

I setup a status page for my fediverse instances using Uptime Robot (have an existing subscription), and the features are kinda lacking. I feel like they haven't really updated anything in the last 5 years which is unfortunate.

1
submitted 11 months ago* (last edited 11 months ago) by jax@lemmy.cloudhub.social to c/homelab@lemmy.cloudhub.social

What's everyone using for status monitoring and/or status pages either in their lab or at work?

I setup a status page for my fediverse instances using Uptime Robot (have an existing subscription), and the features are kinda lacking. I feel like they haven't really updated anything in the last 5 years which is unfortunate.

0
submitted 11 months ago* (last edited 11 months ago) by jax@lemmy.cloudhub.social to c/espresso@infosec.pub

Title - I'm using lactose free milk right now, but I'm wondering if there are any good milk-free alternatives? I tried using Silk's barista almond milk, but it's sour after being frothed?

Edit: I guess I'll have to try some oat-based alternatives, maybe the problem is with the almond milk.

1

cross-posted from: https://lemmy.dcrich.net/post/1150

Boy howdy, there are a lot of people coming to the matrix chat trying to figure out how to get lemmy working on docker who are stuck on the official documentation. This document is my guide on how I got Lemmy working. I'll also share what I don't have working yet to inspire further.

Please feel free to steal anything you want from this and put it into the official docs. I don't know the contributing policy and it sounds hard and I'm busy at the moment.

Of note: I add a nginx container in this setup so that you don't have to do crazy hacks on your end for locations. If you already have an nginx reverse proxy that you are using, just use this one as a 2nd layer of nginx. There is low overhead, so don't worry about it.

Setup

For this guide, I'm requiring that you already have your own reverse proxy setup in place that can handle all the SSL termination. I'm doing this because I think that most people who are setting up Lemmy for the first time on Docker aren't setting up their first Docker container.

Because I'm requiring that you setup your own SSL termination (caddy, ACME, Nginx Proxy Manager, etc.) before you begin, I will not talk further about https, certificates, or rotation. But before I do: Don't host a website in 2023 that doesn't serve content securely. Make sure that you get your stuff setup, including any certificate rotation. If you don't get this setup completed, I suggest that you shouldn't continue or host a public website.

I also require that you be able to use docker-compose.

Get Files

Download these 3 files to your working directory from my github gist. You can download as zip or get them one at a time by scrolling down.

Prepare Working Directory

mkdir -p volumes/pictrs
sudo chown -R 991:991 volumes/pictrs

Edit Config Files

  • In the docker-compose.yml file, change the port, hostname, and database password.
  • In the lemmy.hjson file, change the admin username/password, hostname, database password, and email settings. You can take out the entire email section if you want to.
  • No changes to the nginx.conf file.

Start It Up

Now you're ready to start the containers!

You're pretty much good to go. Login to your lemmy instance. You should be able to use your docker host ip at your defined port OR via your reverse proxy lemmy domain host name.

docker-compose up

Watch the pretty log messages.

You should be able to curl your new admin user and get valid json back: curl -H 'Accept: application/activity+json' https://lemmy.yourdomain.net/u/yourAdminUser

Press Ctrl+C if everything is working great and start it up as docker-compose up -d to make it a persistent running setup.

Troubleshooting

If you get the default nginx start page, it means that your nginx container isn't reading/following any nginx config file. Figure out why. Do you accidently have a blank directory created that is called nginx.conf instead of an actual file? Did you comment out the nginx.conf bind mount?

Update the Images

In order to update the image to the latest release of lemmy, you have to manually go to your docker-compose file and edit the docker image tag to the latest version number. Then, you need to bring your container back up. Steps:

  1. Edit the docker-compose.yml file image tags from 17.3 to whatever else comes out
  2. Run a docker-compose up which will update images as needed:
docker-compose up

Watch the pretty log messages. Press Ctrl+C if everything is working great and start it up as docker-compose up -d to make it a persistent running setup.

Limitations

I don't know anything about docker. I'm a docker noob. Please correct me for anything that you think is a bad idea.

Why are the docker tags for lemmy and lemmy-ui "latest" for arm64/v8? Shouldn't there be a latest-arm and a latest-x86 or something? Annoying that I have to pin my lemmy images to a specific version in docker. I would prefer to let them be set to 1 image that gets updated and have watchtower deal with updating the image on a schedule of my choosing.

Sources

I wouldn't be here without the matrix chat, https://join-lemmy.org/docs/en/administration/install_docker.html, and this post: https://lemmy.ml/post/1127760

Reverse Proxies

There have been some suggested reverse proxy configs for Caddy and Apache!

Caddy

Thanks to @tmpod@lemmy.pt for this caddyfile:

lemmy.tld {
	header {
		# Only connect to this site via HTTPS for the two years
		Strict-Transport-Security max-age=63072000

		# Various content security headers
		Referrer-Policy same-origin
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		X-XSS-Protection "1; mode=block"
		# disable FLoC tracking
		Permissions-Policy interest-cohort=()

		# Hide Caddy
		-Server
	}

	# Enable compression for JS/CSS/HTML bundle, for improved client load times.
	# It might be nice to compress JSON, but leaving that out to protect against potential
	# compression+encryption information leak attacks like BREACH.
	@encode_mime {
		header Content-Type text/css
		header Content-Type application/javascript
		header Content-Type image/svg+xml
	}
	encode @encode_mime gzip

	request_body {
		max_size 8MB
	}

	@pictshare_regexp path_regexp pictshare_regexp \/pictshare\/(.*)
	redir @pictshare_regexp /pictrs/image/{re.pictshare_regexp.1} permanent

	# Supposedly better than having three different named matchers using standard matchers
	# ¯\_(ツ)_/¯
	@backend `
	path('/api/*', '/pictrs/*', '/feeds/*', '/nodeinfo/*', '/.well-known/*')
	|| header({'Accept': 'application/*'})
	|| method('POST')
	`
	reverse_proxy @backend lemmy:8536 {
		# This was needed because of a bug, but it probably has been fixed in the meanwhile.
		# Will have to test later.
		header_down -Transfer-Encoding
	}

	reverse_proxy lemmy-ui:1234
}

Apache

Here are a few apache configs you can draw from.

The best apache config I've seen so far is by DeadCade in the comments here.

<VirtualHost *:443>
        ServerName lemmy.deadca.de
        SSLEngine on
        ProxyRequests on
        ProxyPreserveHost on
        ProxyTimeout 600

        SetEnv proxy-nokeepalive 1
        SetEnv proxy-sendchunked 1

        <Location />
                Allow from all
                ProxyPass http://127.0.0.1:(INTERNAL LEMMY PORT)/
                ProxyPassReverse http://127.0.0.1:(INTERNAL LEMMY PORT)/
        </Location>

    ErrorLog "ERROR LOG LOCATION"
    CustomLog "ACCESS LOG LOCATION" common

    # Enable mod_rewrite (requires "a2enmod rewrite")
    RewriteEngine on

    # WebSockets support (requires "a2enmod rewrite proxy_wstunnel")
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule ^/?(.*) "ws://127.0.0.1:(INTERNAL LEMMY PORT)/$1" [P,L]

SSLCertificateFile FULLCHAIN.PEM LOCATION
SSLCertificateKeyFile PRIVKEY.PEM LOCATION
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

If you need another apache config, this was suggested by Samsonite (though, he knows that it needs cleaned up). Comment if you have suggestions for what to remove:

<VirtualHost *:80>
    ServerName mylemmydomain.com
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteCond %{HTTP_HOST} !^(localhost|internallemmyip)
    RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]


</VirtualHost>

<IfModule mod_ssl.c>
    <VirtualHost *:443>
        ServerName mylemmydomain.com
        SSLEngine on
        ProxyRequests On
        ProxyPreserveHost On
        ProxyTimeout 600

        SSLCertificateFile /etc/letsencrypt/live/mylemmydomain.com/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/mylemmydomain.com/privkey.pem
#       ProxyPreserveHost On

        # Proxy pictshare
        <Location "/pictshare">
                ProxyPass http://internallemmyip:8537/
                ProxyPassReverse http://internallemmyip:8537/
        </Location>

        # Proxy iframely
        <Location "/iframely">
                ProxyPass http://internallemmyip:8061/
                ProxyPassReverse http://internallemmyip:8061/
        </Location>


#        # Correctly proxy websocket traffic
        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteRule /(.*) ws://internallemmyip:80/$1 [P,L]
#
        # Proxy Lemmy
        <Location "/">
                ProxyPass http://internallemmyip/
                ProxyPassReverse http://internallemmyip/
        </Location>

        ErrorLog /var/log/apache2/mylemmydomain-error.log
    </VirtualHost>
</IfModule>

view more: next ›

jax

joined 11 months ago
MODERATOR OF