Is this the place to report server issues or is somewhere else better?

I just now tried to post a comment on transjoy, it gave an error message. I retried and it worked. However, the comment posted twice. I can only assume the first error message was given in error.

I assume the same thing happened to someone else this morning. A double post on transjoy with an apperant error on the first attempt.

I'm having a similar issue when loading comment threads, error first attempt, works the second time. I think they might be related. Tested when logged in on jerboa and alexandrite. I get the following errors upon loading a comment thread:

Lemmy Error: couldn't_mark_post_as_read (status 400)

Lemmy Error: Couldn't find post (status 400) 

Then it works on the second attempt. I've not been able to reproduce the issue when logged out.

First attempt at posting: "couldn't_like_post"

Edit: second time success

Edit 2: no errors during edits

Edit 3: first attempt did post despite error message. I deleted the initial post and then got an error posting this edit.

this has been somewhat confusing to me, especially with federation. ive tried to search for info online but havent found anything that goes into detail, and im very curious about how it works. like i know when they get banned i never see them post anything ever again, but what do they see? are they still posting comments on their instances version of the community and we cant see it, or is the ban enforced across instances through federation? do they even see posts from the community theyre banned from at all?

Is this the place to report server issues or is somewhere else better?

I just now tried to post a comment on transjoy, it gave an error message. I retried and it worked. However, the comment posted twice. I can only assume the first error message was given in error.

I assume the same thing happened to someone else this morning. A double post on transjoy with an apperant error on the first attempt.

I'm having a similar issue when loading comment threads, error first attempt, works the second time. I think they might be related. Tested when logged in on jerboa and alexandrite. I get the following errors upon loading a comment thread:

Lemmy Error: couldn't_mark_post_as_read (status 400)

Lemmy Error: Couldn't find post (status 400)

Then it works on the second attempt. I've not been able to reproduce the issue when logged out.

cross-posted from: https://lemmy.ml/post/50346212

What is Lemmy?

Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to topics that you are interested in, and ignore others. Voting is used to bring the most interesting items to the top.

Changes

This release significantly reduces memory usage for the Lemmy backend. Metrics on production instances show a reduction up to 10 times. Here you can see the statistics from a few different instances. Read on for a technical explanation below. Note that the change only affects x86, there is no difference on ARM (e.g. Raspberry Pi), see here for details.

lemmy.ml (overall RAM usage including OS, Docker, PostgreSQL etc):

leminal.space (backend container only)

lemmy.world (backend API containers)

lemmy.world (backend federation and scheduled tasks)

How was such a major improvement possible? Memory usage in languages like Rust or C is managed by a so-called memory allocator. It requests large chunks of RAM from the operating system, and provides smaller chunks when needed in the program. For example, every string (such as post titles or markdown text) require chunks of memory to store them. When processing the data is completed, the memory should be freed and released, or reused.

Until version 0.19.19 Lemmy used the mimalloc memory allocator, which is supposedly better than the default glibc allocator. Yet a recent blog post points out that mimalloc does not play well with the tokio async runtime. It suggests to use jemalloc instead. Changing memory allocators is very simple in Rust, it only requires a single line of code. So we tried that, deployed the change on lemmy.ml and it immediately proved effective.

Upgrade instructions

There are no breaking changes with this release.

Follow the upgrade instructions for ansible or docker.

If you need help with the upgrade, you can ask in our support forum or on the Matrix Chat.

Thanks to everyone

We'd like to thank our many contributors and users of Lemmy for coding, translating, testing, and helping find and fix bugs. We're glad many people find it useful and enjoyable enough to contribute.

Support development

We (@dessalines and @nutomic) have been working full-time on Lemmy for over five years. This is largely thanks to support from NLnet foundation, as well as donations from individual users.

If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. A recurring donation is the best way to ensure that open-source software like Lemmy can stay independent and alive, and helps us grow our little developer co-op to support more full-time developers.

• Liberapay (preferred option)
• Open Collective
• Patreon
• Cryptocurrency

In light of the dev silently blacklisting blahaj.zone, many of its users and several other instances, we have removed tesseract from the available lemmy front ends we offer.

So I'm on my account and have no issues whilst logged in, but when I want to log in on desktop, I can't change my password in case I forgot it. I use Voyager on mobile, which doesn't let me change my password either ( @aeharding@vger.social ?).

I can't add an email address either, since on Voyager that doesn't seem possible. What do I do?

I was browsing 196 recently and I noticed that stuff between slrpnk and blahaj weren't federating between the 2 properly.

From slrpnk (another lemmy instance) there looks like there hasn't been a post in 196 since yesterday but checking from blahaj it's as alive as always.

I then checked from lemmy.world, beehaw, sopuli.xyz, and lemmy.zip to see if it was just an issue with slrpnk but it appears to be widespread as all of those other instances have the same issue.

Hopefully me making a post here isn't an issue (and that it federates properly) but I figured reporting a bug wouldn't be an issue.

Sorry if this isn't allowed/in the wrong place, but I wanted it to have a little visibility in case others found themselves in a position to chip in. Recovering from a breach is painful and soaks up resources that could be used elsewhere.

I mostly lurk and read Blahaj from a different instance, but when I saw the hacked announcement it reminded me what a great resource and community is here and I figured now would be a good time as any to let the Admins know I appreciate their efforts!

Link to their kofi: https://ko-fi.com/blahajzone (but to be security conscious, you shouldn't click that, you should really go get the link from their sidebar :) ).

cross-posted from: https://lemmy.ca/post/66634564

Cloudflare: Why I left lemmy.world to join another instance

Sad to see. We should all know cloudlfare decrypts our data before it reaches the service it protects. A perfect surveillance man-in-the-middle.

I was happy to find a home in the fediverse that uses Anubis instead.

https://anubis.techaro.lol/

This version again includes a couple of security fixes. Thanks to the people who found and reported them!

The first one in particular requires manual action from instance admins. Lemmy’s default Nginx config uses $proxy_add_x_forwarded_for to set the X-Forwarded-For header, which does not override existing values. So clients can spoof the IP and bypass rate limits. The solution is to use $remote_addr instead. If you use Ansible this will be changed automatically during the upgrade, otherwise you will need to do it manually. If you dont use Nginx, ensure that any X-Forwarded-For headers sent by the client are overwritten.

The remaining security vulnerabilities are in the Lemmy code itself, and will be fixed simply by upgrading.

Security:

• Rate limit bypass via X-Forwarded-For header spoofing in actix-web ConnectionInfo
• Login Endpoint User Enumeration via HTTP Response Code and Timing
• Blocked users can edit private messages sent before the block
• Lower-ranked federated moderator can remove higher-ranked moderators via federation
• Featuring post over federation does not validate community nor write modlog
• Stored XSS via markdown image alt-text in lemmy-ui html5-embed##

Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.

The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn't use postgres). From what we can tell, the attacker did not do anything with that access, so we don't believe any user data was accessed, but we can't be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.

Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.

We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it's as smooth as we think it will be, but we can't be sure!

As stated earlier, our writefreely instance is still up and running as it wasn't impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.

We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don't plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won't be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn't seem worth the cost/effort to run it isolated.

in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked

Edit - Piefed is back now!

pressing "save" in my settings page does nothing, not even show a popup telling me something failed. it's been that way for a few months at least, since april of this year i think.

i've tried modifying and saving my profile in other clients, and it seems to work there! i can change my bio and username without issues.

however, something i can't do, in any client, is change my profile picture, which is unfortunate since mine has been broken* for some time. in the default lemmy ui, as i said, pressing the save button does nothing. however, i've also tried to change my profile picture in tesseract and blorp, and both complain about an invalid url when trying to save the profile pic. both can save other settings tho, and image upload works fine, as the images i try to set appear in my uploads tab. using the js console, i can see that changing the profile pic does a PUT https://lemmy.blahaj.zone/api/v3/user/save_user_settings request, which fails (error 400). trying to change the banner image has the same problem. this happens no matter the browser or device i try this on.

is this also a problem for other people?

* (i've since removed my profile pic in blorp, which is why i've got the default lemmy profile pic now instead of a broken image icon)

(i'm sorry if this kind of post is not allowed, if not i'll delete no worries 😅)

Anybody else experiencing this? Started about an hour ago.

I'm trans and I like programming. I'm interested in discussing topics like homebrew games and web1 sites. is there a community for that?

We've just spun up a brand new photo sharing instance, based on Vernissage. It's similar to pixelfed, but more focused on photography (as opposed to general image and short video sharing).

Signups are open. Currently, it mandates an email address, but you can stick whatever you like in there, and it will be automatically verified. Signups do require approval, but you can also generate invites that (I believe) bypass the requirement for approval.

Anyway, you can check it out at https://photos.blahaj.zone/

As a consequence, we will be taking down our pixelfed instance. We'll leave it up until the end of August to give people a chance to migrate, but honestly, it had really low activity in any case, so it shouldn't have too much of an impact on people.

Vernissage may also end up having low activity, but even if it does, I like it so much more than pixelfed, that we'll keep it running even if it's just me using it :)

Edit - Piefed has been updated and returned to service

~~Thanks to someone posting a piefed security exploit without a disclosure period, we've had to pull blahaj piefed down without warning until a fix is implemented. This could take around 24 hours or so.~~

Edit - We're back!

~~We've had an issue with our databases. One of our fast database servers ran out of space, and then the second fast server ran out of space whilst replicating to the first.~~

~~As a result, we have fallen over to our backup database server, which runs on spinny disks rather than SSDs. Spinny disks means that it's got plenty of space to spare, but it's not fast. The backup DB server is currently replicating to our two main servers to get things back up and running again, but whilst that's happening, all of our services are running slow.~~

~~The good news is, we'll be back up and running as if nothing happened because our backup server saved the day. The bad news is, it may take another 24 hours or so, because the backup server is reliable but not fast!~~

According to https://lestat.org/, the Blåhaj instance is down notably more often. What is happening or causing this?

Is it server costs? If so, I'm willing to contribute, but how should I do so? And if not, what would it be? Host issues? Targeted attacks, perhaps?

Over the last several days, I've noticed significant delays with votes, comments, and notifications from that instance. The longest delay I've noticed was 2 days. Is this a known issue? Is anyone else experiencing this?

Context

TL;DR This is a pledge to defederate from lemmy.world if they end up defederating from any of the instances in the Fediverse Anarchist Flotilla (FAF), which consists of https://lemmy.dbzer0.com/, https://anarchist.nexus/, and https://quokk.au/. Just want to see if the admins are aware of this situation, and see what others think.

With this version user badges are always shown next to usernames. There are also various bug fixes, and again security fixes

• Display UserBadges for Bot, Banned and Deleted users in all PersonListings by @MrKaplan-lw in #4035
• Increase timeouts for db pool by @nutomic in #6441
• Add private IP check for webmention by @nutomic in #6444
• Proper fix for nested comment fetch by @nutomic in #6451

Security

• Lemmy allows an authenticated low-privileged user to create a link post through POST /api/v3/post. When a post is created in a public community, the backend asynchronously sends a Webmention to the attacker-controlled link target. The submitted URL is checked for syntax and scheme, but the audited code path does not reject loopback, private, or link-local destinations before the Webmention request is issued. This lets a normal user trigger server-side HTTP requests toward internal services. https://github.com/LemmyNet/lemmy/security/advisories/GHSA-3jvj-v6w2-h948
• Lemmy fetches metadata for user-supplied post URLs and, under the default StoreLinkPreviews image mode, downloads the preview image through local pict-rs. While the top-level page URL is checked against internal IP ranges, the extracted og:image URL is not subject to the same restriction. As a result, an authenticated low-privileged user can submit an attacker-controlled public page whose Open Graph image points to an internal image endpoint. Lemmy will fetch that internal image server-side and store a local thumbnail that can then be served back to users. https://github.com/LemmyNet/lemmy/security/advisories/GHSA-h6hf-9846-xwrq

We've seen a lot of spam come from lemmy.org and we have therefore defederated from their instance. For the moment they have open registrations which only require filling out a simple captcha to create an account which many people abuse.

The admin is active but rarely takes moderation actions, but I've sent a DM to them with our concerns. We'll see how permanent this is.

try with any lemmy.blahaj.zone post, only lem.lemmy.blahaj.zone embeds.

I made an advice community for Swedish-speakers, and I happened to find one for Russian speakers. I'd mod, but I don't speak Russian.

It's called !advice_russian@lemmy.blahaj.zone

The moderator appears to have been banned