this post was submitted on 09 Sep 2025
10 points (91.7% liked)

Technology

4133 readers
339 users here now

Which posts fit here?

Anything that is at least tangentially connected to the technology, social media platforms, informational technologies and tech policy.

Post guidelines

[Opinion] prefixOpinion (op-ed) articles must use [Opinion] prefix before the title.

Rules

1. English onlyTitle and associated content has to be in English.
2. Use original linkPost URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
3. Respectful communicationAll communication has to be respectful of differing opinions, viewpoints, and experiences.
4. InclusivityEveryone is welcome here regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
5. Ad hominem attacksAny kind of personal attacks are expressly forbidden. If you can't argue your position without attacking a person's character, you already lost the argument.
6. Off-topic tangentsStay on topic. Keep it relevant.
7. Instance rules may applyIf something is not covered by community rules, but are against lemmy.zip instance rules, they will be enforced.

Companion communities

!globalnews@lemmy.zip
!interestingshare@lemmy.zip

Icon attribution | Banner attribution

If someone is interested in moderating this community, message @brikox@lemmy.zip.

founded 2 years ago
MODERATORS
BrikoX@lemmy.zip
10
Nvidia unveils new GPU designed for long-context inference (techcrunch.com)
submitted 3 days ago by misk@piefed.social to c/technology@lemmy.zip
5 comments fedilink hide all child comments
top 5 comments
sorted by: hot top controversial new old
[–] brucethemoose@lemmy.world 4 points 3 days ago* (last edited 3 days ago) (1 children)

Doubling down on flash attention (my interpretation of this) is quite risky, as there are more efficient attention mechanisms seeping into bigger and bigger models.

Deepseek’s MLA is a start. Jamba is already doing hybrid GQA/Mamba attention, and a Qwen3 update is rumored to be using something exotic as well. And Google’s been doing it with Gemini for some time, though we can only guess what since it’s closed source.

In English, this seems like they’re selling the idea of the software architecture not changing much, when that doesn’t seem to be the case.

[–] nymnympseudonym@piefed.social 3 points 3 days ago (1 children)

Any favorites? What do you think about state space models?

[–] brucethemoose@lemmy.world 1 points 3 days ago* (last edited 3 days ago) (1 children)

Jamba (hybrid transformers/space state) is a killer model folks are sleeping on. It's actually coherent at long context, fast, has good world knowledge, even/grounded, and is good at RAG. Its like a straight up better Cohere model IMO, and a no brainer to try for many long context calls.

TBH I didn't try Falcon H1 much when it seemed to break at long context for me. I think most folks (at least publicly) are sleeping on hybrid SSMs because support in llama.cpp is janky at best, hence they're not getting any word-of-mouth. For instance, context caching does not work. And Jamba's janky commercial licensing (unless you pay them) does not help.

...Not sure about others, toy models aside. There really aren't too many to try.

...TBH, Deepseek is the only non-bog-standard transformers grouped-query-attention model folks have mostly played with. The big trainers seem to be risk-averse architecture wise (hence no big bitnet model attempts yet), which is what Nvidia is betting on I guess.

[–] nymnympseudonym@piefed.social 1 points 2 days ago

ty, appreciate this

[–] roofuskit@lemmy.world 1 points 3 days ago

It'll all come crashing down soon enough.