Technology
Which posts fit here?
Anything that is at least tangentially connected to the technology, social media platforms, informational technologies and tech policy.
Post guidelines
[Opinion] prefix
Opinion (op-ed) articles must use [Opinion] prefix before the title.
Rules
1. English only
Title and associated content has to be in English.
2. Use original link
Post URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
3. Respectful communication
All communication has to be respectful of differing opinions, viewpoints, and experiences.
4. Inclusivity
Everyone is welcome here regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
5. Ad hominem attacks
Any kind of personal attacks are expressly forbidden. If you can't argue your position without attacking a person's character, you already lost the argument.
6. Off-topic tangents
Stay on topic. Keep it relevant.
7. Instance rules may apply
If something is not covered by community rules, but are against lemmy.zip instance rules, they will be enforced.
Companion communities
!globalnews@lemmy.zip
!interestingshare@lemmy.zip
Icon attribution | Banner attribution
If someone is interested in moderating this community, message @brikox@lemmy.zip.
view the rest of the comments
Any favorites? What do you think about state space models?
Jamba (hybrid transformers/space state) is a killer model folks are sleeping on. It's actually coherent at long context, fast, has good world knowledge, even/grounded, and is good at RAG. Its like a straight up better Cohere model IMO, and a no brainer to try for many long context calls.
TBH I didn't try Falcon H1 much when it seemed to break at long context for me. I think most folks (at least publicly) are sleeping on hybrid SSMs because support in llama.cpp is janky at best, hence they're not getting any word-of-mouth. For instance, context caching does not work. And Jamba's janky commercial licensing (unless you pay them) does not help.
...Not sure about others, toy models aside. There really aren't too many to try.
...TBH, Deepseek is the only non-bog-standard transformers grouped-query-attention model folks have mostly played with. The big trainers seem to be risk-averse architecture wise (hence no big bitnet model attempts yet), which is what Nvidia is betting on I guess.
ty, appreciate this