Honestly I would get away from ollama. I don't like it for a number of reasons, including:

Suboptimal quants

suboptimal settings

limited model selection (as opposed to just browsing huggingface)

Sometimes suboptimal performance compared to kobold.cpp, especially if you are quantizing cache, double especially if you are not on a Mac

Frankly a lot of attention squatting/riding off llama.cpp''s development without contributing a ton back.

Rumblings of a closed source backend

I could go on and on, inclding some behavior I just didn't like from the devs, but I think I'll stop, as its really not that bad.

Oh, and as for benchmarks, check the huggingface open llm leaderbard. The new one.

But take it with a LARGE grain of salt. Some models game their scores in different ways.

There are more niche benchmarks floating around, such as RULER for long context performance. Amazon ran a good array of models to test their mistral finetune: https://huggingface.co/aws-prototyping/MegaBeam-Mistral-7B-512k

The problem is that splitting models up over a network, even over LAN, is not super efficient. The entire weights need to be run through for every half word.

And the other problem is that petals just can't keep up with the crazy dev pace of the LLM community. Honestly they should dump it and fork or contribute to llama.cpp or exllama, as TBH no one wants to split up LLAMA 2 (or even llama 3) 70B, and be a generation or two behind for a base instruct model instead of a finetune.

Even the horde has very few hosts relative to users, even though hosting a small model on a 6GB GPU would get you lots of karma.

The diffusion community is very different, as the output is one image and even the largest open models are much smaller. Lora usage is also standardized there, while it is not on LLM land.