What is the incentive for instance owners to push people toward the local feed vs the all feed?

I wasn't aware small instances were hiding the all feed, but honestly I'm not opposed. If people want to run Lemmy instances the same way forums used to dot the internet, that'd be cool. Users having the ability to chime in with their kbin account or whatever is just a nice perk for ease of use.

No we don't.

One of the fundamental strengths of a federated approach is that I can, with no muss or fuss, see only my local instance.

I understand where you're coming from, but you're presuming that most instances are simply clones of each other, and that's not true. (or at least that's not how it's designed to be).

For example, my home instance is mstdn.ca. Ostensibly, with a single button, I can focus my entire Lemmy experience on things important to me as a Canadian without wasting through everyone else's bullshit. Canadian news. Canadian politics. Canadian hobbies, etc...

You're idea presumes that all instances are trying to carry the exact same content, which is false.

The All-Feed is very important to increase the content-quality of the threadiverse

How so? For who, and why? I doubt it's true for me. I use my subscribed communities. How does the existence, or the state of existence of the All-Feed affect my content quality?

However, if [small instances] push people toward their the Local-Feed (which they currently do), people will only stay on their instance

What do you mean with "push people towards their local feed"? Are you referring to the instance default setting, wether the instance page should be fed from All, Subscribed or Local? And you mean the default should be All, not Local?

Also doubts about the last part; "people will only stay on their instance". I joined my instance when it had around 100 users, and instantly started searching for communities which interest me, and subscribed to them, regardless on which instance they are hosted.

Maybe the kind of user you are talking about exists. Maybe some people never search for their interests, never get interested in a cross-post community, never get interested in a community when other people talk about it. I have doubts about all this, but maybe it exists. But it's on you who makes that statement to show this user group exists in a number large enough to talk about it as if there was no one else.

Maybe some people will only stay on their instance, but if some is small enough, and a good part of those enjoy exactly that, why should we care or even try to change that?

I do agree however, that community discovery should be improved even further. Especially for people from very small communities, it can be a tricky process to discover new communities on remote instances.

we need a new feed, in which an instance can find a good balance between showing content from other instances and its own.

I'm all for more control, which is why I'd like that power on the user level. Let me choose how to curate and compose my feed. An instance may predefine defaults, yes. Much like instances predefine wether their start page is populated with All, Subscribed or Local, and users can freely choose otherwise or change this setting permanently for them individually.

Ideally, I'd like to assign weights to instances and communities, to control how likely a post from that origin is to show up in my feed.

Honestly, the problem with discovery is not that there are not enough posts in a single timeline. Merging local and global feeds makes discoverability worse on Lemmy and kbin, not better, because the timelines display posts, while the space is organized by communities. This means that smaller or niche communities just drown seas of posts from large or highly active ones.

If you want a real "exploration" timeline, you need one that limits the number of posts from any given community. And that still seems like it's well served by local/global splits, because the website you join should be meaningful.

We do not need, nor should we want, a network of "dumb terminal" Fediverse sites. We should be aiming for the local stream to be the big selling point for any given instance, with the ability to interact with remote communities being a value-add. A merged timeline kills local identity, and tells users that their hosting website is a 2nd class citizen in the Fediverse.