I kind of like the idea of "this person can be a liability if you let them talk to people". D:OS 2 did something like that where party members would ask to take over the conversation and do whatever they wanted with it, and I thought it definitely added to the immersion. I definitely don't think allowing you to magically control who's party to a conversation is an improvement over letting it happen organically.

Even ignoring the story element, from a pure gameplay perspective, I don't think getting a bad speaker trapped into a conversation is really different from having a glass cannon type character get caught out and killed because they can't take damage. It's part of your party construction.