I’m not much into poly-amorous (nearing my 60s, I've been sharing my life with my spouse for almost 30 years now and if neither of us are saints we’ve also never felt any need to ‘enrich’ our couple with more participants) but if this is how you feel maybe that is not for you? I mean that specific partner(s), or the poly-amorous thing in general?
I obviously don’t know what's the issue, how could I, but what I can tell you without any doubt that if you feel like shit, which it seems you do, it’s a safe warning sign that there is an issue. And that is up to you to see what’s causing that. And to decide how to correct it.
No matter how you decide to work on it, alone or with your partner(s) you better have a clear understanding of what the issue is beforehand and be ready to face not real... not pleasant things.
Allow me to insist one last time: I' don't know poly-amorous but I do know that no relationship, no matter its form, should make anyone involved belittled or a mere 'extra'. I don't even need to know you personally to be sure of that :)