If he's down to needing "only" 10-20 holdouts, he's actually gaining significant ground. This should worry you.
The moderates of the party are simply growing tired of all of this and are more likely to vote for him simply because there are no other viable options. The problems with this are: Jim Jordan becomes speaker of the House. The MAGA wing gets rewarded for causing this mess by having one of their own elected speaker once the dust settles, emboldening them to do it again in the future. And it also reinforces how effective the strategy of forcing people out of office by making the position untenable for anyone who actually wants to do the work, leaving only the crazies to fill the void is.
My only real hope is that Jim Jordan is incompetent enough and so hyper-focused on partisan grievance politics that he won't cause too much damage. That, and he only has just over a year in office. The bad thing is that if you think these people are crazy, remember that they're simply paving the way for those who will come after them, many of which are even crazier than they are.