There just needs to be due process to make it harder to abuse.

I can see where you might think that. I previously thought the same. I don't think so anymore.

Because an insurrection against the government is fundamentally a criminal act, one would naturally think that Section 3 of the 14th Amendment would require a criminal conviction to apply. For most crimes you have to be convicted before there is a penalty. This is basic due process (which, BTW, is described in Section 1 of the 14th Amendment), however I don't think that is the case here.

Section 3 of the 14th Amendment reads as follows:

Section 3

No person shall be a Senator or Representative in Congress, or elector of President and Vice-President, or hold any office, civil or military, under the United States, or under any State, who, having previously taken an oath, as a member of Congress, or as an officer of the United States, or as a member of any State legislature, or as an executive or judicial officer of any State, to support the Constitution of the United States, shall have engaged in insurrection or rebellion against the same, or given aid or comfort to the enemies thereof. But Congress may by a vote of two-thirds of each House, remove such disability. (emphasis mine)

Section 3 describes how participating in an insurrection will result in disqualification to hold office. It has no requirement for a criminal conviction. If a State believes that a person who previously took an oath to support the US Constitution engaged in an insurrection, then that person is disqualified from running for office. There is no due process requirement for disqualification due to age or citizenship, so there is no need for due process for any other disqualification.

If Congress feels that the person should be allowed to run for office, they can remove the qualification disability with a two-thirds vote in both houses.

The States can decide for themselves if Trump is disqualified, if Congress disagrees they alone have the ability to fix it.

I thoroughly agree with you that the most likely argument that comes from the Supreme Court in regards to Trump's eligibility to serve will hinge on the application of due process, and what that means in the case of insurrection. As you said, there was not a formal criminal tribunal or insurrection charges filed during the post-war reconstruction. It seems to have just been defacto applied via the actual ratification of the 14th Ammendment, and was understood to unanimously bar confederate representatives from serving in government.

I think all of the historical context as well as the spirit of the law does need to be closely examined. However, my fear is that no matter what the outcome of that examination may be it will only serve to further erode public trust in the judicial system due to the (valid) perception of political corruption within the Supreme Court, as well as their continuous inaction, which serves to telegraph a lack of concern or desire to avoid the prudential and consistent application of the law.