you're nowhere close to RAM exhaustion. I had similar mishaps on an all-AMD system a few gens back and it manifested itself as micro-stutters that occasionally grew to such manifestations. I think I remember it was a combination of kernel switches and progressively better performance as new versions of kernel and modules/drivers progressed.
no idea what KDE Neon is based on (Ubuntu LTS?), but I'm guessing you rock pretty old kernels and relatively modern hardware, which is a pain. also you don't need a swapfile, use zram. or just switch to fedora or sumsuch that takes care of all them things for you.