Controlnet isn't really that suitable, it's better at forcing poses and composition from a reference while with faces you want to force a style from reference into any composition. The newest thing is an IPAdapter (on steroids, which is why it's taking ages (in SD terms) to implement into Comfy/A1111, it's been two weeks and it's still not done...). Reportedly doesn't use any additional VRAM so you should be fine with 4G (if using ComfyUI). Meanwhile go here which uses this which is actually ancient (2018), face detection isn't exactly rocket science in the AI world.

None of those one-image reference things will ever be able to compete with actually training the unet and producing a LoRA but OTOH that's heavy iron territory. Hit generate a couple more times and choose your reference picture well you'll be fine.

The kinda disappointing thing is that the approach is rather specialised: It does faces, and faces only, and only does faces well because it only does faces: It contains a layer specialised on facial geometry so you'll have no luck trying to, say, copy a specific belt buckle design over. Having a lightweight way to do that kind of thing in a completely generic manner would be a true game-changer.

Cool, I was wondering about that