No, not all violence is physical, and while systemic violence can be violent, it very often isn't. But it's any damage or harmful action that's carried out through normal operation.

Systemic violence against the poor includes economic and social violence, like tax cuts, safety regulations repealed, or social programs being shutdown. None of these forms of systemic violence are physical, but it's yet another metaphorical hit against people who can barely stand as it is. The racism baked onto the system (shit like redlining) isn't necessarily physically violent, but it hurts and kills people all the same. Hell, the UHC guy getting killed and everyone cheering is because shit like the insurance industry is systemically violent against its consumers.

The nonviolent forms of systemic violence due regularly lead to physical violence, such as houseless populations being rousted and their encampments torn down because it's easier to be cruel than suffer the eyesore and help people.