The short answer is yes, you should worry about them. Are they ok in moderation? Sure. Are they ok when they are the main staple of your diet? Absolutely not.

For those that don't know what the term means:

Ultra-processed foods are made mostly from substances extracted from foods, such as fats, starches, added sugars, and hydrogenated fats. They may also contain additives like artificial colors and flavors or stabilizers. Examples of these foods are frozen meals, soft drinks, hot dogs and cold cuts, fast food, packaged cookies, cakes, and salty snacks. - source

Ultra-processed foods don't really contain any actual food. They are derived from food, but they have basically been stripped of all of their naturally occurring nutrients.

There are quite a few studies out there that show that while eating an ultra-processed diet people tend to eat a fair amount more calories (I've seen multiple places say 500 more calories) per day than when eating whole foods, or minimally processed. They also tend to gain weight (over a surprisingly short period of time), have higher incident of cardiovascular disease, had higher increase in fat and carb consumption, but not protein, and high incident of some cancers.

I'd also add that most people probably feel A LOT worse when eating ultra-processed foods. Just observation on my part, but people who eat terribly seem to lack energy, and seem to struggle more with things like sleeping, exercise, etc.