Lipids are one of the four macromolecules found in life. Moderation of lipids is the key part here. LDL (bad cholesterol) is needed for digestion, cell signaling, and immune cells require it for certain functions. It's the goldilocks principle, too much or too little is an issue.

I had to look it up, but reducing milk by 2% fat increases the calcium ~6mg/cup. A little calcium goes a long way for human cell functions, so it's a decent increase for sure.

Turns out that saturated fat isn't the big awful thing that has been peddled for decades. What turns out to be better for you is shorter chain saturated fats produced by a health gut microbiome, and that feeding your microbiome is essential to better health. Also, not all calories are equal. It turns out items higher in resistant starch feeds a healthy microbiome and in turn those bacteria convert the food calories into short chain saturated fats which feed you. See this recent Nature Comm paper.