Probably my favorite Scotty moment.
And yes, he absolutely was bluffing. Montgomery Scott would not kill innocent civilians, even under direct orders. Recall that in this same episode Scotty refused a direct order to lower the shields and was fully willing to accept the consequences. (See also Star Trek 2009 where Scotty resigns rather than violate his professional ethics.)
I also like how Uhura reacts. She heard Kirk give General Order 24 but didn't connect that with destroying the civilization below until Scotty utters his threat. This reinforces my headcanon that General Order 24 is actually a public health ordinance that was never intended to be used as a threat.
I disagree with this characterization. Scotty's line is, "We can't fire full phasers with our screens up, and We can't lower our screens with their disruptors on us. Of course I could treat them to a few dozen photon torpedoes." From the tone and delivery I took it as an expression of frustration rather than a serious idea he had, let alone one he had already decided on.
Which is a really weird thing for Starfleet to have a protocol for. It makes more sense if we assume that GO24 is actually some kind of quarantine/sterilization procedure intended to prevent the spread of dangerous life forms.
If he was bluffing in the first place then he would still be bluffing here. There would be no point in tipping his hand until the situation was under control, which at that point it was not.