Effortpost incoming!
This is literally my exact area of specialty (foundations of model building in weather and climate science), so I have a bunch of thoughts. There are two assumptions here: first, that all we want out of our models is predictive accuracy, and second that we should be using only the model that generates the best predictions. Neither is true.
Imagine an alien came down from space and handed you a kind of souped up Magic 8 Ball. The alien 8 Ball works just like a regular one--you ask it a question, flip it over, and it gives you an answer in a little window--except it is really accurate. You can ask it "where will this hurricane be in 3 days?" and it'll tell you, with great accuracy, every time. The same is true for basically any other physical prediction you want to make.
Would this be a good reason to shut down every scientific investigation program in the world? Would the 8 Ball just have "solved" science for us? Pretty clearly not, I think. It might be a really useful tool, but science isn't just a machine that you feed questions into and read predictions off of. Part of why we do science is to explain things and identify general patterns in how the world changes over time. Among other things, that helps us refine the questions we're asking, as well as discover new ones that we hadn't thought to ask.
There'd be a natural question, I think, of why something like the 8 Ball worked the way that it did. We'd want to know what it was about the internal structure of the ball and its relationship to the structure of the world that made it a good model. That's because part of what we use models for is structural investigation, not just point-forecasting. I want to be able to look at the structure of the model and learn something about the structure of the target system I'm studying--in fact, a lot of physical insights into the structure and function of the global climate have come about this way rather than from (as I think most people would expect) studying the climate system directly. Having a physics-based model facilitates that, because we can see how the structures and outputs of the model fit in with other patterns in the physical world with which we are more familiar. In addition, understanding the way that the model behavior emerges from low-level physical principles helps us have confidence in the stable projectability of the model's predictions. We can be a lot more confident, that is, that our model will remain reliable even across very different conditions because the physical principles are stable.
I don't have the same kind of confidence in a machine learning model because it is merely a pattern recognition engine--it's an 8 Ball. In fact, this is exactly the methodology we used for weather forecasting before meteorology emerged as a mature model-based science. The most popular method for forecasting the weather during the first part of the 20th century involved the use of purely qualitative maps of past weather activity. Forecasters would chart the current state to the best of their ability, noting the location of clouds, the magnitude and direction of prevailing winds, the presence of precipitation, etc. Once the current state was recorded on a map of the region of interest, the forecasters would refer back to past charts of the same region until they found one that closely resembled the chart they had just generated. They would then check to see how that past state had evolved over time, and would base their forecast of the current situation on that past record. This turned forecasting into the kind of activity that took years (or even decades) to become proficient in; in order to make practical use of this kind of approach, would-be forecasters had to have an encyclopedic knowledge of past charts, as well as the ability to make educated guesses at how the current system might diverge from the most similar past cases. This approach faded away as people got a better theoretical understanding of atmospheric physics and other relevant theoretical processes underwriting the weather and climate systems. Eventually, it was replaced entirely by computational modeling that's grounded in stuff like fluid dynamics, thermodynamics, and other relatively well-understood physical processes.
It seems natural (and correct) to say that something was added to meteorology when people started making predictions based on things like atmospheric circulation models and other well-articulated theories grounded in formal models, rather than just looking at past weather maps. It also seems like whatever it was that was added in that transition is at least partially independent of predictive success: even if the weather map method was about as good at predicting tomorrow's weather as the computational modeling method, the latter seems more like a mature science in virtue of explaining why tomorrow's weather prediction is what it is. In both this case and the 8-Ball case, the thing that seems missing is explanation.
Using machine learning (and only machine learning) is a step back to this way of doing forecasting. It would effectively be just relying on the old "charting" approach to weather forecasting, just done by a system that is very, very good at it--better than any human. An 8 Ball. It cuts off a whole branch of explanatory (and eventually predictively relevant) investigation, and may well fail catastrophically at even prediction when confronted with conditions it hasn't seen before. It's hugely shortsighted, and represents a fundamental misunderstanding of the role of models in science.