technology

Interesting blog post by a mathematician about the potential for LLMs to do math research. He goes beyond simple reports of "ChatGPT solved X problem" to try to understand what LLMs are doing and whether they're consistent enough to be useful. He concludes that what the models do well is access and generalize from existing knowledge that a single researcher might not be aware of, but a lot of human labor is still involved weeding through the garbage to find the gems, because models are not capable of consistent self-critique. He also notes that they're unable to invent new concepts or terms, just apply existing ones.

He concludes with speculation that eventually models will be able to solve a lot of problems autonomously, but they won't be able to determine which ones are most interesting to humans, and that that curiosity is both a strength (because humans can determine what to do with new knowledge) and a limitation (because focus is a limited resource).