“State-of-the-art” (Sota) artificial intelligence models excel at solving complex Olympiad maths but still struggle with everyday enterprise tasks, according to an executive from a top AI unicorn in the US.
David Meyer, senior vice-president of product at US data processing and analysis company Databricks, told the South China Morning Post in a recent interview that the very traits making models state-of-the-art could cause issues in basic office work. For instance, when tasked with identifying an erroneous number on an invoice, a Sota model “will oftentimes fix the mistake” rather than simply extracting the error for downstream correction, he said.
The discrepancy extends to other highly technical domains as well. While advanced models such as Anthropic’s Claude were powerful at coding, they could lag in tasks like data engineering compared with models with significantly more specialised training and data in this area, according to Meyer.
Advertisement
Data engineering involves transforming datasets at scale and performing cleaning tasks, such as handling null values and zeros.
“A single model, no matter how large, can’t be equally good at all things,” he said.
Advertisement
To solve these specific complexities more efficiently, Meyer pointed to the use of small open-source models refined with reinforcement learning. This allowed for a specific purpose at a level of training cost “orders of magnitude lower” than Sota models, according to Meyer.
‘State-of-the-art’ models can struggle with basic office work, says AI executive
