The State of AI 2024: AI Will Get Even Better
- By Paul Mah
- October 23, 2024
Now in its seventh year, the annual State of AI Report 2024 is out. Reviewed by leading AI researchers and professionals, the latest edition looks at the technology breakthroughs and capabilities of AI, as well as important considerations such as the commercial application, regulation, and risks of AI.
Proprietary models lose edge
One point that jumped out at me is how proprietary models are losing their edge, as the gap between the state-of-the-art (SOTA) AI models and the rest closes. For now, OpenAI o1 has placed OpenAI back at the top of the charts, though it remains to be seen if it will be dethroned soon, notes the report.
As noted by the report: “Meta dropped the Llama 3 family, 3.1 in July, and 3.2 in September. Llama 3.1 405B, their largest to date, is able to hold its own against GPT-4o and Claude 3.5 Sonnet across reasoning, math, multilingual, and long-context tasks. This marks the first time an open model has closed the gap with the proprietary frontier.”
With “open” models such as Meta’s Llama, the industry is also questioning whether the terms “open source” and “open” models are used misleadingly, considering the inability of outsiders to contribute back or build their own Llama from scratch.
Conclusion by the report: “[The term] has been used to lump together vastly different openness practices across weights, datasets, licensing, and access methods.”
More research required
New breakthroughs continue to happen with GenAI on a regular basis, which suggests that there is much room for progress, especially in areas such as making Models smaller without sacrificing performance.
“Research suggests that models are robust in the face of deeper layers – which are meant to handle complex, abstract, or task-specific information – being pruned intelligently. Maybe it’s possible to go even further,” noted the report.
According to it, a combined Meta/MIT team looking at open-weight pre-trained LLMs concluded that it’s possible to do away with up to half a model’s layers with only negligible performance drops on question-answering benchmarks. Optimal layers were pruned based on similarity and then patched through small amounts of efficient fine-tuning.
In the meantime, developers are also diving into defending AI models against jailbreaking. OpenAI had proposed a fix to the “ignore all previous instructions” attack via a technique known as “instruction hierarchy” to ensure LLMs don’t assign equal priority to users’ and developers’ instructions. This has been deployed in GPT-4o Mini.
Separately, safety specialists at Gray Swan AI have piloted the use of “circuit breakers” that focus on remapping harmful representations so the model either refuses to comply or produces incoherent outputs. This technique apparently outperforms standard refusal training.
Clearly, a lot more research and work remains.
What next
While many of the start-ups working on generative AI are raising record amounts, many of these currently have no identified path to profitability. However, this isn’t true for everyone, notes the report, and the biggest model providers are seeing revenue ramp up.
But while a handful of AI firms have started to generate serious revenue, doubts around long-term sustainability persist as AI models get cheaper. After all, staying at the frontier of AI is costly due to the increasingly large clusters that are required, which increases the pressure on revenue.
The report also offered some predictions. Apple’s on-device research will accelerate the momentum around personal on-device AI, it notes, and also predicts that challengers will fail to make any meaningful dent in Nvidia’s market position.
What do you think?
The full 213-page report can be accessed here.
Image credit: iStock/alice-photo
Paul Mah
Paul Mah is the editor of DSAITrends, where he report on the latest developments in data science and AI. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose.