show index hide index
Meta’s presentation of Llama 4 was full of excitement, promising to revolutionize the world of artificial intelligence with its multimodal models. However, initial tests reveal a striking gap between the rhetoric and the actual performance. Blatant limitations are emerging, criticism is flying, and users are questioning the veracity of the presented benchmarks. The illusion is crumbling, and competition is intensifying as expectations meet harsh reality. Meta’s launch of Llama 4 promised spectacular advances in the world of artificial intelligence, but reality seems to be clashing with expectations. While multimodal models such as Scout and Maverick were making waves across the technology world, recent analyses and tests are raising questions about their true effectiveness. Contrary to the performance promises, the actual results reveal notable technical limitations and puzzling inconsistencies. A Multimodal Ambition When it was launched, Llama 4was touted as capable of revolutionizing the way we interact with machines. With their ambitions for multimodality, Scout and Maverick sought to establish a performance standard unprecedented in the market. The Llama 4 Behemoth , with its2 trillion parameters , was supposed to rival giants such as GPT-40 and Gemini 2.5. However, the first benchmarks raise suspicions about these lofty claims. This raises the question: do they really live up to expectations? Performance That Defys Reality One of Scout’s announced strengths was its context window of 10 million tokens . However, tests are accumulating to reveal a completely different side. For example, running a context of 1.4 million tokens requires no fewer than eight Nvidia H100 GPUs, a configuration few users can afford. Meanwhile, services like Groq cap at 128,000 tokens, while Together AI only provides 328,000. This gap between advertised and actual usage intensifies skepticism and frustration among developers and users. Relentless criticism
Criticism of Llama 4 is pouring in, especially on social media where users share their often disappointing experiences. Scout’s test results on advanced tasks, such as summarizing
20,000 tokens
, show alarming inconsistency. Voices like Andriy Burkov’s speak out against monolithic models, proposing reasoning based on reinforcement learning. Simultaneously, users on Reddit point to weaknesses in Llama 4’s coding compared to competitors like DeepSeek or Qwen. This gap between predictions and reality tarnishes Llama 4’s initial image. Relative Openness and BenchmarksAlthough Meta describes Llama 4 as open source, licensing restrictions cast doubt on this claim. It might be more accurate to use the term « open weight » to characterize this accessibility. Moreover, comparative studies show that Mavericks’ performance sometimes exceeds that of GPT-4o, even ranking second on Chatbot Arena with an ELO score of 1417. However, this does not mean that the results are representative of everyday use, as distinctions appear between variants optimized for testing and publicly available models. A significant lack of transparency thus emerges, raising questions about Llama 4’s true performance.
Contested Explanations
In an attempt to be transparent, Meta’s Ahmad Al-Dahle attributes the observed performance gaps to technical instabilities, denying any deliberate manipulation of the results. This type of argument is reminiscent of the controversies surrounding biased smartphone benchmarks. In any case, Al-Dahle defends the progress made by Llama 4, while admitting that there are still bugs to be fixed. This raises a crucial question: is the community ready to trust Meta to resolve these issues and deliver a valid product?
