I keep seeing these Grok 4 intelligence claims, so I tried something very simple: "Animate a round robin tournament for 10 people."
Results:
Claude: ~10s, perfect working demo
ChatGPT: ~20s, solid solution
Grok 4: ~1000s, failed completely, gave me a truncated base64 blob
This wasn't some obscure edge case... it was basic data visualization that any decent model should handle. Yet somehow Grok 4 is "competing with humans" and has "99% tool accuracy"...
Grok 4 has about 99% accuracy in picking the right tools and making tool calls with proper arguments almost every single time.
Where did this number come from? What is "the right tool"? I find this extremely subjective. As most engineers know, there is no right tool, but mostly a compromise where you pick the least worst tool and choose what risks you're willing to manage or not.
I believe in this context it means "tool" as in the MCP definition, e.g. "of the catalog of MCP integrations, it doesn't try to use the playright one to browse the web, it'd use the AWS docs one directly"
This is just my speculation, though, as I've never used Grok anything
That's langchain terminology. LLMs usually are exposed to a set of tools. It's usually pretty obvious which are obvious, since there's only one tool that's even remotely associated with the task at hand.
On your intelligence graph where it shows Grok 4 and OpenAI o4-mini as comparable (and among the highest intelligence rated models), it doesn’t have OpenAI o3 or o3-pro.
Yet all of my tests show o3 blows o4-mini out of the water.
I can't take anything seriously with phrases like "it has not yet achieved AGI, but it is one leap forward in the race to AGI" - based on what? Nobody knows whether LLMs are a viable approach to AGI, nobody really agrees on what AGI is, hell, people don't really agree on what "I" is.
This is just not even science at all at this point, we're just into solid cargo cult.
> Grok 4 is [...] the most intelligent model so far
A bit too much praise for a model that's barely ahead of the competition in a subset of benchmarks...
> To be honest, this model not only competes with other AI models but also with humans, making it the first of its kind
I'm out
I keep seeing these Grok 4 intelligence claims, so I tried something very simple: "Animate a round robin tournament for 10 people."
Results: Claude: ~10s, perfect working demo ChatGPT: ~20s, solid solution Grok 4: ~1000s, failed completely, gave me a truncated base64 blob
This wasn't some obscure edge case... it was basic data visualization that any decent model should handle. Yet somehow Grok 4 is "competing with humans" and has "99% tool accuracy"...
I don't buy it..
links: Claude: https://claude.ai/share/7a413a6a-5c01-44a1-aaed-8b237e5e9e94 Chatgpt: https://chatgpt.com/canvas/shared/687a9f9d4304819187ac7d98d3... Grok 4: https://grok.com/share/c2hhcmQtMw%3D%3D_20b61291-e1bb-45e5-a...
These benchmarks are either just wrong or measuring something completely divorced from practical utility imo...
Grok 4 has about 99% accuracy in picking the right tools and making tool calls with proper arguments almost every single time.
Where did this number come from? What is "the right tool"? I find this extremely subjective. As most engineers know, there is no right tool, but mostly a compromise where you pick the least worst tool and choose what risks you're willing to manage or not.
I believe in this context it means "tool" as in the MCP definition, e.g. "of the catalog of MCP integrations, it doesn't try to use the playright one to browse the web, it'd use the AWS docs one directly"
This is just my speculation, though, as I've never used Grok anything
Yeah, based on a previous comment, that makes sense. I am a little reassured that is what the author meant.
That's langchain terminology. LLMs usually are exposed to a set of tools. It's usually pretty obvious which are obvious, since there's only one tool that's even remotely associated with the task at hand.
Thanks for the info. This makes the article slightly less intolerable!
On your intelligence graph where it shows Grok 4 and OpenAI o4-mini as comparable (and among the highest intelligence rated models), it doesn’t have OpenAI o3 or o3-pro.
Yet all of my tests show o3 blows o4-mini out of the water.
What are you classifying as intelligence?
This article seems like pure garbage
Didn't the tldr of grok 4 was their over tuned for bencmhark results but in day to day tasks . It's actual not better than o3 / gpt5
grok4 is tortiously slow compared to all the other LLMs I use :(
Ya, even I feel its slow, Thats why I use it only for architecture planning and finding complex issue
> To be honest, this model not only competes with other AI models but also with humans, making it the first of its kind
Is this a joke
I can't take anything seriously with phrases like "it has not yet achieved AGI, but it is one leap forward in the race to AGI" - based on what? Nobody knows whether LLMs are a viable approach to AGI, nobody really agrees on what AGI is, hell, people don't really agree on what "I" is.
This is just not even science at all at this point, we're just into solid cargo cult.
If the answer involves giving even more money to Elon Musk, you asked the wrong question.
[flagged]