Generative AI has the potential to revolutionize nearly every industry, including healthcare, finance, and education
Imagine an artificial intelligence that not only answers questions, but also independently researches, analyzes data, interprets images and makes informed decisions - all in a fluid process. With the introduction of OpenAI's o3 model in April 2025, this has become a reality. This model marks a turning point in AI development by being able to autonomously use a variety of tools to accomplish complex tasks for the first time.
For the first time, these models can integrate images directly into their chain of thought. They don’t just see an image—they think with it. This unlocks a new class of problem-solving that blends visual and textual reasoning, reflected in their state-of-the-art performance across multimodal benchmarks.
One particularly impressive example: o3 was able to perform over 600 different tool calls in a single run to solve a particularly challenging task. This ability to use tools independently - also known as “agentic AI” - opens up completely new possibilities in the interaction between humans and machines.
But what exactly does tool-usage mean - and how does it relate to the other improvements? Let's find out.
What Does “Agentic Tool Use” Mean?
For the first time, our reasoning models can agentically use and combine every tool within ChatGPT—this includes searching the web, analyzing uploaded files and other data with Python, reasoning deeply about visual inputs, and even generating images.
Traditional AI models provide answers based on predefined data and algorithms. Agentic AI models such as o3, on the other hand, go one step further: they decide independently when and how to use different tools to fulfill a task. These include, for example, web searches, executing Python code, analyzing images or interpreting files.
This capability is based on advanced reinforcement learning, in which the model learns through rewards which actions lead to success. As a result, o3 can not only process information, but also actively act and make decisions - similar to a human assistant. The magic word here is “reinforcement learning”:
“We also trained both models to use tools through reinforcement learning—teaching them not just how to use tools, but to reason about when to use them. Their ability to deploy tools based on desired outcomes makes them more capable in open-ended situations—particularly those involving visual reasoning and multi-step workflows. This improvement is reflected both in academic benchmarks and real-world tasks, as reported by early testers.” (OpenAI)
The term “agentic tool usage” in the context of OpenAI's o3 model refers to the model's ability to independently decide when and how to use different tools to solve complex tasks. This represents a significant advance over previous models, which only used tools at the direct instruction of the user. In other words, it is trained like a good craftsman: it does not simply hammer away, but first considers which tool is useful and when - and reinforcement learning was helpful for this.
Practical Examples: How o3 Uses Tools
Some concrete use cases illustrate the potential of o3:
Data analysis: o3 can independently collect data from the Internet, analyze it with Python and present the results in comprehensible graphics.
Image processing: The model is capable of interpreting images, extracting relevant information and placing it in the context of the task at hand.
Complex problem solving: For particularly difficult tasks, o3 can use a large number of tools in a logical sequence to arrive at the solution step by step - as in the case of the over 600 tool calls.
These examples show that o3 not only processes information, but actively acts and uses a variety of tools in a coordinated manner.
o3’s ability to autonomously sequence over 600 tool calls in a single task marks a clear break from prior models—signaling a shift from reactive to reflective AI. Why? Because it marks a qualitative leap: Away from purely reactive AI, which only uses tools in response to direct user commands, towards an agent-like system that decides autonomously when, how and in what order it uses tools to solve a complex problem.
“They actually use these tools in their chain of thought as they’re trying to solve a hard problem. For example, we’ve seen o3 use like 600 tool calls in a row trying to solve a really hard task.” (Greg Brockman, OpenAI)
Earlier models - including GPT-4 with plugins or tools - were already powerful, but usually operated within a limited and linear tool framework. o3, on the other hand, demonstrates a kind of strategic thinking. It recognizes what information is missing, searches for it independently on the web, analyses it with Python if necessary, generates visualizations, checks intermediate results, discards partial results and starts a new path if necessary - up to 600 times in a single solution process. This is no longer a stubborn process, but an iterative, self-reflective analysis process.
This ability is not just a technical gimmick, but has very practical consequences. In areas such as data analysis, market observation, scientific research or even crisis management, this can mean Efficiency gains, deeper analysis, more autonomous processes. The human only sets the direction - the model does the rest by gradually working its way through the problem.
The Importance of Scaling AI
Continuing to scale reinforcement learning Throughout the development of OpenAI o3, we’ve observed that large-scale reinforcement learning exhibits the same “more compute = better performance” trend observed in GPT‑series pretraining. By retracing the scaling path—this time in RL—we’ve pushed an additional order of magnitude in both training compute and inference-time reasoning, yet still see clear performance gains, validating that the models’ performance continues to improve the more they’re allowed to think. At equal latency and cost with OpenAI o1, o3 delivers higher performance in ChatGPT—and we've validated that if we let it think longer, its performance keeps climbing.
A key issue in AI development is scaling - the ability to efficiently apply models to larger tasks or data volumes. Traditionally, this has been achieved by increasing the size of the model, but this is associated with increasing costs and resource consumption.
With agentic tool usage, o3 offers an alternative approach: instead of increasing the size of the model itself, its ability to use external tools effectively is improved. This enables more efficient processing of complex tasks without the need to massively scale the model itself. The graph below shows how much the tool-usage improves the benchmark results.
However, this approach also poses challenges, particularly in terms of coordinating the various tools and ensuring that they function reliably.
With the increasing use of tool usage in modern AI systems, a new dimension of scaling is emerging that goes beyond the classic principles of more parameters, more data or longer inference time. Tool usage allows models to make targeted use of external aids such as computational tools, web searches or visual analysis - similar to a person taking notes or consulting an encyclopedia while thinking. This behavior can be scaled: On the one hand by integrating additional tools, on the other hand by improving the decision logic of when and how these tools should be used.
Tool usage has a special effect in combination with traditional scaling laws. While larger models with more data build deeper representations of knowledge, tool usage enables the models to act situationally - i.e. not only to know, but also to make smart decisions about how to deal with uncertainty, information gaps or computational effort. This dynamic becomes particularly clear in the context of inference scaling: the more time and computing capacity a model has at its disposal, the more precisely it can plan which tool should be used in which order. Tool usage is therefore not a substitute for classic scaling, but a complementary path - an additional lever to make AI systems more practical, more capable and more intelligent.
Conclusion: A Look Into the Future of AI
The introduction of OpenAI's o3 model marks a significant step in the development of artificial intelligence. The ability to use a variety of tools autonomously transforms AI from a passive information processor to an active problem solver.
This development opens up new opportunities in areas such as research, education, medicine and many others. At the same time, it presents us with new challenges, especially in terms of controlling and understanding the decisions that such agentic AI systems make.
The integration of tool usage into modern AI models opens up new possibilities, but also presents a number of complex challenges. First of all, the AI must not only learn how a tool works, but above all when and why it should be used sensibly. This decision requires a form of “meta-intelligence”, i.e. the ability to reflect on its own thought process - something that AI has only approximately mastered so far. It becomes particularly problematic with tools such as web searches, image analysis or code interpreters, whose results are not always unambiguous. The AI must then evaluate the quality, relevance and trustworthiness of the results - a task that is often difficult even for humans. There is a risk here that models uncritically adopt content, for example from blogs with hidden advertising or from outdated sources.
Another problem lies in the dependence on tool output: If a tool works incorrectly or provides manipulated information, it is difficult for the AI to recognize this. There is a lack of real “judgment” in the human sense. Tool usage also increases the complexity of the system, which leads to longer response times, higher costs and potentially new attack surfaces - for example through prompt injection via external sources. Finally, there is also an ethical issue: when AI models automatically access web content without clearly identifying the author and context, the boundaries between information, interpretation and reproduction become blurred.
Nevertheless: o3 demonstrates with its outstanding tool-usage that we will now see improvements in the tool-usage in addition to scaling in pre-training and inference-scaling, which will also improve future models.
—
Ready for more content from Kim Isenberg? Subscribe to FF Daily for free!
