Give the model the facts, or give it tools

Give a bot a fish

Here the application gathers everything the model needs before it asks the model anything. [source]

Suppose a matter dashboard should open with a short note about outstanding items. The application queries the docket and the undertakings list, places the results in the prompt, and asks for a summary. The model might write: "You have three undertakings outstanding. The oldest, from the examination of 4 September, is now overdue; the most recent is due 31 October." Every fact in that sentence came from data the application supplied. The model only had to write.

The same pattern handles more complex tasks. To help schedule an examination for discovery, the application can ask the user for the witness and window, then, behind the scenes, pull counsel's calendars, the reporter's availability, and the firm's scheduling policy into the prompt. The model then proposes options and refines them in conversation, using real availability and real constraints. [source]

Web-connected chat products work this way too. In 2023, Microsoft's Bing chat had the model generate search queries, ran those searches, and placed summarized results in the hidden context for the model to answer from. [source] Retrieval-augmented generation over your own documents (How document-grounded AI answers work) is the same idea applied to a document collection.

The principle: change the context to match what the user is trying to do. When you can supply the facts, do. It is the most consistent and reliable approach. [source]

Teach a bot to fish

Sometimes you cannot know in advance what the model will need. A user may ask for every invoice over $10,000 in the last 90 days, a chart of hours by month, or to add a note to a particular document. For these, the model needs to request data or actions itself. [source]

From command lists to tool calling

In 2023, the common method was a command grammar: a table of commands described in the prompt, with the model asked to write out a sequence of them for the application to run. [source] A legal version might look like this:

The 2023 guides asked for the command sequence in a compact notation the application could parse: reverse Polish notation in one example, JSON with TypeScript type definitions in another. There was no standard format, and the authors called it an area of active research. [source]

Today this is built into models and their APIs as tool calling (also called function calling). You describe each tool's name, purpose, and inputs in a structured schema; the model decides when to call a tool and returns a structured request; your application runs it and sends back the result. [source] [source] Some APIs can enforce that tool requests match the declared schema exactly. [source] The Model Context Protocol (MCP) is an open standard for connecting AI applications to tools and data sources, so one integration can serve many AI products. [source]

Choose the right level of abstraction

The design lessons from command grammars still apply to tool design. [source]

• Too narrow: summarize_last_90_days_of_invoices does one thing and cannot be combined with anything else.
• Too low-level: read_byte_at_offset gives the model so much to orchestrate that it will make mistakes.
• About right: search_record and get_document are general building blocks the model can combine in useful ways.

With large or complicated tool sets, models tend to invent tools or arguments that plausibly could exist but do not. [source] Clear descriptions and examples of correct use are the best remedy; few-shot examples cost tokens, so balance them against the context budget.

Capable models often join information across tools without being told how. Asked to add a note to "the inspection email", a model may list the matter's documents, find the one that matches, take its ID, and pass that ID to add_note. [source] That is helpful, and it is also exactly the kind of inference to check, because the "matching" document may be the wrong one.

The reason-act loop

The ReAct pattern links reasoning, tool actions, and observations. In October 2022, researchers from Princeton and Google published ReAct, a pattern in which the model alternates between a thought, an action, and an observation of the result, repeating until it can answer. [source] This loop is the basis of today's agent loops.

A worked example: asked "Has the inspection mentioned in A02 ever been scheduled?", a model with the tools above might:

1. Think: I need to find any later document mentioning inspection. Act: search_record("M-104", "inspection"). Observe: A02, A03, A07.
2. Think: A02 and A03 are known. Check A07. Act: get_document("A07"). Observe: A07 is a 2 April email proposing "inspection week of 14 April, date TBC".
3. Think: Proposed, not fixed. Search for confirmation. Act: search_record("M-104", "inspection confirmed"). Observe: no results.
4. Answer: An inspection week was proposed in A07 (2 April), but no document in the record confirms a date. [A02, A03, A07]

The pattern is powerful because the model can recover from a dead end. The same feature means it can wander, repeat searches, or stop too early. Log every step so a reviewer can follow the path.

Model capability matters most here

In 2023, the difference between GPT-3.5 and GPT-4 was small for many simple tasks but large for tool use: the command-grammar examples did not work on the weaker model without many examples, and it was more likely to invent commands. [source] The models have changed; the lesson has not. Tool use and multi-step agent work are where weaker or smaller models (including many local models) fall short first. Test the specific model on your specific tools before relying on it.

Limits outside the prompt

Each tool is a permission. A tool that reads documents is low risk; a tool that sends email or deletes files is not. Give a task only the tools it needs, separate reading from writing or sending, and require human approval before consequential actions. See least privilege, and the lesson Recognize prompt injection for why a document the model reads should never be able to direct its tools. [source]