openAgent

1# @opencode-ai/llm
2 
3Schema-first LLM core for opencode. One typed request, response, event, and tool language; provider quirks live in adapters, not in calling code.
4 
5```ts
6import { Effect } from "effect"
7import { LLM, LLMClient } from "@opencode-ai/llm"
8import { OpenAI } from "@opencode-ai/llm/providers"
9 
10const model = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY }).responses("gpt-4o-mini")
11 
12const request = LLM.request({
13  model,
14  system: "You are concise.",
15  prompt: "Say hello in one short sentence.",
16  generation: { maxTokens: 40 },
17})
18 
19const program = Effect.gen(function* () {
20  const response = yield* LLMClient.generate(request)
21  console.log(response.text)
22})
23```
24 
25Run `LLMClient.stream(request)` instead of `generate` when you want incremental `LLMEvent`s. The event stream is provider-neutral — same shape across OpenAI Chat, OpenAI Responses, Anthropic Messages, Gemini, Bedrock Converse, and any OpenAI-compatible deployment.
26 
27## Public API
28 
29- **`LLM.request({...})`** — build a provider-neutral `LLMRequest`. Accepts ergonomic inputs (`system: string`, `prompt: string`) that normalize into the canonical Schema classes.
30- **`LLM.generate` / `LLM.stream`** — re-exported from `LLMClient` for one-import use.
31- **`Message.user(...)` / `Message.assistant(...)` / `Message.tool(...)`** — message constructors from the canonical schema model.
32- **`Model.make(...)` / `ToolCallPart.make(...)` / `ToolResultPart.make(...)` / `ToolDefinition.make(...)`** — model and tool-related constructors from the canonical schema model.
33- **`LLMClient.prepare(request)`** — compile a request through protocol body construction, validation, and HTTP preparation without sending. Useful for inspection and testing.
34- **`LLMEvent.is.*`** — typed guards (`is.textDelta`, `is.toolCall`, `is.finish`, …) for filtering streams.
35 
36## Caching
37 
38Prompt caching is **on by default**. Every `LLMRequest` resolves to `cache: "auto"` unless the caller opts out with `cache: "none"`. Each protocol translates `CacheHint`s to its wire format (`cache_control` on Anthropic, `cachePoint` on Bedrock; OpenAI and Gemini do implicit caching server-side and don't need inline markers — auto is a no-op there).
39 
40### Auto placement
41 
42`"auto"` places three breakpoints — last tool definition, last system part, latest user message. The last-user-message boundary is the load-bearing detail: in a tool-use loop, a single user turn expands into many assistant/tool round-trips, all sharing that prefix. Caching at that boundary lets every intra-turn API call hit.
43 
44The math justifies the default: Anthropic's 5-minute cache write is 1.25× base, read is 0.1×, so a single reuse within 5 minutes already wins. One-shot completions below the per-model minimum-cacheable-token threshold silently no-op on the wire, so the worst case is harmless.
45 
46### Opting out
47 
48```ts
49LLM.request({
50  model,
51  system,
52  prompt: "one-off question",
53  cache: "none",
54})
55```
56 
57### Granular policy
58 
59```ts
60cache: {
61  tools?: boolean,
62  system?: boolean,
63  messages?: "latest-user-message" | "latest-assistant" | { tail: number },
64  ttlSeconds?: number,         // ≥ 3600 → 1h on Anthropic/Bedrock; else 5m
65}
66```
67 
68### Manual hints
69 
70Inline `CacheHint` on any text / system / tool / tool-result part overrides automatic placement. The auto policy preserves manual hints; it only fills gaps.
71 
72```ts
73LLM.request({
74  model,
75  system: [
76    { type: "text", text: "stable system prompt", cache: { type: "ephemeral" } },
77  ],
78  ...
79})
80```
81 
82### Provider behavior table
83 
84| Protocol                | `cache: "auto"`                                                           |
85| ----------------------- | ------------------------------------------------------------------------- |
86| Anthropic Messages      | emits up to 3 `cache_control` markers (4-breakpoint cap enforced)         |
87| Bedrock Converse        | emits up to 3 `cachePoint` blocks (4-breakpoint cap enforced)             |
88| OpenAI Chat / Responses | no-op (implicit caching above 1024 tokens)                                |
89| Gemini                  | no-op (implicit caching on 2.5+; explicit `CachedContent` is out-of-band) |
90 
91Normalized cache usage is read back into `response.usage.cacheReadInputTokens` and `cacheWriteInputTokens` across every provider.
92 
93## Providers
94 
95Provider facades configure endpoint/auth/deployment details first, then expose model selectors that take only a model or deployment id. The selected model carries the executable route value used at runtime.
96 
97```ts
98import { OpenAI, CloudflareAIGateway } from "@opencode-ai/llm/providers"
99 
100const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY }).responses("gpt-4o-mini")
101const gateway = CloudflareAIGateway.configure({
102  accountId: process.env.CLOUDFLARE_ACCOUNT_ID,
103  gatewayApiKey: process.env.CLOUDFLARE_API_TOKEN,
104}).model("workers-ai/@cf/meta/llama-3.1-8b-instruct")
105```
106 
107Included providers: OpenAI, Anthropic, Google (Gemini), Amazon Bedrock, Azure OpenAI, Cloudflare AI Gateway, Cloudflare Workers AI, GitHub Copilot, OpenRouter, xAI, plus generic OpenAI-compatible helpers for DeepSeek, Cerebras, Groq, Fireworks, Together, etc.
108 
109## Provider options & HTTP overlays
110 
111Three escape hatches in order of stability:
112 
1131. **`generation`** — portable knobs (`maxTokens`, `temperature`, `topP`, `topK`, penalties, seed, stop).
1142. **`providerOptions: { <provider>: {...} }`** — typed-at-the-facade provider-specific knobs (OpenAI `promptCacheKey`, Anthropic `thinking`, Gemini `thinkingConfig`, OpenRouter routing).
1153. **`http: { body, headers, query }`** — last-resort serializable overlays merged into the final HTTP request. Reach for this only when a stable typed path doesn't yet exist.
116 
117Route/provider defaults are overridden by request-level values for each axis.
118 
119## Routes
120 
121Adding a new model or deployment is usually 5-15 lines using `Route.make({ protocol, endpoint, auth, framing, ... })`. The route owns endpoint/auth/framing and the protocol owns body construction plus stream parsing. Transports are reusable IO templates that receive route endpoint/auth at compile time. Capability/catalog metadata lives outside this low-level package; unsupported request shapes fail during protocol lowering. See `AGENTS.md` for the architectural detail.
122 
123## Effect
124 
125This package is built on Effect. Public methods return `Effect` or `Stream`; provide `LLMClient.layer` for runtime dispatch and import the provider/protocol modules for the routes you use. The example at `example/tutorial.ts` is a runnable walkthrough.
126 
127## See also
128 
129- `AGENTS.md` — architecture, route construction, contributor guide
130- `example/tutorial.ts` — runnable end-to-end walkthrough
131- `test/provider/*.test.ts` — fixture-first protocol tests; `*.recorded.test.ts` files cover live cassettes
132