A practical visual guide · 20-minute overview · Reading reference

Local AIFrom Model Weights
to Useful Agents

Understanding the tradeoffs—and getting more from the hardware available to you.

Read the guide

Source checked November 13, 2026 · Calculations and schematic demonstrations · No local benchmark measurements

Act 1 · Why local AI?1 / 34

Your task. Your choice.

The task stays the same. You choose where the model runs.

Your task. Your choice. The task stays central as inference moves from a hosted service to a local runtime. Model choice, data handling and availability are controls around the same coding task.
analysis.py
def summarize(data):
  values = clean(data)
  return mean(values)

One task. Keep working.

Hosted serviceLocal runtime

Choose the model, data path and availability.

Schematic · deployment choices, not measured performance

A document needs another pass. A test is failing. Local inference offers control over model choice, runtime settings, data handling, and availability. It can complement the hosted products you already use. The useful question is whether a particular deployment helps you finish this particular task.

Technical notes, examples & sources

Opening prompt: imagine you are halfway through debugging a function. Keep the task visible while switching deployment. Control permits deliberate choices; it does not guarantee quality, security or reliability. The four-scene opening takes about four minutes within a 20-minute overview. Reading mode retains the detail.

Primary sources · checked 2026-11-13

Act 1 · Why local AI?2 / 34

Different limits. Different causes.

Quota, context, session, and timeout failures are different problems.

Different limits. Different causes. Four separate interruption symbols surround the same coding task: a quota ring, full context container, broken session and stopped clock. These are categories, not a sequence of failures.
Usage quota
Product allowance reached
Context limit
Request too large
Session failure
History unavailable
Execution timeout
Run stopped
Schematic · deployment choices, not measured performance

Illustrative hosted-product interruptions can look similar while having different causes. A usage quota is a product allowance; a context limit bounds a request; a session failure affects stored conversation access; an execution timeout bounds a run. Local software can also hit limits, lose state, or fail.

Technical notes, examples & sources

These are generic examples, not quotations attributed to a provider. A quota reset will not repair a malformed request. A larger context will not repair a broken session. Ask the audience which kind of interruption they have encountered.

Primary sources · checked 2026-11-13

Act 1 · Why local AI?3 / 34

Choose the tradeoffs.

There is no universal winner. Compare the whole workflow.

Choose the tradeoffs. A local device and hosted service flank the same task. Paired rows compare control, capability, operations and availability. These are qualitative tradeoffs, with no scores or universal winner.
Qualitative tradeoffs · no universal winner
LocalFrontier cloud
Runtime & model choiceManaged experience
Test the chosen modelMay handle harder tasks
Hardware, energy, upkeepPlan or API charges
Installed resources & limitsNetwork, service & quotas
Schematic · deployment choices, not measured performance

Frontier cloud models may be stronger on difficult tasks and offer managed infrastructure and polished tools. Local models offer direct runtime control and no provider inference quota, but still consume hardware, electricity, and maintenance time. Privacy depends on interfaces, extensions, tools, telemetry, and network access; cloud policies depend on product and contract.

Technical notes, examples & sources

Cost includes acquisition or rental, energy, time, and opportunity cost. Latency and context are properties of a specific model, runtime, workload, and service. Avoid unsupported rankings. Local data can still leave through a search tool or extension.

Choose the tradeoffs.
CriterionLocalFrontier cloud
CapabilityModel and hardware dependentMay excel on difficult tasks
ConvenienceYou manage the runtimeManaged service and integrations
PrivacyInspect the entire tool chainInspect product and contract
CostHardware, energy, maintenancePlan or API charges
AvailabilityInstalled resources; local failuresNetwork, quotas, service availability
LatencyPlacement and workload dependentNetwork and service dependent
ContextArchitecture and memory limitsModel and product limits

Primary sources · checked 2026-11-13

Act 1 · Why local AI?4 / 34

Start with the work.

Local, cloud, and hybrid are all reasonable outcomes.

Start with the work. Private documents, repeated work, coding and experiments, and offline work have different constraints. Local, cloud and hybrid choices highlight candidate uses, not performance rankings.

A hybrid approach can keep private drafts local and use a hosted model for a hard, shareable problem.

Private documentsTrace the data path
Repeated workCost & availability
Coding & experimentsCapability & tools
Offline workInstall resources first
Schematic · deployment choices, not measured performance

Private document work favors careful control of the data path. Repetitive processing rewards predictable marginal costs. Offline use requires weights, software, and dependencies to be installed first. Coding assistance needs capable tool use and a harness that can work with the repository. Experimentation benefits from runtime access.

Technical notes, examples & sources

A hybrid workflow might use a local model for private drafts and a hosted model for a hard, appropriately shareable problem. This is a decision framework, not a performance leaderboard. Transition by asking what the local runtime actually loads.

Primary sources · checked 2026-11-13

Act 2 · What an LLM does5 / 34

Text becomes tokens

Tokens are pieces of text, not necessarily whole words.

Text becomes tokens Tokens are pieces of text, not necessarily whole words. The Then function Then returns Then a value.. Thefunctionreturnsa value.
  1. 1The
  2. 2function
  3. 3returns
  4. 4a value.
Schematic · steps are not measured inference time

A tokenizer maps text into token IDs. The model processes these IDs rather than raw words. Spaces, punctuation, code, and language influence the count. The chip segmentation here is schematic; it is not output from Qwen’s tokenizer. Exact counts require the selected model’s tokenizer and chat template.

Technical notes, examples & sources

Keep these token chips recognizable: later they become cached keys and values, then occupy the context budget. A displayed word chip may correspond to more or fewer real tokens. Do not use this diagram to estimate a bill or memory usage.

Primary sources · checked 2026-11-13

Act 2 · What an LLM does6 / 34

Weights are learned numbers

Inference uses learned parameters to predict the next token.

Weights are learned numbers Inference uses learned parameters to predict the next token. Tokens Then Layer 1 Then Layer 2 Then Next-token scores. TokensLayersNext-token scores
  1. 1Tokens
  2. 2Layer 1
  3. 3Layer 2
  4. 4Next-token scores
Schematic · steps are not measured inference time

Training adjusts many numerical parameters. During ordinary inference the loaded weights remain fixed. Each pass transforms token representations through layers, scores possible next tokens, and selects a continuation according to the sampling settings. The selected token joins the sequence and the process repeats.

Technical notes, examples & sources

A conversation changes the supplied context, not the weights. Fine-tuning is a separate operation. The displayed candidate probabilities are schematic and are not recorded model outputs. A multimodal artifact can include additional encoders and projections.

Primary sources · checked 2026-11-13

Act 2 · What an LLM does7 / 34

Precision determines storage

The same parameter count can occupy very different amounts of memory.

Precision determines storage The same parameter count can occupy very different amounts of memory. FP32 · 108 GB Then 16-bit · 54 GB Then 8-bit · 27 GB Then 4-bit · 13.5 GB. FP32 · 108 GB16-bit · 54 GB8-bit · 27 GB4-bit · 13.5 GB
  1. 1FP32 · 108 GB
  2. 216-bit · 54 GB
  3. 38-bit · 27 GB
  4. 44-bit · 13.5 GB
Calculated weight storage

For 27 billion parameters, multiply the count by bits per weight and divide by eight. The theoretical weight-only sizes are 108 GB at FP32, 54 GB at 16-bit, 27 GB at 8-bit, and 13.5 GB at 4-bit. These use decimal GB, not GiB.

Technical notes, examples & sources

These are idealized storage calculations. Actual files mix tensor types, scales, metadata, and potentially vision or draft components. Weight storage format is distinct from arithmetic and accumulation precision. Four-bit storage does not mean every operation executes in four-bit arithmetic.

Precision determines storage
StorageBits / parameterIdeal decimal GB
FP3232108
FP16 / BF161654
Ideal 8-bit827
Ideal 4-bit413.5

Primary sources · checked 2026-11-13

Act 2 · What an LLM does8 / 34

Attention connects the context

A token can draw on relevant information earlier in the sequence.

Attention connects the context A token can draw on relevant information earlier in the sequence. The function Then reads a file Then It returns Then the contents. The functionreads a fileIt returnsthe contents
  1. 1The function
  2. 2reads a file
  3. 3It returns
  4. 4the contents
Schematic · steps are not measured inference time

In “The function reads a file. It returns the contents,” the continuation may depend on an earlier object. Attention forms queries and keys, computes compatibility scores, normalizes them, and combines values. The highlighted connections are a teaching analogy, not a measurement of what this model attends to.

Technical notes, examples & sources

Actual scaled dot-product attention is softmax(QKᵀ / √d)V, with causal masking during next-token prediction. Real models use multiple heads and layers, plus normalization and other modifications. Keys are not literal keyword indexes; values are learned representations.

Primary sources · checked 2026-11-13

Act 2 · What an LLM does9 / 34

The KV cache

Store earlier keys and values so generation can reuse them.

The KV cache Store earlier keys and values so generation can reuse them. Fixed weights Then Token 1 · K/V Then Token 2 · K/V Then Token 3 · K/V. Fixed weightsRetained KV cache
  1. 1Fixed weights
  2. 2Token 1 · K/V
  3. 3Token 2 · K/V
  4. 4Token 3 · K/V
Schematic · steps are not measured inference time

During generation, earlier tokens’ keys and values can be reused rather than recomputed at every step. Weights stay broadly fixed for the loaded model. Cache demand changes with context, architecture, representation, and the number of simultaneous sequences. Recurrent layers retain state differently.

Technical notes, examples & sources

Caching saves computation; it does not make context free. Some runtimes reserve capacity ahead of actual occupancy. Reused prefixes may save work but still belong to the logical prompt. The schematic shows retained entries, not a byte-for-byte memory layout.

Primary sources · checked 2026-11-13

Act 2 · What an LLM does10 / 34

Reading the prompt, writing the answer

Prefill and decode are different workloads.

Reading the prompt, writing the answer Prefill and decode are different workloads. Known prompt Then Parallel prefill Then First token Then Sequential decode. Known promptParallel prefillFirst tokenSequential decode
  1. 1Known prompt
  2. 2Parallel prefill
  3. 3First token
  4. 4Sequential decode
Schematic · steps are not measured inference time

Prefill processes known prompt tokens with substantial parallelism. Ordinary autoregressive decode produces dependent tokens sequentially. Time to first token includes waiting, loading, and prompt work. Prompt throughput, generation throughput, and total request time answer different questions.

Technical notes, examples & sources

Ollama timing fields are nanoseconds. For versions reporting prompt_eval_cached_count, uncached prefill tokens/s = (prompt_eval_count − prompt_eval_cached_count) / prompt_eval_duration × 10⁹. Decode tokens/s = eval_count / eval_duration × 10⁹. Inspect installed-version field semantics; a missing cached count does not prove an uncached run. Measure user-visible first token with a streaming client. Keep cold loading, queueing, prefix reuse, and full request time separate.

Primary sources · checked 2026-11-13

Act 3 · Hardware and memory11 / 34

Capacity, bandwidth, and compute

Fitting, moving data, and doing arithmetic are separate constraints.

Capacity, bandwidth, and compute Fitting, moving data, and doing arithmetic are separate constraints. Capacity Then Bandwidth Then Compute Then Workload. CapacityBandwidthComputeWorkload
  1. 1Capacity
  2. 2Bandwidth
  3. 3Compute
  4. 4Workload
Schematic · steps are not measured inference time

Capacity determines whether the working set fits. Bandwidth determines how quickly required data can move. Compute determines how quickly kernels can perform operations. Single-stream decode is often bandwidth-sensitive; prefill often benefits more from compute. Neither is a universal rule.

Technical notes, examples & sources

Change the highlighted bottleneck in the schematic. It does not predict tokens/s. Kernel support, batch size, cache traffic, offloading, and power or thermal constraints can change the bottleneck. Peak TOPS alone does not characterize a complete model workload.

Primary sources · checked 2026-11-13

Act 3 · Hardware and memory12 / 34

One shared pool, or separate pools

Memory arrangement changes placement and transfer costs.

One shared pool, or separate pools Memory arrangement changes placement and transfer costs. System + GPU pool Then OS & headroom Then System RAM Then Discrete GPU VRAM. Unified physical poolSystem RAMSeparate GPU VRAM
  1. 1System + GPU pool
  2. 2OS & headroom
  3. 3System RAM
  4. 4Discrete GPU VRAM
Schematic · steps are not measured inference time

Apple unified memory provides a shared physical pool for CPU and GPU use. The operating system still needs memory, GPU working sets have limits, and some operations may execute on the CPU. Discrete GPUs have VRAM separate from system RAM; offloading can introduce transfers and slower execution.

Technical notes, examples & sources

The same workload appears in both arrangements. Unified memory does not mean unlimited GPU allocation. Matrix acceleration helps only when the backend supports the relevant numerical format and kernels. These generic arrangements describe no personal system.

Primary sources · checked 2026-11-13

Act 3 · Hardware and memory13 / 34

GPU residency and automatic placement

A model can load successfully while layers execute on the CPU.

GPU residency and automatic placement A model can load successfully while layers execute on the CPU. Layer 1 · GPU Then Layer 2 · GPU Then Layer 3 · CPU Then Request → verify. Layer 1 · GPULayer 2 · GPULayer 3 · CPURequest → verify
  1. 1Layer 1 · GPU
  2. 2Layer 2 · GPU
  3. 3Layer 3 · CPU
  4. 4Request → verify
Schematic · steps are not measured inference time

Automatic placement uses estimates. When sufficient capacity exists, a supported explicit offload request can help if automatic estimates leave layers on the CPU. In the checked Ollama runner, num_gpu is a layer-offload setting, not a count of physical GPUs.

Technical notes, examples & sources

For Ollama v0.40.16, native API options.num_gpu: 999 requests an oversized layer count passed to the runner as -ngl. It is not a guarantee that all layers fit. It is an implementation-level option, so recheck after upgrades. In unified memory the execution regions are not separate physical memory pools.

Documentation/source checked · 2026-11-13 · not executed inference

curl http://localhost:11434/api/generate -d '{"model":"local-coder","prompt":"Reply OK","stream":false,"options":{"num_ctx":8192,"num_gpu":999}}'

Primary sources · checked 2026-11-13

Act 3 · Hardware and memory14 / 34

Verify what actually happened

Loaded → context allocated → realistic workload completed.

Verify what actually happened Loaded → context allocated → realistic workload completed. Requested settings Then Effective runner Then Allocated context Then Completed workload. Requested settingsEffective runnerAllocated contextCompleted workload
  1. 1Requested settings
  2. 2Effective runner
  3. 3Allocated context
  4. 4Completed workload
Schematic · steps are not measured inference time

Requested settings, effective runner settings, and observed behavior are three different kinds of evidence. Start with ollama ps. Inspect layer-offload logs, cache types, runner batch flags, and memory measurements. Then run the prompt lengths and tool loops you intend to use.

Technical notes, examples & sources

Full GPU layer placement does not mean every operation runs on the GPU. A successful allocation reserves capacity; it is not evidence that a long request completed under pressure. Save exact artifact, runtime version, effective options, logs with identifiers removed, and test conditions.

Documentation/source checked · 2026-11-13 · not executed inference

ollama --version
ollama ps
ollama show local-coder
ollama show --modelfile local-coder

Primary sources · checked 2026-11-13

Act 4 · Making models practical15 / 34

Architecture changes cache growth

Not every layer needs a full history of keys and values.

Architecture changes cache growth Not every layer needs a full history of keys and values. Many query heads Then Shared KV heads Then Sliding history Then Recurrent state. RecurrentRecurrentRecurrentFull attentionRecurrentRecurrentRecurrentFull attentionSlidingSlidingSlidingFull attentionSlidingSlidingSlidingFull attention
  1. 1Many query heads
  2. 2Shared KV heads
  3. 3Sliding history
  4. 4Recurrent state
Schematic · steps are not measured inference time

Grouped-query attention shares KV heads across query heads. Sliding attention limits each relevant layer’s attention span. Hybrid recurrent models combine a bounded recurrent state with full-attention layers. The resulting memory formula depends on the actual architecture and the runtime’s cache policy.

Technical notes, examples & sources

GQA reduces KV head count. Sliding-window attention bounds retained history only when the implementation uses a bounded cache. Recurrent state is not a compressed transcript; full attention in the same hybrid still grows with context. None of these designs eliminates all context-dependent computation.

Primary sources · checked 2026-11-13

Act 4 · Making models practical16 / 34

Qwen and Muse Glimmer

Two hybrid designs, two different cache budgets.

Qwen and Muse Glimmer Two hybrid designs, two different cache budgets. Qwen · 3 recurrent Then Qwen · 1 full Then Muse · 3 sliding Then Muse · 1 full. RecurrentRecurrentRecurrentFull attentionRecurrentRecurrentRecurrentFull attentionSlidingSlidingSlidingFull attentionSlidingSlidingSlidingFull attention
  1. 1Qwen · 3 recurrent
  2. 2Qwen · 1 full
  3. 3Muse · 3 sliding
  4. 4Muse · 1 full
Schematic · steps are not measured inference time

Qwen3.8-27B has 64 text layers: 48 Gated DeltaNet layers and 16 full-attention layers. Muse Glimmer-30B has 52: 39 sliding-window layers and 13 full-attention layers, with a 2,048-token window. Each repeats three specialized layers followed by one full-attention layer.

Calculated retained KV payload · one sequence · FP16 K/V. Qwen adds recurrent state; both exclude other allocations. Muse assumes bounded sliding caches.
Exact values and formulas
Cache payload in GiB, not full resident memory
TokensQwen full-attention KVMuse bounded-window KV
8,1920.5000.178
32,7682.0000.482
65,5364.0000.889
131,0728.0001.701
Qwen bytes = 2 × 16 × 4 × 256 × 2 × tokens
Muse bytes = 2 × 2 × 128 × 2 ×
             (13 × tokens + 39 × min(tokens, 2048))
GiB = bytes / 2^30
Qwen recurrent matrices = 144 MiB per sequence at FP32,
plus convolution state (runtime allocation differs).
Technical notes, examples & sources

Checked November 13, 2026 against pinned configurations. Qwen full attention: four KV heads of dimension 256. Muse: two KV heads of dimension 128. Qwen native context is 262,144; Muse 131,072. Compare cache payload, not model capability or full resident memory. Qwen also has recurrent and convolution state; both can have vision or draft resources.

Primary sources · checked 2026-11-13

Act 4 · Making models practical17 / 34

Weight quantization

Fewer stored bits trade memory against numerical fidelity.

Weight quantization Fewer stored bits trade memory against numerical fidelity. Original values Then Quantization groups Then Representable values Then Smaller weights. Same learned tensors · smaller representations
  1. 1Original values
  2. 2Quantization groups
  3. 3Representable values
  4. 4Smaller weights
Schematic · steps are not measured inference time

Quantization maps many possible weight values to fewer representable values. Format names such as IQ4_XS, Q4_K_M, Q6_K, and Q8_0 describe different schemes, block structures, and precision mixes. Similar nominal bit widths do not imply identical size, quality, or speed.

Technical notes, examples & sources

Choose an exact artifact, not merely “a 4-bit model.” Quality is task-dependent. Execution efficiency depends on kernels and hardware. Quantization scales and metadata add storage; some tensors retain higher precision. The distribution graphic is schematic and does not describe a specific quantizer’s exact algorithm.

Primary sources · checked 2026-11-13

Act 4 · Making models practical18 / 34

Dynamic quantization

Some components benefit from more precision than others.

Dynamic quantization Some components benefit from more precision than others. Sensitive region Then More precision Then Other regions Then Fewer stored bits. Mixed precision · retain detail in sensitive regions
  1. 1Sensitive region
  2. 2More precision
  3. 3Other regions
  4. 4Fewer stored bits
Schematic · steps are not measured inference time

Mixed-precision quantization can retain more detail in sensitive components while reducing others more aggressively. Unsloth’s Dynamic v3.1 preview update for Qwen3.8 was dated November 11, 2026. Its reported accuracy gains are publisher benchmark claims, not independently measured speed improvements.

Technical notes, examples & sources

Unsloth reports roughly 5–10% accuracy improvements at comparable sizes on its benchmark selection. Results depend on evaluation protocol and context budget. This diagram shows the idea of unequal precision; it is not a literal implementation of an undocumented selection algorithm. A new quant does not automatically cause a dramatic throughput gain.

Primary sources · checked 2026-11-13

Act 4 · Making models practical19 / 34

Cache precision is a separate control

Weights are one allocation. Cache and state are another.

Cache precision is a separate control Weights are one allocation. Cache and state are another. Weight allocation Then Cache / state Then Draft + workspace Then Headroom. Weight allocationCache / stateDraft + workspaceHeadroom
  1. 1Weight allocation
  2. 2Cache / state
  3. 3Draft + workspace
  4. 4Headroom
Schematic · steps are not measured inference time

Reducing weight size frees a mostly fixed allocation. Reducing KV precision changes the context-dependent portion. Total memory also includes recurrent state, draft resources, temporary working buffers, and headroom. Runtime allocations can exceed a theoretical payload estimate.

Technical notes, examples & sources

Conventional KV bytes = sequences × layers × KV heads × head dimension × tokens × (key bytes + value bytes). Sum layer-specific terms for hybrid/windowed designs. Q8_0 and Q4_0 have block overhead; the calculator uses 34/32 and 18/32 bytes per element. Qwen recurrent matrices add 144 MiB per sequence at FP32, plus convolution state and allocator overhead.

Primary sources · checked 2026-11-13

Act 5 · Configure, measure, tune20 / 34

The software around the model

An artifact, engine, server, API, and interface do different jobs.

The software around the model An artifact, engine, server, API, and interface do different jobs. Model artifact Then Inference engine Then Server + API Then Interface / harness. Model artifactInference engineServer + APIInterface / harness
  1. 1Model artifact
  2. 2Inference engine
  3. 3Server + API
  4. 4Interface / harness
Schematic · steps are not measured inference time

GGUF is an artifact format. llama.cpp is an inference implementation with command-line and server tools. Ollama manages models and serves APIs. MLX LM builds on MLX for Apple silicon. vLLM is an inference and serving engine. A chat interface or coding harness sits above these layers.

Technical notes, examples & sources

These projects overlap in capability; they are not interchangeable boxes or a mandatory stack. Ollama is the main path here because model management and integrations make it approachable. This is not a claim that it is universally fastest.

Primary sources · checked 2026-11-13

Act 5 · Configure, measure, tune21 / 34

Establish a working baseline

Choose an exact artifact, then verify a small complete request.

Establish a working baseline Choose an exact artifact, then verify a small complete request. Exact artifact Then Server settings Then Small request Then Saved baseline. Exact artifactServer settingsSmall requestSaved baseline
  1. 1Exact artifact
  2. 2Server settings
  3. 3Small request
  4. 4Saved baseline
Schematic · steps are not measured inference time

Record the runtime version and exact downloaded quantization. Import it under a fictional alias, establish a conservative context, load it, complete a small request, and inspect the effective runner. Save the baseline before changing settings. A download’s size is not a universal memory minimum.

Technical notes, examples & sources

The baseline uses qwen3.8:27b-q4_K_M, the explicit non-MTP tag. Record the local digest/ID because tags can change. Its download size is not a memory requirement. Commands were documentation/source checked for Ollama v0.40.16; no download or inference was performed for this guide. Stop any existing app/server through its normal controls before starting a foreground server. Environment variables must reach the server process. The 8,192-token baseline is a small smoke test, not an agent recommendation. For a publisher GGUF, use FROM ./models/model.gguf after recording the exact revision, filename, and SHA-256.

Documentation/source checked · 2026-11-13 · not executed inference

ollama --version
# First terminal, after stopping any existing server:
OLLAMA_NO_CLOUD=1 OLLAMA_CONTEXT_LENGTH=8192 OLLAMA_NUM_PARALLEL=1 OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=f16 ollama serve

# Second terminal:
ollama pull qwen3.8:27b-q4_K_M
ollama show qwen3.8:27b-q4_K_M
ollama show --modelfile qwen3.8:27b-q4_K_M
printf 'FROM qwen3.8:27b-q4_K_M\nPARAMETER num_ctx 8192\nPARAMETER num_batch 256\nPARAMETER num_predict 256\nPARAMETER draft_num_predict 0\n' > Modelfile
ollama create local-coder -f Modelfile
curl http://localhost:11434/api/generate -H 'Content-Type: application/json' -d '{"model":"local-coder","prompt":"Reply OK","stream":false,"options":{"num_predict":32}}'
ollama ps

Primary sources · checked 2026-11-13

Act 5 · Configure, measure, tune22 / 34

Draft, verify, accept

Speculation helps only when saved work exceeds its overhead.

Draft, verify, accept Speculation helps only when saved work exceeds its overhead. Committed token Then Draft A ✓ Then Draft B ✓ Then Draft C ×. CommittedDraft ADraft BDraft C
  1. 1Committed token
  2. 2Draft A ✓
  3. 3Draft B ✓
  4. 4Draft C ×
Schematic · steps are not measured inference time

A drafter proposes a continuation. The target checks it, commits accepted tokens, and replaces a rejected continuation. Native multi-token prediction uses model-trained draft machinery; other speculative methods use a separate assistant. Qwen has an MTP layer. Muse’s companion uses DFlash block drafting.

Technical notes, examples & sources

Model support is not proof a selected runtime or imported artifact enables it. Ollama v0.40.16 has speculation controls in source; availability depends on engine, artifact, and backend. Keep the baseline without speculation, verify effective draft settings before comparisons, and measure acceptance and end-to-end duration. The schematic acceptance pattern is illustrative, not a benchmark.

/* Native Ollama options, compatible MTP artifact only */
{"draft_num_predict": 0}
/* Compare the same workload with: */
{"draft_num_predict": 4}

Check the selected artifact includes usable MTP tensors; inspect effective draft flags. Measure acceptance and complete request duration.

Primary sources · checked 2026-11-13

Act 5 · Configure, measure, tune23 / 34

Batch and microbatch

Prompt chunks and physical execution chunks are different.

Batch and microbatch Prompt chunks and physical execution chunks are different. Prompt tokens Then Logical batch Then Physical microbatch Then Temporary workspace. Prompt tokensLogical batchPhysical microbatchTemporary workspaceRetained cache · separateTemporary buffers
  1. 1Prompt tokens
  2. 2Logical batch
  3. 3Physical microbatch
  4. 4Temporary workspace
Schematic · steps are not measured inference time

A logical batch groups tokens submitted for evaluation. A physical microbatch determines how much is processed at once. Larger batches may improve prefill while requiring more working memory. Smaller ones can ease allocation pressure. These temporary buffers are separate from retained cache.

Technical notes, examples & sources

For Ollama v0.40.16 the checked runner passes num_batch to both -b and -ub. Do not infer independent microbatch control from a single API field. llama.cpp exposes separate batch/microbatch controls; verify their actual flags for the selected release. Inspect the runner command after changing a setting.

Primary sources · checked 2026-11-13

Act 5 · Configure, measure, tune24 / 34

Tune one variable, then combinations

A memory estimate can guide a test. It cannot predict useful speed.

Tune one variable, then combinations A memory estimate can guide a test. It cannot predict useful speed. Choose a baseline Then Change one variable Then Measure the workload Then Test combinations. Choose a baselineChange one variableMeasure the workloadTest combinations
  1. 1Choose a baseline
  2. 2Change one variable
  3. 3Measure the workload
  4. 4Test combinations
Schematic · steps are not measured inference time

Explore weight size, KV type, context, batch, draft resources, concurrency, and GPU placement in the calculator. Only quantities with a stated model become calculated values. Batch, draft length, and placement generate test guidance rather than invented speed or quality scores.

Calculated scenario · Qwen text model · not a fit guarantee

Partial estimate: unentered convolution state, vision resources, workspace, allocator overhead, and draft resources are excluded. Cache uses retained-payload formulas; runtime policy may reserve more. No speed or quality prediction.

Technical notes, examples & sources

The calculator estimates Qwen text weights from 27B × ideal bits, full-attention KV block payload, and recurrent matrices. Enter artifact and measured extra allocations for a better budget. It excludes unentered convolution, vision, workspace, allocator, and draft resources. A total below capacity is a candidate, not a fit guarantee. Change one factor, record results, then retest promising combinations.

Primary sources · checked 2026-11-13

Act 5 · Configure, measure, tune25 / 34

Benchmark the experience

Fast short answers do not prove reliable long-context agents.

Benchmark the experience Fast short answers do not prove reliable long-context agents. Cold load Then Uncached / reused prompt Then Decode Then Complete agent task. Cold loadUncached / reused promptDecodeComplete agent task
  1. 1Cold load
  2. 2Uncached / reused prompt
  3. 3Decode
  4. 4Complete agent task
Schematic · steps are not measured inference time

Separate cold load, uncached prefill, cached-prefix reuse, decode, and complete task duration. Use matched prompts, output lengths, sampling settings, exact artifacts, runtime versions, and speculation settings. Repeat trials and test progressively longer occupied contexts.

Technical notes, examples & sources

Record failures as outcomes. Watch memory pressure, swapping, allocation failures, sustained cooling, and per-device headroom. A benchmark should say whether prefixes were cached and whether model loading was included. The reference template contains no measured values: this presentation does not publish personal benchmark logs or claim local experiments succeeded.

Benchmark the experience
MeasureRecord
IdentityArtifact revision/hash; engine and harness versions
RequestPrompt/token count, sampling, output cap, cache state
RuntimeContext, batch/microbatch, KV type, placement, draft settings
TimingLoad, user-visible TTFT, prefill, decode, total task
ReliabilityTrial count, failures, peak memory, pressure, cooling
QualityTask success criteria, tests passed, manual review

Primary sources · checked 2026-11-13

Act 6 · From chat to agents26 / 34

From a reply to an agent loop

The harness acts, observes the result, and asks the model again.

From a reply to an agent loop The harness acts, observes the result, and asks the model again. Model decision Then Tool request Then Harness execution Then Tool result. Model decisionTool requestHarness executionTool result
  1. 1Model decision
  2. 2Tool request
  3. 3Harness execution
  4. 4Tool result
Schematic · steps are not measured inference time

A familiar chat request can expand into a repeated cycle. The model emits a structured tool request. The harness checks permission, runs the tool, returns its result, and asks what to do next. Chat products can also use tools; product labels do not define a clean boundary.

Technical notes, examples & sources

Tool execution is not a hidden ability of the weight file. The harness supplies tool definitions and decides how to execute them. A malformed tool call, denied permission, or failed shell command should become an observation, not an invented success.

Primary sources · checked 2026-11-13

Act 6 · From chat to agents27 / 34

What the harness provides

Model capability and harness behavior are separate.

What the harness provides Model capability and harness behavior are separate. Instructions + files Then Tool schemas Then Permission checks Then Stopping conditions. Instructions + filesTool schemasPermission checksStopping conditions
  1. 1Instructions + files
  2. 2Tool schemas
  3. 3Permission checks
  4. 4Stopping conditions
Schematic · steps are not measured inference time

System instructions, tools, permissions, repository files, shell execution, search, task state, and stopping conditions shape the outcome. A capable model needs a compatible chat template and structured tool-call handling. A good harness also manages errors and keeps a useful record of progress.

Technical notes, examples & sources

Permissions belong around tool execution. Search may send data externally. Choose a small task with an observable result before granting broader file or shell access. Define done in terms of the artifact and checks, not the model saying it is done.

Primary sources · checked 2026-11-13

Act 6 · From chat to agents28 / 34

Ollama, OpenCode, and VS Code

The editor, harness, API, and inference server have distinct roles.

Ollama, OpenCode, and VS Code The editor, harness, API, and inference server have distinct roles. Coding interface Then Harness + permissions Then Ollama API Then Loaded model. Coding interfaceHarness + permissionsOllama APILoaded model
  1. 1Coding interface
  2. 2Harness + permissions
  3. 3Ollama API
  4. 4Loaded model
Schematic · steps are not measured inference time

OpenCode sends a model request to an Ollama endpoint, parses streamed responses and tool calls, and runs approved tools. In VS Code, an extension supplies the model or agent integration. The editor itself is not an inference engine. Confirm the exact model alias and effective context/output limits on both sides.

Technical notes, examples & sources

The disclosure targets OpenCode V2 documentation checked November 13, 2026. V1 uses a different config shape. Native Ollama options such as num_ctx belong to its API/Modelfile; an OpenAI-compatible request uses fields such as max_tokens. A harness limit is planning metadata, not a server allocation request. Test one harmless tool call and streaming before a long agent task.

A separate agent allocation

After progressively testing that 65,536 tokens and an 8,192-token output allowance fit your workload, create a distinct alias. These values are an example, not a hardware recommendation.

FROM local-coder
PARAMETER num_ctx 65536
PARAMETER num_predict 8192

Save the preceding file as Modelfile.agent, then run:

ollama create local-coder-agent -f Modelfile.agent
npm install -g @opencode/cli
opencode --version
# Confirm OpenCode 2.x before using the V2 configuration below.

OpenCode V2 · opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "model": "ollama/local-coder-agent",
  "providers": {
    "ollama": {
      "settings": { "baseURL": "http://localhost:11434/v1" },
      "models": {
        "local-coder-agent": {
          "name": "Local coding model",
          "limit": { "context": 65536, "output": 8192 }
        }
      }
    }
  },
  "compaction": {
    "auto": true,
    "keep": { "tokens": 8000 },
    "buffer": "20%"
  }
}

Run opencode from a generic project directory. Select the alias with /models. Ask it to list a small repository’s files before allowing edits. Inspect the tool request, permission decision, returned result, and subsequent response. Check streaming and cancellation. The harness declarations do not allocate server context.

OpenCode V1 uses provider / npm / options; V2 uses providers / package / settings. Keep examples within one version. A custom V2 compatible provider uses @opencode/ai/providers/openai-compatible. Declaring tool capability cannot make a model follow a tool schema.

VS Code with the official Ollama extension

  1. Use VS Code 1.127 or newer and a running Ollama server.
  2. Install the Ollama extension linked in the official guide below.
  3. Open VS Code Chat, then choose the alias in the Ollama model picker.
  4. Use the integration’s agent mode and permissions for a small observable coding task.
  5. After changing allocation, reload VS Code and recheck effective context. The displayed model maximum may exceed runtime allocation.

If models are missing, use Ollama: Refresh Models, Ollama: Diagnose Models, and the Ollama output channel. Alternatively, run OpenCode in VS Code’s integrated terminal.

Three kinds of configuration

LayerExamples
Native Ollamanum_ctx, num_batch, num_gpu, draft_num_predict, num_predict
Compatible HTTPmessages, tools, stream, supported max_tokens
HarnessModel ID, context/output declarations, compaction, permissions

Tool calls are structured requests, not ordinary prose naming a command. The server formats them, the integration parses them, and the harness executes permitted tools. The compatible endpoint does not have a standard context-allocation field; configure the Ollama alias separately.

Primary sources · checked 2026-11-13

Act 6 · From chat to agents29 / 34

Why agents consume context

Every observation can become part of the next request.

Why agents consume context Every observation can become part of the next request. Instructions + tools Then Messages + files Then Tool output Then Response reserve. Instructions + toolsMessages + filesTool outputResponse reserve
  1. 1Instructions + tools
  2. 2Messages + files
  3. 3Tool output
  4. 4Response reserve
Schematic · steps are not measured inference time

Instructions, tool schemas, dialogue, source files, command output, and sometimes reasoning occupy the supplied context. Tool loops add information even when the user writes nothing new. Reserve room for the next response and future observations. Allocation and actual token occupancy are distinct.

Technical notes, examples & sources

Use the model’s real tokenizer and rendered chat template to count the complete request. Some reasoning is returned or replayed and some is not; behavior depends on model and harness. Limit noisy command output, select relevant files, and leave enough output allowance for useful continuation.

Primary sources · checked 2026-11-13

Act 6 · From chat to agents30 / 34

Compaction preserves task state

A shorter history can carry the goal, decisions, and unfinished work.

Compaction preserves task state A shorter history can carry the goal, decisions, and unfinished work. Goal + constraints Then Decisions + files Then Unresolved issues Then Shorter summary. Goal + constraintsDecisions + filesUnresolved issuesShorter summary
  1. 1Goal + constraints
  2. 2Decisions + files
  3. 3Unresolved issues
  4. 4Shorter summary
Schematic · steps are not measured inference time

Compaction replaces some conversation history with a shorter representation. A useful summary preserves the goal, constraints, decisions, changed files, checks already run, and unresolved issues. It can omit details or introduce errors. It does not update weights or provide perfect memory.

Technical notes, examples & sources

Keep source files and test outputs authoritative. If a summary says a command succeeded, retain the evidence or rerun an appropriate check. Compaction should be inspectable and should preserve enough provenance to continue the actual task.

Primary sources · checked 2026-11-13

Act 6 · From chat to agents31 / 34

When compaction itself runs out of room

Input capacity and summary output allowance are separate limits.

When compaction itself runs out of room Input capacity and summary output allowance are separate limits. Input fits Then Output cap reached Then Inspect the summary Then Role-specific settings. Input fitsOutput cap reachedInspect the summaryRole-specific settings
  1. 1Input fits
  2. 2Output cap reached
  3. 3Inspect the summary
  4. 4Role-specific settings
Schematic · steps are not measured inference time

A summarization request can fit its input and still stop before producing a useful summary. Inspect the actual request, output cap, finish reason, and returned summary. The same weights can serve interactive work and summarization with different role-specific settings.

Technical notes, examples & sources

Compaction summarizes history. Truncation removes content. Retrieval selects external information. Persistent memory stores information for future use. Prefix caching reuses computation without shortening the logical prompt. Raising limits repeatedly does not repair poor selection or a truncated summary.

When compaction itself runs out of room
MechanismWhat changes
CompactionReplaces history with a shorter representation
TruncationDrops content
RetrievalSelects relevant external information
Persistent memoryStores information for later use
Prefix cachingReuses computation; prompt stays logically long

Primary sources · checked 2026-11-13

Act 7 · Put the pieces together32 / 34

Use a model from another device

Inference location does not determine tool execution location.

Use a model from another device Inference location does not determine tool execution location. Laptop / phone UI Then Authenticated connection Then Inference host Then Tools stay with harness. Laptop / phone UIAuthenticated connectionInference hostTools stay with harness
  1. 1Laptop / phone UI
  2. 2Authenticated connection
  3. 3Inference host
  4. 4Tools stay with harness
Schematic · steps are not measured inference time

A fictional laptop or phone can use an inference host through a private authenticated connection. The interface, model runtime, and shell tools may run in different places. Sending a model request does not automatically transfer repository files or move shell execution to the inference host.

Technical notes, examples & sources

This is a generic example, not a description of the author’s infrastructure. Use a private authenticated tunnel or equivalent access control. Keep the inference endpoint bound to loopback behind it; do not expose an unauthenticated API as a default. Verify where each extension and tool process actually executes.

Primary sources · checked 2026-11-13

Act 7 · Put the pieces together33 / 34

A practical decision guide

Match the symptom to the part of the stack that can cause it.

A practical decision guide Match the symptom to the part of the stack that can cause it. Cannot load Then Slow first token Then Slow generation Then Agent loses track. Cannot loadSlow first tokenSlow generationAgent loses track
  1. 1Cannot load
  2. 2Slow first token
  3. 3Slow generation
  4. 4Agent loses track
Schematic · steps are not measured inference time

Start with the smallest failing workload. Separate loading, allocation, prompt processing, generation, tool execution, and context management. Inspect effective settings before changing more knobs. Keep the troubleshooting table nearby when a configuration works briefly and fails later.

Technical notes, examples & sources

A successful short smoke test is the first rung, not proof of sustained reliability. Treat each hypothesis as a controlled change with a repeatable workload and a recorded outcome.

A practical decision guide
SymptomInvestigate next
Cannot loadTotal budget, exact quant, context, concurrency
Unwanted CPU placementAllocation estimates, supported offload controls, logs
Slow first responseCold load, prefill, batch, prefix reuse
Slow generationResidency, bandwidth, speculation, request shape
Agent loses trackFile selection, tool output volume, compaction
Long tasks failActual occupancy, buffers, output caps, sustained pressure

Primary sources · checked 2026-11-13

Act 7 · Put the pieces together34 / 34

Return to the task

Choose deliberately. Make the complete workflow work.

Return to the task Choose deliberately. Make the complete workflow work. Capable model Then Working set fits Then Verified runtime Then Useful completed task. Capable modelWorking set fitsVerified runtimeUseful completed task
  1. 1Capable model
  2. 2Working set fits
  3. 3Verified runtime
  4. 4Useful completed task
Schematic · steps are not measured inference time

Choose a capable model. Make the working set fit. Verify placement. Measure real tasks. Tune the runtime. Configure the harness. Manage context deliberately. The outcome can be local, cloud, or hybrid AI—an informed choice that helps complete the original task.

Technical notes, examples & sources

Return to the same coding task from the opening. Invite the audience to name one setting they would verify, one measurement they would record, and one context-management change they would make. The reading view retains formulas, configuration examples, glossary, sources, and troubleshooting.

Primary sources · checked 2026-11-13

Keep as a reference

The essential vocabulary

Parameter
A learned numerical value in the model.
Token
An element of the encoded input or output sequence; not necessarily a word.
Inference
Using learned weights to produce model output.
Attention
A computation combining value representations using query–key compatibility.
KV cache
Retained keys and values used to avoid recomputing earlier representations.
Recurrent state
A retained state updated by recurrent layers.
Prefill
Processing known input tokens before or between generation steps.
Decode
Generating dependent output tokens one step at a time.
TTFT
User-visible time from request submission to first output token.
Quantization
Representing values with fewer levels, often reducing storage.
GB / GiB
Decimal gigabytes (10⁹ bytes) / binary gibibytes (2³⁰ bytes).
Batch / microbatch
Logical tokens submitted / physical tokens executed at once.
Speculative decoding
Drafting continuations and verifying them with a target model.
MTP
Multi-token prediction; trained machinery that can support drafting.
Residency / offload
Where model data and layers are placed for execution.
Harness
Software managing model calls, tools, permissions, state, and stopping.
Compaction
Replacing part of the history with a shorter representation.
Prefix caching
Reusing computation for an unchanged prefix without shortening its logical length.

Troubleshooting guide · Benchmark checklist · Memory calculator · Ollama baseline

What the evidence means

Diagrams and playback are schematic. Their timing is not inference speed. Memory figures are calculated retained payloads with stated exclusions, not measured allocations. Unsloth accuracy results are publisher reports. This guide contains no independent throughput measurements or claims of completed inference experiments.

Model configurations are pinned to upstream revisions. Commands and schemas were checked against Ollama v0.40.16 and OpenCode V2 documentation on November 13, 2026. Recheck after upgrades. A successful download, model load, or context allocation is not proof that a realistic long task will finish.