Answer accuracy
Product ambition for defined private-knowledge tasks; above the current ≥80% release floor.
MEASURE WHAT MATTERS
Evidence before
every claim.
Quality, latency, context efficiency, and control. Compared under explicit conditions, with the costs in view.
Targets & scoped internal measurements · Pre-release01 / PERFORMANCE OBJECTIVES
A clear bar for useful intelligence.
Expected performance is shown as a target. Targets are not measured results or a promise of future performance.
Citation support
Release target on the frozen knowledge benchmark.
Unanswerable abstention
Release target when evidence is insufficient.
Fewer input tokens
Context research target against the strongest eligible baseline, with quality and latency constraints.
02 / MEASURED SYSTEMS STUDY46.43% lower
46.43% lower
overall p95 latency.
Guarded compiled execution versus Cavumo’s fixed configuration. Same Gemma model. Identical generated outputs across all 1,020 pairs.
M2 · 8 GiB2,040 generationsHeadless development profile
Read the complete study CAVUMO / COMPILER
SCOPED STUDY / p95 LATENCY
Fixed4,983 ms
Compiled2,669 ms
Same model. Identical generated outputs.
01Profile + identity
02Permission check
03Bounded plan
04Local execution
Study conditions, measured quality & limitations
Three paired repetitions of 340 original synthetic queries. Overall p95: 4,983.29 → 2,669.39 ms; paired bootstrap interval for reduction: 32.36%–62.51%. Cold-runtime improvement: 7.41%; warm: 65.92%; memory: 12.44%; shared context: 9.85%. Cold and shared intervals include zero.
Exact synthetic answer correctness: 64.12% in both configurations; unanswerable abstention: 96.10%. Generated citation support was not measured. Short-answer fixtures and related repeated queries do not establish independent customer-domain quality. Production artifacts and subsequent source changes need a fresh run.
This is an internal configuration comparison, not a frontier-model, Ollama, or cloud API benchmark. The compiler passed its scoped gate; the complete desktop release gates have not passed.
Download measured study JSONPROTOCOL / RESEARCH & RELEASE GATESA fair comparison starts
A fair comparison starts
before the first run.
Freeze datasets, baselines, metrics, and thresholds before evaluating a candidate. Research failures lead to documented iteration, not weaker retrospective gates.
Held-out by design
At least 300 held-out queries across two distinct corpora. Separate memory and cross-agent scenarios. Split by collection or task family.
Same conditions
Pin model, artifact, hardware, decoding, and UI profile. Include cold and warm runs, repeated measurements, and specialist overhead.
Isolation first
Reset evaluation state between runs. Keep held-out examples out of training and persistent agent memory used during evaluation.
Report the trade-offs
Publish uncertainty, quality regressions, p95 latency, resource use, compiler ablations, and specialist ablations.
Research gates TARGETS / NOT RESULTS
| Component | Passing target | Quality / latency constraint | Result |
|---|---|---|---|
| Compiler | ≥15% lower p95 latency | ≤2 percentage points quality loss | Scoped compiler study passed; production retest pending |
| Context | ≥20% fewer input tokens | ≤2 pp quality loss; ≤5% p95 regression | Candidate not promoted; see research notebook |
| Verify / Adapt | Beat a frozen task-specific heuristic | Report calibration and error costs | Operational baselines; learned models not trained yet |
Desktop release gates
≥80%Answer correctness target≥95%Citation support target≥90%Abstention target on unanswerable questions
These are targets for a frozen knowledge benchmark, alongside privacy, security, recovery, accessibility, and resource tests. They are not Cavumo performance results.
COMPARISON / PLANNED STUDYDifferent systems.
Different systems.
Explicit trade-offs.
Local models and hosted frontier models serve different needs. Task fit, total cost, and operational responsibility matter more than a single leaderboard.
| Dimension | Cavumo local MVP / direction | API-based model setup |
|---|---|---|
| Execution | Customer-controlled device or server | Provider-hosted inference |
| Data path | Local context processing; no automatic cloud fallback | Prompts and context sent to the selected provider; policies vary |
| Costs | Hardware, energy, maintenance, and deployment costs | Usage or contracted charges, plus integration costs |
| Capability | Task-specific quality must be validated | Model capability and service behavior vary by provider |
| Context & memory | Scoped context and approved durable memories | Application-managed or provider features; policies vary |
| Offline use | Working local inference after required artifacts are installed | Normally requires a connection to the provider |
| Operations | Customer hardware and runtime remain a responsibility | Provider manages inference infrastructure |
| Measured quality / speed | Internal compiler study only; no head-to-head result | No API baselines run by Cavumo yet |
Internal product tools
Use the local endpoint with permitted company knowledge. Value to validate: fewer external API calls for suitable recurring tasks.
Project & operations teams
Reuse owner-approved decisions across authorized agents. Value to validate: less repeated context assembly and clearer provenance.
Specialized business tasks
Fit a separate adapter on explicitly approved data. Value to validate: better task fit within local resource budgets; behavior and activation gates remain unfinished.
Future API comparisons will record exact provider/model versions, observation conditions, dataset permission, token accounting, and price assumptions. Private customer data is not an automatic benchmark export.
WORKSHEET / ILLUSTRATIVE INPUTSPut your assumptions
Put your assumptions
on the table.
A transparent arithmetic worksheet, not a benchmark or pricing quote. Enter your own rates. Defaults are hypothetical and do not represent a provider or a measured Cavumo workload.
MONTHLY ESTIMATES / USD
API token charges$12.00
Local cost allocation$63.00
API = requests × (input tokens × input rate + output tokens × output rate) ÷ 1,000,000.
Local = hardware allocation + electricity + operations.
Capacity, throughput, quality, idle power, provider minimums, taxes, network charges, and other costs are not modeled. Local hardware must actually support the workload; these numbers do not establish feasibility or savings.