Local Intelligence
Why I Kept the Intelligence Close to the Application
The default pattern in early AI application development was sending every single task to a cloud API endpoint. While cloud APIs are convenient for initial prototypes, relying entirely on remote endpoints introduces network latency, recurring per-token billing, network fragility, and data privacy concerns.
Across three different projects—Conclave, Phoenix, and Check-Mate-Analyse—I explored keeping compute, vector indexing, and intelligence processing as close to the application boundary as possible. However, 'local intelligence' means very different things depending on your system constraints.
1 / Local Model Inference in Conclave
In Conclave, local intelligence meant running open-weights LLMs directly on local host hardware using Ollama instances.
Why local model inference mattered in Conclave:
- Zero Per-Token Costs: Running multi-agent coordination loops locally allowed executing thousands of iterative prompts without accumulating cloud API bills.
- Offline Resilience: Multi-agent task execution could continue functioning without an active internet connection.
- Privacy Guarantees: Sensitive context data remained on local hardware rather than being transmitted to third-party endpoints.
The trade-off, of course, was hardware dependency: local throughput depends directly on available GPU and RAM, requiring strict token budget limits and model quantization.
2 / Local Vector Indexing in Phoenix
In Phoenix, relying on third-party SaaS vector databases meant index updates were subject to network latency and remote rate quotas. Moving vector storage to local PostgreSQL instances with `pgvector` enabled transactional vector updates alongside relational document metadata.
Benefits of local vector infrastructure:
- Deterministic Testing: Local vector indices can be spun up in Docker containers during automated CI integration tests.
- Reduced Network Latency: Cosine similarity queries execute in sub-millisecond local database memory space rather than over WAN connections.
- No Cloud Lock-In: Vector schema and index configurations remain fully portable.
3 / Browser-Side Compute in Check-Mate-Analyse
In CheckMate, running chess engine evaluations on a backend server would have meant severe server costs under high concurrent user load. Compiling Stockfish 16 into WebAssembly allowed the engine to run entirely within the user's browser thread.
Benefits of browser-side compute:
- Zero Backend Compute Costs: Tens of thousands of position evaluations execute directly on client CPU threads.
- Instant Feedback: Board evaluation updates instantly as user pieces move, bypassing server round-trip latency.
4 / Evaluating Trade-offs: Local vs Remote Compute
Local intelligence isn't a silver bullet. It introduces clear engineering trade-offs:
- Hardware Limits: Client or local server memory and GPU limits constrain model parameter sizes.
- Distribution Size: Shipping WebAssembly binaries or local model weights increases initial download payloads.
- Maintenance Overhead: Managing local Docker containers, local vector indices, and runtime environments requires explicit operational tooling.
Architectural takeaway: Local-first intelligence empowers developers to build resilient, privacy-preserving applications where core functionality remains available regardless of remote network conditions.