<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:blog="https://jonesrussell.github.io/blog/ns"><channel><title>Concurrency on Web Developer Blog</title><link>https://jonesrussell.github.io/blog/tags/concurrency/</link><description>Recent content in Concurrency on Web Developer Blog</description><image><title>Web Developer Blog</title><url>https://jonesrussell.github.io/blog/images/og-default.png</url><link>https://jonesrussell.github.io/blog/images/og-default.png</link></image><generator>Hugo -- 0.165.0</generator><language>en-us</language><lastBuildDate>Mon, 31 Aug 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://jonesrussell.github.io/blog/tags/concurrency/feed.xml" rel="self" type="application/rss+xml"/><item><title>Idempotent, budget-bounded AI queries on a single SQLite writer</title><link>https://jonesrussell.github.io/blog/idempotent-budget-bounded-queries-sqlite/</link><pubDate>Mon, 31 Aug 2026 00:00:00 +0000</pubDate><guid>https://jonesrussell.github.io/blog/idempotent-budget-bounded-queries-sqlite/</guid><category>go</category><blog:tag>go</blog:tag><blog:tag>sqlite</blog:tag><blog:tag>concurrency</blog:tag><blog:tag>api-design</blog:tag><description>How northway&amp;rsquo;s SQLite store keeps paid AI-provider queries idempotent and budget-bounded, using a single writer, lease-based claims, and a reserve-then-settle spend model.</description><content:encoded><![CDATA[<p>Ahnii!</p>
<p><a href="https://github.com/jonesrussell/northway">northway</a> is a Go service that turns approved sources into small, ranked, source-backed news feeds for AI agents — one process, embedded SQLite, deployed Pi-first. Every feed query that reaches an AI provider costs real money, and a network retry or a crashed request must never turn into a second charge for the same query. <code>internal/sqlite</code> is what makes that guarantee hold: one writer, scoped credentials, a reserve-then-settle budget, and an idempotent claim lifecycle for the provider call itself.</p>
<h2 id="one-writer-strictly-locked">One writer, strictly locked</h2>
<p><code>Store</code> owns a single SQLite file and refuses to share it. <code>Open</code> takes an <code>flock</code>-style exclusive lock on the database file and requires the directory to be private:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">if</span> !<span style="color:#a6e22e">info</span>.<span style="color:#a6e22e">IsDir</span>() <span style="color:#f92672">||</span> <span style="color:#a6e22e">info</span>.<span style="color:#a6e22e">Mode</span>().<span style="color:#a6e22e">Perm</span>()<span style="color:#f92672">&amp;</span><span style="color:#ae81ff">0077</span> <span style="color:#f92672">!=</span> <span style="color:#ae81ff">0</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#66d9ef">nil</span>, <span style="color:#e6db74">&#34;&#34;</span>, <span style="color:#a6e22e">errors</span>.<span style="color:#a6e22e">New</span>(<span style="color:#e6db74">&#34;database directory must be private (0700) and not a symlink&#34;</span>)
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Writes go through one <code>*sql.DB</code> with <code>_txlock=immediate</code>, and a buffered channel (<code>writeGate</code>) serializes callers before any transaction opens:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">func</span> (<span style="color:#a6e22e">s</span> <span style="color:#f92672">*</span><span style="color:#a6e22e">Store</span>) <span style="color:#a6e22e">write</span>(<span style="color:#a6e22e">ctx</span> <span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">Context</span>, <span style="color:#a6e22e">fn</span> <span style="color:#66d9ef">func</span>(<span style="color:#f92672">*</span><span style="color:#a6e22e">sqlc</span>.<span style="color:#a6e22e">Queries</span>) <span style="color:#66d9ef">error</span>) <span style="color:#66d9ef">error</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">select</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">case</span> <span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">writeGate</span> <span style="color:#f92672">&lt;-</span> <span style="color:#66d9ef">struct</span>{}{}:
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">case</span> <span style="color:#f92672">&lt;-</span><span style="color:#a6e22e">ctx</span>.<span style="color:#a6e22e">Done</span>():
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">return</span> <span style="color:#a6e22e">ctx</span>.<span style="color:#a6e22e">Err</span>()
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">defer</span> <span style="color:#66d9ef">func</span>() { <span style="color:#f92672">&lt;-</span><span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">writeGate</span> }()
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">tx</span>, <span style="color:#a6e22e">err</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">writer</span>.<span style="color:#a6e22e">BeginTx</span>(<span style="color:#a6e22e">ctx</span>, <span style="color:#66d9ef">nil</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Reads use a separate read-only pool (<code>query_only(1)</code>) with up to two connections, so lookups never queue behind the write gate. WAL mode, <code>foreign_keys(1)</code>, and <code>synchronous(FULL)</code> are all asserted — not just set — by <code>Ready()</code>, which fails startup if the file&rsquo;s actual pragmas don&rsquo;t match what the binary expects.</p>
<h2 id="scoped-keys-one-cross-tenant-lookup">Scoped keys, one cross-tenant lookup</h2>
<p>Every credential is tenant-scoped. <code>identity.go</code> stores a key&rsquo;s SHA-256 digest, never the key itself, and validates the full record shape before writing:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">if</span> <span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">TenantID</span> <span style="color:#f92672">!=</span> <span style="color:#a6e22e">tenant</span> <span style="color:#f92672">||</span> !<span style="color:#a6e22e">identity</span>.<span style="color:#a6e22e">ValidKeyID</span>(<span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">ID</span>) <span style="color:#f92672">||</span> !<span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">Scopes</span>.<span style="color:#a6e22e">Valid</span>() <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>    !<span style="color:#a6e22e">validTimestamp</span>(<span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">CreatedAt</span>) <span style="color:#f92672">||</span> <span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">Digest</span> <span style="color:#f92672">==</span> [<span style="color:#ae81ff">32</span>]<span style="color:#66d9ef">byte</span>{} <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">LastUsedAt</span> <span style="color:#f92672">!=</span> <span style="color:#66d9ef">nil</span> <span style="color:#f92672">||</span> <span style="color:#a6e22e">key</span>.<span style="color:#a6e22e">RevokedAt</span> <span style="color:#f92672">!=</span> <span style="color:#66d9ef">nil</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#a6e22e">errors</span>.<span style="color:#a6e22e">New</span>(<span style="color:#e6db74">&#34;invalid key metadata&#34;</span>)
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p><code>LookupAPIKey</code> is deliberately the <strong>only</strong> function in the package that resolves a key without an existing tenant scope — its own comment spells out the boundary: it &ldquo;exposes no corpus data and is consumed only by identity.Service, never a public lookup endpoint.&rdquo; Everything downstream of that lookup — <code>TouchAPIKey</code>, <code>RevokeAPIKey</code>, every query and mutation — takes a <code>tenant</code> and checks access before touching a row.</p>
<h2 id="reserve-first-settle-later">Reserve first, settle later</h2>
<p>Budgets are tracked in micros (<code>LimitMicros</code>, <code>SpentMicros</code>, <code>HeldMicros</code>) and set through an operator-only call:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">func</span> (<span style="color:#a6e22e">s</span> <span style="color:#f92672">*</span><span style="color:#a6e22e">Store</span>) <span style="color:#a6e22e">SetBudget</span>(<span style="color:#a6e22e">ctx</span> <span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">Context</span>, <span style="color:#a6e22e">principal</span> <span style="color:#a6e22e">identity</span>.<span style="color:#a6e22e">Principal</span>, <span style="color:#a6e22e">limitMicros</span> <span style="color:#66d9ef">int64</span>) <span style="color:#66d9ef">error</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">tenant</span>, <span style="color:#a6e22e">err</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">principal</span>.<span style="color:#a6e22e">RequireOperator</span>()
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The spend itself never just decrements a counter after the fact. A query first <strong>reserves</strong> a worst-case amount; only once the real cost is known does it <strong>settle</strong> the difference back. That two-step is what makes an in-flight AI call safe to retry, fail, or recover from a crash without double-billing.</p>
<h2 id="the-query-claim-lifecycle">The query claim lifecycle</h2>
<p><code>BeginQuery</code> is the entry point, and it&rsquo;s built to be replay-safe. It hashes the caller&rsquo;s idempotency key together with the endpoint (<code>sha256(&quot;POST /v1/feed-queries\x00&quot; + key)</code>) and looks for existing work under that hash:</p>
<ul>
<li><strong>Cache hit</strong> (same feed/corpus/entitlement/ranker revision as an existing snapshot) → returns the cached result, no budget touched.</li>
<li><strong>Existing work found, same request digest</strong> → returns the in-progress or completed claim; a second call with the same key can never start a second paid attempt.</li>
<li><strong>Existing work found, different request digest</strong> → <code>ErrConflict</code>. Reusing an idempotency key for a different request is rejected outright.</li>
<li><strong>No existing work, budget configured</strong> → reserves <code>policy.WorstCaseMicros</code> and returns a <code>WorkID</code> the caller can use to actually invoke the provider.</li>
<li><strong>No existing work, no budget</strong> → still creates work, but <code>ProviderAllowed</code> is false — the caller is limited to deterministic (non-AI) results.</li>
</ul>
<p>The function&rsquo;s own comment is blunt about the safety property: &ldquo;A replay never returns WorkID, so it cannot authorize another call.&rdquo;</p>
<p>From there, three methods carry a claim through its lifecycle:</p>
<table>
	<thead>
			<tr>
					<th>Call</th>
					<th>Precondition</th>
					<th>Effect</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>StartProvider</code></td>
					<td>work is <code>pending</code>, spend is <code>reserved</code>, lease not expired, scope unchanged</td>
					<td>flips spend to <code>started</code> — must commit <em>before</em> the provider is actually called</td>
			</tr>
			<tr>
					<td><code>CompleteQuery</code></td>
					<td>work is <code>pending</code>, lease valid, feed/entitlement revision unchanged</td>
					<td>validates every selected article against current storage, settles spend, writes an immutable snapshot</td>
			</tr>
			<tr>
					<td><code>FailQuery</code></td>
					<td>work is <code>pending</code></td>
					<td>settles the hold as <code>uncertain</code> if the provider had started, marks work <code>failed</code></td>
			</tr>
	</tbody>
</table>
<p><code>StartProvider</code>&rsquo;s doc comment states the ordering guarantee directly: &ldquo;An error/ambiguous commit never authorizes a call&hellip; No provider code runs inside storage.&rdquo; The database transaction is the gate, not the HTTP call to the provider.</p>
<p><code>CompleteQuery</code> re-checks the feed&rsquo;s revision and each selected article&rsquo;s content hash before it will settle spend or write a snapshot — an unrelated ingest arriving mid-request can&rsquo;t get charged for or attached to a query that started against older data.</p>
<h2 id="recovering-from-crashes-without-guessing">Recovering from crashes without guessing</h2>
<p>Every claim carries a <code>LeaseUntil</code>. If a process dies mid-call, the spend is left in <code>reserved</code> or <code>started</code>, not silently lost or silently spent. <code>RecoverQueries</code> sweeps expired leases and fails them the same way <code>FailQuery</code> would — its comment notes it processes bounded batches and &ldquo;preserves uncertain holds,&rdquo; since an operator later needs to reconcile those, not assume they were free.</p>
<p>That reconciliation is <code>ReconcileQuery</code>, and it requires actual evidence, not elapsed time, before it will move a hold out of <code>uncertain</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">if</span> <span style="color:#a6e22e">w</span>.<span style="color:#a6e22e">SpendState</span> <span style="color:#f92672">==</span> <span style="color:#e6db74">&#34;settled&#34;</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> <span style="color:#a6e22e">w</span>.<span style="color:#a6e22e">ActualMicros</span>.<span style="color:#a6e22e">Int64</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">actualMicros</span> {
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">return</span> <span style="color:#66d9ef">nil</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#a6e22e">query</span>.<span style="color:#a6e22e">ErrConflict</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> <span style="color:#a6e22e">w</span>.<span style="color:#a6e22e">SpendState</span> <span style="color:#f92672">!=</span> <span style="color:#e6db74">&#34;uncertain&#34;</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#a6e22e">query</span>.<span style="color:#a6e22e">ErrConflict</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Repeating the same settlement is a safe no-op; a contradictory number is rejected outright rather than silently overwritten.</p>
<h2 id="timestamps-get-the-same-suspicion">Timestamps get the same suspicion</h2>
<p>Money isn&rsquo;t the only thing storage refuses to trust blindly. <code>timestamps_test.go</code> asserts that out-of-range values — before the epoch, past <code>9999-12-31</code>, or offsets that push a boundary value across it — are rejected on write and never partially applied:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#e6db74">&#34;positive wrap&#34;</span>:    <span style="color:#a6e22e">time</span>.<span style="color:#a6e22e">Date</span>(<span style="color:#ae81ff">600000</span>, <span style="color:#ae81ff">1</span>, <span style="color:#ae81ff">1</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">time</span>.<span style="color:#a6e22e">UTC</span>),
</span></span><span style="display:flex;"><span><span style="color:#e6db74">&#34;offset past end&#34;</span>:  <span style="color:#a6e22e">time</span>.<span style="color:#a6e22e">Date</span>(<span style="color:#ae81ff">9999</span>, <span style="color:#ae81ff">12</span>, <span style="color:#ae81ff">31</span>, <span style="color:#ae81ff">23</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">time</span>.<span style="color:#a6e22e">FixedZone</span>(<span style="color:#e6db74">&#34;west&#34;</span>, <span style="color:#f92672">-</span><span style="color:#ae81ff">3600</span>)),
</span></span></code></pre></div><p>A rejected write leaves the row, its version history, and its full-text index completely unchanged — the test checks the version count and search index directly, not just the returned error.</p>
<h2 id="why-this-shape">Why this shape</h2>
<p>None of this is exotic SQLite usage — WAL mode, a single writer, a lease column. What makes it hold together is that every step that touches money or identity states its own precondition in code, not just in a comment: wrong state, expired lease, mismatched revision, or a reused key against a different request all fail the transaction outright. For a Raspberry Pi-deployed, single-process service fronting paid AI calls, that&rsquo;s the difference between &ldquo;retry-safe&rdquo; as a claim and as a property you can point to in the diff.</p>
<p>Baamaapii</p>
]]></content:encoded></item></channel></rss>