Swap published columns for 24 entries from random.md
Some checks failed
Deploy to S3 / deploy (push) Failing after 1m24s
Some checks failed
Deploy to S3 / deploy (push) Failing after 1m24s
Retract the seven published columns into drafts/ and promote the 24 per-date entries split out of Notes/random.md. Bodies are verbatim; titles, subjects and slugs are new. Make publication a matter of location. Notes/Notes.11tydata.js gives anything under Notes/ permalink: false unless tagged `column`; drafts/drafts.11tydata.js renders drafts at their real permalink under `yarn dev` and excludes them from collections entirely in a build, so no template can surface one. Publishing is now `git mv drafts/<file> columns/`, and moving it back retracts it. Untagged notes were previously rendered into _site/ and deployed — including Notes/random.md, which was live at /Notes/random/.
This commit is contained in:
13
drafts/2026-04-19-happy-idiots.md
Normal file
13
drafts/2026-04-19-happy-idiots.md
Normal file
@@ -0,0 +1,13 @@
|
||||
---
|
||||
title: Happy Idiots
|
||||
date: 2026-04-19
|
||||
layout: column.njk
|
||||
tags: [column, society]
|
||||
permalink: /columns/2026/happy-idiots/
|
||||
---
|
||||
|
||||
On the billboards and TV ads, they'd call it the "Auto Mile." It was a strip of state highway, not far from where I lived, sandwiched by two enormous parking lots. Despite being anchored by dealerships for several major American car brands, as a local landmark, it felt pretty weak. It wouldn't even add a click to your odomoter if you drove the length of it.
|
||||
|
||||
But if you turned around instead and drove a couple (proper) miles to yet another strip mall, you'd find all of the churches. Arguably, an even larger, more impressive assembly of major American brands (and even a few imports from Europe.) There was the Catholic church, and the Methodist church. There was the First Baptist church, then the (other) Baptist church, and then the (other) Catholic church. There was the Episcopal church, the Catholic convent (Sisters of Jesus and Mary), and then Sacred Heart (the other, other Catholic Church.) There was the American Legion (with Sunday Services), and something called Victory Bible, which I initially mistook for a boxing gym. Reviewers online praised its "ample parking lot."
|
||||
|
||||
If you then decided to turn back again towards the car dealerships, you'd soon encounter yet another enormous parking lot situated at the literal cross-roads of these two highways. This one surrounding a sprawling outdoor shopping village. Six lanes of traffic delivered visitors by the thousands. So as I came to see it, being raised staunchly Catholic-adjacent, God, gas prices and credit card interest had to have been the Holy Trinity. And by not being beholden to such spirits, I was about as far from grace as you could get down by exit 38.
|
||||
9
drafts/2026-05-14-thrilling.md
Normal file
9
drafts/2026-05-14-thrilling.md
Normal file
@@ -0,0 +1,9 @@
|
||||
---
|
||||
title: 12-Watt Bulbs
|
||||
date: 2026-05-14
|
||||
layout: column.njk
|
||||
tags: [column, society]
|
||||
permalink: /columns/2026/thrilling/
|
||||
---
|
||||
|
||||
Was I late? Was the party over? It seemed like everyone in the know had already gone off to launch some new venture I wasn't involved in. Raptured while the rest of us left behind shuffled around and shared second-hand stories about how great things used to be, so we'd heard. Meanwhile the excitement seemed to have long since faded. But what do I know, really? Maybe this has always been the case. Everywhere you look there are pockets of nostalgics lamenting the way things used to be (whether they had been there or not). The excited people had maybe just gone to lunch, doing exciting things over their exciting salads. So it's all a bit ridiculous. If we find ourselves hanging around dullards, and choosing to stay, then the lack of luminance in our jaded lives might have less to do with their nostalgia or cynicism. If we can't muster our own excitement, would we even know where to find it? Would we even try?
|
||||
104
drafts/2026-06-23-hamming-it-up.md
Normal file
104
drafts/2026-06-23-hamming-it-up.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: Hamming It Up
|
||||
date: 2026-06-23
|
||||
layout: column.njk
|
||||
tags: [column, computing]
|
||||
permalink: /columns/2026/hamming/
|
||||
---
|
||||
|
||||
I started reading a chapter on error correcting codes last night. But as I skim through again today, it's pretty clear that none of it really got through to me. So let's work through an example. Couldn't hurt:
|
||||
|
||||
```
|
||||
Let's run the (7,4) Hamming code end to end
|
||||
```
|
||||
The what? Shit, I remember enough from last night to know that I should understand what I just read. Let's step back and recap:
|
||||
|
||||
```
|
||||
// basically this
|
||||
class HammingCode {
|
||||
private:
|
||||
const int codeword_len;
|
||||
const int message_len;
|
||||
|
||||
static int validate(int n, int k) {
|
||||
if (n <= 0 || k < 0 || k >= n)
|
||||
throw std::invalid_argument("0 <= k < n is a must");
|
||||
|
||||
const int r = n - k; // parity bits
|
||||
|
||||
if ((1u << r) < static_cast<unsigned>(n) + 1u)
|
||||
throw std::invalid_argument("2^(n-k) must be >= n + 1");
|
||||
|
||||
return n;
|
||||
}
|
||||
public:
|
||||
HammingCode(int n, int k): codeword_len(validate(n, k)), message_len(k) {};
|
||||
};
|
||||
|
||||
```
|
||||
|
||||
Basically the first number tells us the length of the entire codeword (check _and_ message bits). The second number (must be smaller than the first) is the length of the actual message.
|
||||
|
||||
And "must be smaller" is... weak. You can see a more rigorous constraint definition given above:
|
||||
|
||||
$$2^{n-k} \geq n + 1$$
|
||||
|
||||
Anyway, lots of preliminaries. How about the example:
|
||||
|
||||
```
|
||||
Let's encode the message **1 0 1 1**. Drop those four bits into the data positions 3, 5, 6, 7:
|
||||
|
||||
position: 1 2 3 4 5 6 7
|
||||
value: ? ? 1 ? 0 1 1
|
||||
```
|
||||
|
||||
Oh f🙂ck off... what the hell are `data positions`?
|
||||
|
||||
```
|
||||
Anything that isn't a power of two. Because those are the **check** positions (for reasons... we'll get back to that, I'm positive).
|
||||
|
||||
so 1, 2, 4, ..., etc being reserved, we get 3, 5, 7, ...,
|
||||
```
|
||||
|
||||
What's the deal with powers of two?
|
||||
|
||||
```
|
||||
They're just a single bit at some position:
|
||||
|
||||
1: 0 0 1
|
||||
2: 0 1 0
|
||||
4: 1 0 0
|
||||
...
|
||||
|
||||
The "f🙂ck you" moment comes from how the checks are defined. And I'll spoil the suprise so we can talk through the explanation next: when a single bit gets corrupted, the checks "spell out" precisely which bit flipped.
|
||||
```
|
||||
|
||||
Yeah, I've got some ques...
|
||||
|
||||
```
|
||||
So checks follow the powers of two (1, 2, 4, ...), that single bit shifting up and down. They tie together every bit in the codeword sitting at a position that has the two's bit flipped. Example:
|
||||
|
||||
Check 1 (0 0 1) - gets pos1 (001), pos3(011), pos5(101), pos7 (111)
|
||||
^
|
||||
|
||||
Where pos 1, 2, and 4 are the parity bits. By our rule, these appear in at most one check.
|
||||
```
|
||||
|
||||
_...dialtone_
|
||||
|
||||
```
|
||||
Ok, great. Yeah, so we now effectively have these flags telling us when something is wrong with their cohort of bits. We define 'wrong' like this:
|
||||
|
||||
Check 1: pos1 ⊕ pos3 ⊕ pos5 ⊕ pos7 = 0
|
||||
|
||||
The definition of XOR then implies:
|
||||
pos1 = pos3 ⊕ pos5 ⊕ pos7
|
||||
|
||||
So we know how to compute our parity bit value for each cohort. Say:
|
||||
|
||||
pos1 = 1 ⊕ 0 ⊕ 1 => 0
|
||||
|
||||
We now have all the pieces needed for our checks. And remember this critical detail: parity bits serve a single check at most (powers of two - a single bit), but all other positions will be tracked by multiple checks: wherever the check 'index' overlaps with the position. pos5 will be tracked by checks 1 and 4, since 5 (1 0 1) is made up of 1 (0 0 1) and 4 (1 0 0). That means, when position 5 is corrupted, checks 1 and 4 will fail. And, crucially, what emerges when you string together the results of every check is the position itself (1 0 1) - 5.
|
||||
```
|
||||
|
||||
This reminds me of a binary search. I don't know that it's really fair to characterize the process as such. But it appears that every failed check effectively halves the pool of possible corrupted bits (choosing between those with the nth bit set or not), before the culprit necessarily emerges from the narrowing.
|
||||
353
drafts/2026-07-04-eviction.md
Normal file
353
drafts/2026-07-04-eviction.md
Normal file
@@ -0,0 +1,353 @@
|
||||
---
|
||||
title: Eviction Notice
|
||||
date: 2026-07-04
|
||||
layout: column.njk
|
||||
tags: [column, computing]
|
||||
permalink: /columns/2026/eviction/
|
||||
---
|
||||
|
||||
## Topics to hit
|
||||
* session management in LLM chats
|
||||
* Cache mgmt strategies in general
|
||||
* securing sessions
|
||||
|
||||
---
|
||||
|
||||
## The detour: cache eviction, done properly
|
||||
|
||||
### The setup
|
||||
|
||||
A cache is a bounded fast store sitting in front of a larger, slower, or
|
||||
more-expensive backing store. Registers cache RAM; RAM caches disk; a CDN
|
||||
caches an origin; a browser caches the network. The bet is always the same:
|
||||
**most requests will hit the small fast thing, so the average access looks fast
|
||||
even though the truth lives somewhere slow.**
|
||||
|
||||
The bet only pays off if the right items are resident. And since the cache is
|
||||
bounded, admitting something new usually means throwing something out. That
|
||||
decision — *what to evict* — is the entire subject. Everything below is a
|
||||
different guess at the same question: **which resident item am I least likely to
|
||||
need soon (or cheapest to reconstruct if I'm wrong)?**
|
||||
|
||||
Hold onto the agent-session framing while you read. An LLM agent runs against a
|
||||
fixed **context window** — a few hundred thousand tokens of working memory in
|
||||
front of an effectively unbounded backing store (the full transcript, the
|
||||
filesystem, tool outputs, the codebase, the web). The moment the session
|
||||
outgrows the window, something has to leave. Context management *is* a cache
|
||||
eviction policy over tokens, whether or not anyone designed it as one. So every
|
||||
policy below is also a claim about how an agent should decide what to forget.
|
||||
|
||||
### The yardstick you can't have: Bélády's OPT
|
||||
|
||||
Bélády (1966) gave the provably optimal offline policy, usually called **OPT** or
|
||||
**MIN**: when you must evict, evict the item whose *next* use is farthest in the
|
||||
future.
|
||||
|
||||
$$\text{evict} \;=\; \arg\max_{i \,\in\, \text{cache}} \; \big(\text{next-use time of } i\big)$$
|
||||
|
||||
It's optimal for minimizing misses and completely unimplementable — it requires
|
||||
knowing the future request stream. Its job is to be the ceiling: every real
|
||||
policy is a heuristic trying to *predict* next-use distance from the past. Keep
|
||||
OPT in mind for the agent case, because there we get a strange gift the CPU
|
||||
never had — the model can partly *narrate its own future* ("I'm done with this
|
||||
file"), which is a cheap approximation of clairvoyance. More on that later.
|
||||
|
||||
### The classics, and where each one breaks
|
||||
|
||||
**FIFO** — evict the oldest-inserted item. O(1), dead simple, and it ignores
|
||||
usage entirely: a hot item gets evicted just for being old. It also suffers
|
||||
**Bélády's anomaly** — giving it *more* cache can produce *more* misses.
|
||||
FIFO is the naive "keep the last N tokens, drop the front" truncation, and it
|
||||
fails in agents the same way it fails in hardware: it will happily discard the
|
||||
original task statement or a hard constraint because they're old, not because
|
||||
they're useless.
|
||||
|
||||
**LRU** — evict the Least Recently Used item. This is the workhorse. It bets on
|
||||
**temporal locality**: recently touched ⇒ likely touched again soon. Classic
|
||||
implementation is a hash map over a doubly linked list — move-to-front on
|
||||
access, evict the tail, all O(1). LRU is a **stack algorithm**, so it's immune
|
||||
to Bélády's anomaly. Its two real weaknesses:
|
||||
|
||||
- **No scan resistance.** One sweep over a large dataset (a full table scan, a
|
||||
one-time file crawl) touches a flood of items exactly once and evicts the
|
||||
entire genuine working set to make room for garbage you'll never see again.
|
||||
- **Recency ≠ importance.** LRU assigns infinite value to a single recent touch.
|
||||
In a conversation the task statement is rarely re-quoted verbatim yet is the
|
||||
single most important thing in the window.
|
||||
|
||||
**LFU** — evict the Least Frequently Used item, by access count. Captures
|
||||
popularity, which recency misses. Its failure modes are the mirror of LRU's:
|
||||
|
||||
- **Stale popularity / cache pollution.** An item that was hot last hour keeps
|
||||
its high count forever and refuses to leave. Counts must **age** — an
|
||||
exponential decay is typical:
|
||||
$$\text{score}_i \leftarrow \text{score}_i \cdot e^{-\lambda \Delta t} + \mathbb{1}[\text{access}]$$
|
||||
- **Cold start.** A brand-new item enters at count 1 and gets evicted
|
||||
immediately, even if it's about to become the hottest thing in the cache
|
||||
(the "one-hit wonder vs. future star" problem). Naive LFU is also O(log n)
|
||||
on a heap — though there's a well-known O(1) construction with frequency
|
||||
buckets (Shah, Matani, Mitzenmacher).
|
||||
|
||||
The whole rest of the field is a **recency-vs-frequency reconciliation**, because
|
||||
neither signal dominates and the two mistakes above are opposite mistakes.
|
||||
|
||||
### Blends and adaptivity
|
||||
|
||||
**LRU-K** (O'Neil, O'Neil & Weikum, 1993) — remember the last *K* reference
|
||||
times per item and evict on the *K*-th-most-recent one (the "backward
|
||||
K-distance"). LRU-2 in particular cleanly separates items referenced twice from
|
||||
one-hit wonders, folding a little frequency into recency. Costs more bookkeeping.
|
||||
|
||||
**2Q** (Johnson & Shasha, 1994) — a cheaper LRU-2 approximation. First-timers go
|
||||
into a FIFO probation queue; only on a *second* reference do they graduate to the
|
||||
main LRU. Scans wash through the probation queue and never pollute the main list
|
||||
— scan resistance for almost free.
|
||||
|
||||
**LIRS** (Jiang & Zhang, 2002) — rank by **reuse distance** (inter-reference
|
||||
recency) rather than plain recency, which is a much better next-use predictor.
|
||||
Strongly scan-resistant; it's the ancestor of MySQL/InnoDB's buffer pool policy.
|
||||
|
||||
**CLOCK / second-chance** (Corbató, 1968) — an O(1) LRU approximation for OS page
|
||||
replacement. Pages sit in a ring, each with a reference bit; a rotating hand
|
||||
clears the bit or, if already clear, evicts. **CLOCK-Pro** (Jiang, Chen & Zhang,
|
||||
2005) does for LIRS what CLOCK does for LRU. The lesson that matters downstream:
|
||||
often you don't implement the ideal policy, you implement a cheap gadget that
|
||||
*approximates* it well enough.
|
||||
|
||||
### ARC: stop tuning, start adapting
|
||||
|
||||
**ARC — Adaptive Replacement Cache** (Megiddo & Modha, IBM, 2003) is the elegant
|
||||
one, and the closest hardware analog to what good agent context managers
|
||||
actually do. It keeps four lists:
|
||||
|
||||
- **T1** — items seen once recently (the recency half)
|
||||
- **T2** — items seen at least twice recently (the frequency half)
|
||||
- **B1, B2** — *ghost* lists: the keys **recently evicted** from T1 and T2, with
|
||||
the data thrown away but the metadata kept
|
||||
|
||||
A target parameter *p* splits capacity between T1 and T2. The trick is that ARC
|
||||
learns *p* from its own recent mistakes: a hit in ghost list **B1** means "I
|
||||
evicted a recency item I then wanted back" ⇒ grow T1; a hit in **B2** means the
|
||||
same for frequency ⇒ grow T2. It continuously re-balances recency against
|
||||
frequency with **no magic constants and no workload-specific tuning**, and it's
|
||||
scan- and burst-resistant. (It's also patented by IBM, which is the boring
|
||||
reason ZFS ships ARC while the Linux page cache doesn't — worth knowing.)
|
||||
|
||||
Two ideas from ARC get reused verbatim in the agent story below: **ghost
|
||||
entries** (remember *that* something existed and was relevant even after you drop
|
||||
its contents) and **self-tuning the recency/frequency split**.
|
||||
|
||||
### The move most people miss: admission, not just eviction
|
||||
|
||||
Every policy so far assumes the newcomer belongs in the cache and only argues
|
||||
about the victim. **TinyLFU / W-TinyLFU** (Einziger, Friedman & Manes, 2017 —
|
||||
the policy behind Java's Caffeine and Go's Ristretto) flips that:
|
||||
|
||||
$$\text{admit candidate } c \text{ over victim } v \iff \widehat{\text{freq}}(c) > \widehat{\text{freq}}(v)$$
|
||||
|
||||
On a miss, before evicting the chosen victim, estimate how popular the incoming
|
||||
item actually is and **refuse to admit it if it's less popular than what it would
|
||||
displace.** Frequencies are estimated cheaply and approximately with a
|
||||
**Count-Min Sketch** that's periodically halved (aging), so the metadata is tiny.
|
||||
W-TinyLFU fronts this with a small **window LRU** to catch fresh bursts, then
|
||||
guards a larger SLRU main region with the TinyLFU admission filter. It posts
|
||||
some of the best hit ratios known at a fraction of the metadata cost.
|
||||
|
||||
The reframing is the takeaway: **sometimes the correct action is to leave the
|
||||
newcomer out entirely rather than evict a good resident to hold it.** For an
|
||||
agent that means: not every verbose tool result has earned a place in the
|
||||
window. Admission control is arguably the highest-leverage and most-neglected
|
||||
lever in context engineering.
|
||||
|
||||
### When items aren't the same size or cost
|
||||
|
||||
CPU pages are uniform; web objects, files, and tool outputs are not. **GreedyDual-Size**
|
||||
(Cao & Irani, 1997) generalizes recency to a cost/size utility. Each object gets
|
||||
|
||||
$$H(p) = L + \frac{c(p)}{s(p)}$$
|
||||
|
||||
where *c* is fetch cost, *s* is size, and *L* is a running "inflation" clock. You
|
||||
evict the minimum-*H* object and set *L* to its *H* (so age accrues), and reset
|
||||
*H(p)* on access. The instructive part is the ratio: **value per unit of space.**
|
||||
That single idea — evict by value density, not by age — is exactly what a token
|
||||
budget forces on an agent, where a 20-token hard constraint outweighs a
|
||||
40,000-token file dump many times over.
|
||||
|
||||
### Concepts worth having names for
|
||||
|
||||
- **Temporal / spatial locality** — the empirical regularity every policy mines.
|
||||
- **Working set** (Denning, 1968) — the set of items in active use over a window;
|
||||
a cache earns its keep when it holds the working set and little else.
|
||||
- **Reuse distance / stack distance** (Mattson et al., 1970) — the analytical
|
||||
tool; the whole memory-hierarchy analysis rests on it.
|
||||
- **Scan resistance** — robustness to one-shot floods.
|
||||
- **Aging / decay** — frequency without forgetting is a slow-motion leak.
|
||||
- **Admission vs. eviction** — the two knobs; most systems only turn one.
|
||||
- **Ghost entries** — metadata about evicted items, kept to learn from misses.
|
||||
- **Stack algorithms & Bélády's anomaly** — the theory floor: some policies are
|
||||
monotone in cache size, some perversely aren't.
|
||||
|
||||
---
|
||||
|
||||
## A second cache, one level down (don't conflate these)
|
||||
|
||||
"Cache eviction in LLMs" names **two different things at two different levels**,
|
||||
and it's worth nailing the distinction before the story, because they rhyme
|
||||
loudly enough to blur:
|
||||
|
||||
1. **Context management (this column).** Message-level. *Outside* the model. You
|
||||
decide which turns, tool results, and files occupy the prompt. The unit is a
|
||||
semantic chunk; the policy is yours to write.
|
||||
|
||||
2. **KV-cache eviction.** Tensor-level. *Inside* the model. During
|
||||
autoregressive decoding the attention keys/values for every past token are
|
||||
cached so you don't recompute them; that store grows linearly with sequence
|
||||
length and becomes the memory bottleneck. Policies here decide which *tokens'*
|
||||
K/V tensors to drop:
|
||||
- **StreamingLLM** (Xiao et al., 2023) — keep a few initial "**attention
|
||||
sink**" tokens plus a sliding window; astonishingly, those first tokens
|
||||
matter far past their content.
|
||||
- **H2O** (Zhang et al., 2023) — evict tokens with low **accumulated attention**
|
||||
("heavy hitters" stay).
|
||||
- **Scissorhands** (Liu et al., 2023), **SnapKV** (Li et al., 2024),
|
||||
**FastGen** (Ge et al., 2023) — variations on importance-scored token
|
||||
eviction and per-head budgeting.
|
||||
|
||||
Same verb, different altitude. The neat part is that a single request is
|
||||
governed by eviction policies at *both* levels simultaneously — you managing
|
||||
messages, the runtime managing tensors — plus a **third** cache, the **prompt /
|
||||
prefix KV cache** (Anthropic's prompt caching): a stable prompt *prefix* is
|
||||
cached across calls, so reshuffling early context to save tokens can *invalidate*
|
||||
that cache and cost you more than it saves. Eviction decisions at one level
|
||||
perturb the economics at another. That interaction is a genuine, underexplored
|
||||
design surface.
|
||||
|
||||
## Story tying it together
|
||||
|
||||
The throughline is the oldest idea in systems: the **memory hierarchy**.
|
||||
Registers → L1 → L2 → L3 → RAM → SSD → disk → network, each tier a cache of the
|
||||
one below it, each governed by a replacement policy, the whole stack engineered
|
||||
so the fast tiers *usually* have what you need. MemGPT (Packer et al., 2023) made
|
||||
the analogy explicit for LLMs: treat the **context window as RAM**, treat
|
||||
external stores as **disk**, and let the model act as its own **memory-management
|
||||
unit** — paging information in and out. Agent context management is, quite
|
||||
literally, the memory hierarchy re-derived over semantic units. Which means
|
||||
decades of replacement-policy research is sitting right there, reusable:
|
||||
|
||||
| Cache concept | Agent-context analog |
|
||||
| --- | --- |
|
||||
| Bounded fast store | The context window (finite tokens) |
|
||||
| Backing store | Transcript on disk, files, vector DB, codebase, the web |
|
||||
| Cache line / page | A message, turn, tool result, file chunk, memory note |
|
||||
| Miss + penalty | Needed info no longer resident → re-read file, re-run tool, or **lost reasoning** (latency + tokens + sometimes unrecoverable) |
|
||||
| Hit ratio | Fraction of needed info already in the window |
|
||||
| FIFO / sliding window | Naive truncation ("keep the last N tokens") |
|
||||
| LRU | Keep recently-referenced turns |
|
||||
| LFU | Keep frequently-referenced facts |
|
||||
| Pinning | System prompt, task goal, hard constraints — non-evictable |
|
||||
| Ghost entries | Breadcrumbs to evicted content ("we discussed X, see file Y") |
|
||||
| Admission control (TinyLFU) | Don't admit the whole 40k-token file; filter verbose tool output *before* it enters the window |
|
||||
| Cost/size-aware (GreedyDual-Size) | Evict by **value per token**, not by age |
|
||||
| Adaptive split (ARC) | Shift budget between recent-turns and reference-facts on the fly |
|
||||
| Compressed cache (zswap/zram) | **Summarization / compaction** — lossy re-encode instead of drop |
|
||||
| Tiering + fetch-on-miss (MemGPT) | External memory / RAG / scratchpad files; **recall = the fetch on miss** |
|
||||
| OPT / Bélády | Evict what won't be needed before the task ends — and agents can *self-predict* this |
|
||||
|
||||
But the analogy isn't clean, and the places it *breaks* are where the
|
||||
interesting research is. Four twists that make the agent version harder — and
|
||||
more interesting — than the CPU version:
|
||||
|
||||
1. **Items are compressible.** A CPU cache line is opaque; you keep it or you
|
||||
drop it. A conversation turn can be *summarized* — replaced by a lossy, smaller
|
||||
version that still carries the gist. The only real hardware analog is the
|
||||
**compressed-memory cache** (Linux zswap/zram: compress the page before
|
||||
spilling it). Summarization/compaction is the single most-used agent technique
|
||||
and it lives in a gap the classic policies barely cover.
|
||||
|
||||
2. **Value and size vary wildly and semantically.** Pages are uniform; tokens are
|
||||
not. A one-line "never touch prod" constraint can outweigh a giant file dump.
|
||||
This is GreedyDual-Size taken to its limit: **value density is everything**,
|
||||
and value is semantic, not countable.
|
||||
|
||||
3. **Misses can be unrecoverable.** A CPU miss is just a slower fetch. Evicting a
|
||||
chain of reasoning an agent can't reconstruct isn't a re-fetch — it's *gone*.
|
||||
That raises the cost of a wrong eviction and argues hard for ghost breadcrumbs
|
||||
and pinning over silent truncation.
|
||||
|
||||
4. **The cache steers its own workload.** This is the deep one. A CPU cache is
|
||||
passive: the reference stream is fixed and the cache just tries to serve it.
|
||||
An agent's context **determines what the agent does next**, which determines
|
||||
the future access pattern — the workload is *endogenous*. The standard
|
||||
"fixed reference string" analysis doesn't even apply. Evicting the wrong thing
|
||||
doesn't just cause a miss; it can send the agent down a different path
|
||||
entirely.
|
||||
|
||||
And one twist in our favor:
|
||||
|
||||
5. **Self-predicted reuse.** OPT is unattainable for a CPU because the future is
|
||||
opaque. An agent can *tell you* when it's done with a file, or that a subtask
|
||||
is closed. That's a cheap, imperfect approximation of Bélády's clairvoyance
|
||||
from the inside — arguably the most promising lever context engineering has,
|
||||
and it has no hardware analog at all.
|
||||
|
||||
The neat story, then: **agent context management is the memory hierarchy
|
||||
reinvented over meaning instead of bytes** — same eviction question, same
|
||||
recency/frequency tension, same admission-vs-eviction and pinning-and-ghosts
|
||||
toolkit — but with items that can be *compressed* rather than dropped, *value*
|
||||
that is semantic rather than counted, *misses* that can be fatal rather than
|
||||
slow, a *workload the cache itself writes*, and a model that can *see enough of
|
||||
its own future* to approach the optimum no CPU can reach.
|
||||
|
||||
*(Third topic — securing sessions — hangs off twist #3 and the ghost/external-memory
|
||||
tier: what persists in context or spills to durable memory is an attack surface.
|
||||
Prompt-injection content that survives a compaction, or poisons the recalled
|
||||
memory store, is an eviction/admission problem wearing a security hat. Thread to
|
||||
develop later.)*
|
||||
|
||||
## References
|
||||
|
||||
**Classic cache replacement**
|
||||
- L. A. Bélády, "A study of replacement algorithms for a virtual-storage
|
||||
computer," *IBM Systems Journal*, 1966. (OPT/MIN)
|
||||
- P. J. Denning, "The working set model for program behavior," *CACM*, 1968.
|
||||
- F. J. Corbató, "A paging experiment with the Multics system," 1968. (CLOCK)
|
||||
- R. Mattson, J. Gecsei, D. Slutz, I. Traiger, "Evaluation techniques for storage
|
||||
hierarchies," *IBM Systems Journal*, 1970. (stack distance, stack algorithms)
|
||||
- E. O'Neil, P. O'Neil, G. Weikum, "The LRU-K page replacement algorithm for
|
||||
database disk buffering," *SIGMOD*, 1993.
|
||||
- T. Johnson, D. Shasha, "2Q: A low overhead high performance buffer management
|
||||
replacement algorithm," *VLDB*, 1994.
|
||||
- P. Cao, S. Irani, "Cost-aware WWW proxy caching algorithms," *USENIX USITS*,
|
||||
1997. (GreedyDual-Size)
|
||||
- S. Jiang, X. Zhang, "LIRS: An efficient low inter-reference recency set
|
||||
replacement policy," *SIGMETRICS*, 2002.
|
||||
- N. Megiddo, D. Modha, "ARC: A self-tuning, low overhead replacement cache,"
|
||||
*USENIX FAST*, 2003 (and the *IEEE Computer* 2004 write-up).
|
||||
- S. Jiang, F. Chen, X. Zhang, "CLOCK-Pro: An effective improvement of the CLOCK
|
||||
replacement," *USENIX ATC*, 2005.
|
||||
- K. Shah, A. Matani, M. Mitzenmacher, "An O(1) algorithm for implementing the
|
||||
LFU cache eviction scheme," 2010.
|
||||
- G. Einziger, R. Friedman, B. Manes, "TinyLFU: A highly efficient cache
|
||||
admission policy," *ACM TOS*, 2017. (see also the Caffeine design notes)
|
||||
|
||||
**KV-cache eviction (tensor level)**
|
||||
- G. Xiao et al., "Efficient streaming language models with attention sinks"
|
||||
(StreamingLLM), 2023.
|
||||
- Z. Zhang et al., "H2O: Heavy-hitter oracle for efficient generative inference
|
||||
of large language models," *NeurIPS*, 2023.
|
||||
- Z. Liu et al., "Scissorhands: Exploiting the persistence of importance
|
||||
hypothesis for LLM KV cache compression," *NeurIPS*, 2023.
|
||||
- Y. Li et al., "SnapKV: LLM knows what you are looking for before generation,"
|
||||
*NeurIPS*, 2024.
|
||||
- S. Ge et al., "Model tells you what to discard: Adaptive KV cache compression
|
||||
for LLMs" (FastGen), *ICLR*, 2024.
|
||||
|
||||
**Agent context, memory, and long-context behavior**
|
||||
- C. Packer et al., "MemGPT: Towards LLMs as operating systems," 2023. (the
|
||||
memory-hierarchy analogy, now Letta)
|
||||
- P. Lewis et al., "Retrieval-augmented generation for knowledge-intensive NLP
|
||||
tasks," *NeurIPS*, 2020. (RAG = fetch-on-miss)
|
||||
- N. Liu et al., "Lost in the middle: How language models use long contexts,"
|
||||
*TACL*, 2023. (position bias — *where* in context you keep something matters)
|
||||
- Anthropic, "Effective context engineering for AI agents" and the prompt-caching
|
||||
/ context-editing docs. (admission, compaction, prefix caching in practice)
|
||||
17
drafts/_template.md
Normal file
17
drafts/_template.md
Normal file
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: <Title Case>
|
||||
date: <YYYY-MM-DD>
|
||||
layout: column.njk
|
||||
tags: [column, <computing | philosophy | society | writing>]
|
||||
permalink: /columns/<year>/<slug>/
|
||||
---
|
||||
|
||||
Open in the middle of the thought. No headline restated, no throat-clearing —
|
||||
column.njk renders `title` as the h1 already, and the shortest columns are a
|
||||
single unbroken paragraph that starts where the idea starts and stops when it
|
||||
runs out.
|
||||
|
||||
Copy this to drafts/<YYYY-MM-DD>-<slug>.md and fill the front matter in. It
|
||||
renders at its real permalink on the dev server and nowhere else. Publishing is
|
||||
`git mv drafts/<file> columns/` — the front matter is already complete, so
|
||||
leaving this directory is the whole act, and moving it back retracts it.
|
||||
82
drafts/do-people-think.md
Normal file
82
drafts/do-people-think.md
Normal file
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: Do ~~Machines~~ People Think?
|
||||
date: 2026-04-02
|
||||
layout: column.njk
|
||||
tags: [column, philosophy]
|
||||
permalink: /columns/2026/do-people-think/
|
||||
---
|
||||
|
||||
## Rough Notes
|
||||
- Thinking:
|
||||
- Can we flip this around, and define it by reduction? What isn't thinking?
|
||||
1. Reaction
|
||||
2. Reflex
|
||||
3. Conditioned behavior
|
||||
- But what part of thinking isn't grounded in some conditioned behavior? Some reflex triggered by a problem to solve?
|
||||
- But thinking seems to involve "modeling," and for our purposes, we mean a mental model.
|
||||
- This mental model gets tested, calibrated, exercised.
|
||||
- So how is this different than a program?
|
||||
- Digital vs. Analog?
|
||||
- Inputs, the decisions, are much fuzzier for organic mental models (and thinking)
|
||||
- Computers are (seemingly) deterministic
|
||||
- But, if we shed this assumption, what happens to our distinction?
|
||||
- If computers can "approximate" the fuzziness with digital values...
|
||||
- same question, what happens to our distinction?
|
||||
- Thinking as a survival skill
|
||||
- Thinking as communication
|
||||
- Socialization
|
||||
- Connecting with neighbors and "vibing."
|
||||
- Thinking as an emergent behavior, among a pool of people
|
||||
- Maybe the mob mentality is the purest expression of thought.
|
||||
- Without self, or ego.
|
||||
- What does it map to, in the scheme of things?
|
||||
- I keep thinking about connections
|
||||
- And how stifling it is to exist in isolation
|
||||
- But to become ingrained in something
|
||||
- And to connect with the people, the pieces
|
||||
- To have an exchange, that's the essence of thinking
|
||||
- Some kind of transformation, born from the coordination and interaction of many players
|
||||
- Bottling it within youself, just as senses, impressions without a voice, with no expression or transformation
|
||||
- We form groups, we organize, and we process together. That's thinking
|
||||
|
||||
|
||||
## Quotes
|
||||
> "[he thinks] that the idealistic creations of his mind... also represent reality." - Claude Bernard
|
||||
|
||||
> "Much thinking is done in completely personal, idiosyncratic terms, so much so that how it is done is incommunicable." - Gerald Weinberg
|
||||
|
||||
|
||||
## Questions
|
||||
|
||||
- Who is Claude Bernard?
|
||||
- Famous French Physiologist (1813-78).
|
||||
|
||||
- How would I define "thinking?"
|
||||
|
||||
- Is "thinking" in isolation really thinking? Or, is thinking in a social / anthropological sense a connector?
|
||||
|
||||
- Do machines Think?
|
||||
- Yes
|
||||
- True thought has structure, a sequence, clear logical steps.
|
||||
- It is neither random nor unpredictable. It follows clear, irrefutable logic.
|
||||
- Anything else is noise
|
||||
- No
|
||||
- Machines are bound by rules. They can only follow a pre-programmed sequence of steps
|
||||
- The sequence may have rich variety, but lacks in originality.
|
||||
- Thinking is about novelty. Making leaps of intuition and instinct.
|
||||
- If we can pave the way between with logic or reasoning, that's nice, but not essential.
|
||||
- I reject this approach as flawed, lacking both nuance and merit.
|
||||
- Total aside: the point shouldn't be to subscribe absolutely to one side or the other, and claim it the undeniable truth.
|
||||
- These are devices, anchor points around which we can scaffold our reasoning, and draw tighter bounds around the concept being examined.
|
||||
- They themselves are not to be confused with the subject at hand.
|
||||
- by extension, we won't say that one or the other, both or neither are the thing itself.
|
||||
- Like confusing the map for the terrain.
|
||||
- these are navigational aids
|
||||
- So probably not a single definition of thinking
|
||||
- there's the intuitive kind, that machines might struggle with
|
||||
- There's the expansive, logical kind, machines are better suited for.
|
||||
|
||||
|
||||
## Links
|
||||
[1] - Hamming, R. (1997). The art of doing science and engineering: Learning to learn. CRC. <https://archive.org/details/artofdoingscienc0000rich>
|
||||
[2] - Weinberg, Gerald M. An Introduction to General Systems Thinking. Wiley, 1975. Internet Archive, <https://archive.org/details/introductiontoge00gera>.
|
||||
35
drafts/drafts.11tydata.js
Normal file
35
drafts/drafts.11tydata.js
Normal file
@@ -0,0 +1,35 @@
|
||||
// Drafts are visible on the dev server and nowhere else.
|
||||
//
|
||||
// ELEVENTY_RUN_MODE is "serve" or "watch" under `yarn dev`, and "build" under
|
||||
// `yarn build` — which is the only thing CI and the Dockerfile ever run. So the
|
||||
// gate fails safe: a draft can be previewed at its real permalink locally, but
|
||||
// no code path that produces a deployable _site/ will emit one.
|
||||
//
|
||||
// Collection membership is itself gated on the run mode, and that is the hard
|
||||
// guarantee rather than a courtesy:
|
||||
//
|
||||
// build — a draft belongs to NO collection, so no template can render its
|
||||
// title, date, or URL even by accident. Eleventy auto-creates a
|
||||
// collection per tag, and drafts carry `tags: [column, <subject>]`
|
||||
// so that promoting one is a plain `git mv`. That means they land
|
||||
// in collections.philosophy and friends, which /subjects/ iterates
|
||||
// directly — it listed every draft with href="false" until this
|
||||
// gate existed. Filtering the two hand-written collections was not
|
||||
// enough; only excluding them wholesale is.
|
||||
// serve — a draft joins collections so /drafts/ can enumerate them. The
|
||||
// `draft` flag then keeps them out of the homepage and the sidebar
|
||||
// (see .eleventy.js) and out of /subjects/ (see subjects.njk), so
|
||||
// every page except /drafts/ shows exactly what production shows.
|
||||
//
|
||||
// Promoting a draft is `git mv drafts/x.md columns/`: the front matter is
|
||||
// already complete and correct, and leaving this directory is what publishes it.
|
||||
|
||||
const PREVIEW = process.env.ELEVENTY_RUN_MODE !== "build";
|
||||
|
||||
module.exports = {
|
||||
draft: true,
|
||||
eleventyExcludeFromCollections: !PREVIEW,
|
||||
eleventyComputed: {
|
||||
permalink: (data) => (PREVIEW ? data.permalink : false),
|
||||
},
|
||||
};
|
||||
36
drafts/how-to-write.md
Normal file
36
drafts/how-to-write.md
Normal file
@@ -0,0 +1,36 @@
|
||||
---
|
||||
title: How To Write
|
||||
date: 2026-04-06
|
||||
layout: column.njk
|
||||
tags: [column, writing]
|
||||
permalink: /columns/2026/how-to-write/
|
||||
---
|
||||
|
||||
## Prompt
|
||||
- The kernel of an idea
|
||||
- It's generally recommended to plant a seed first
|
||||
- something you can return to over time
|
||||
- reference other ideas you come across
|
||||
- develop and refine
|
||||
|
||||
## Questions
|
||||
- Questions give structure to your work
|
||||
- Even implicit ones - the unsaid queries that motivate the composition of the piece
|
||||
- what's included, and when?
|
||||
- as important: what's left out?
|
||||
- make everything a deliberate choice
|
||||
- Why write, if not to test an idea against your own specific qualities? To see if there's a reaction, some expansiveness or contraction.
|
||||
- They provide the substrate to thought, and the nourishment, encouraging the frontier to expand
|
||||
|
||||
## Rough Notes
|
||||
- The heap
|
||||
- The outlet for stream of consciousness thinking
|
||||
- See what associations bubble up to the surface
|
||||
- See what resonates, what clusters, the affinities that form
|
||||
- The warmup
|
||||
|
||||
## Quotes
|
||||
- For the sake of having an opinion, I say it's better to rearticulate someone's idea in your own words, to digest it with your own acuity, than to simply reexpress the idea as a quote.
|
||||
- Other than stating something factually: at such a time, such a thing was said
|
||||
- Refernce it, so motivated readers can connect with more writing that may expand on the idea.
|
||||
- But don't just repeat the statement verbatim. Connect with the idea, exppress it in your own way for your own audience. Make it available.
|
||||
26
drafts/index.njk
Normal file
26
drafts/index.njk
Normal file
@@ -0,0 +1,26 @@
|
||||
---
|
||||
layout: base.njk
|
||||
title: Drafts
|
||||
permalink: /drafts/
|
||||
eleventyExcludeFromCollections: true
|
||||
---
|
||||
|
||||
<h1>Drafts</h1>
|
||||
|
||||
<p>Unpublished. Visible on the dev server only — <code>yarn build</code> emits
|
||||
none of this. Promote one with <code>git mv drafts/<file> columns/</code>.</p>
|
||||
|
||||
{%- if collections.drafts.length %}
|
||||
<ul class="toy-list">
|
||||
{%- for draft in collections.drafts %}
|
||||
<li>
|
||||
<a href="{{ draft.url }}">{{ draft.data.title | inlineMarkdown | safe }}</a>
|
||||
<span class="toy-description">
|
||||
{{ draft.date | isoDate }} · {{ draft.data.tags | reject("equalto", "column") | join(", ") }}
|
||||
</span>
|
||||
</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
{%- else %}
|
||||
<p>No drafts.</p>
|
||||
{%- endif %}
|
||||
40
drafts/systems.md
Normal file
40
drafts/systems.md
Normal file
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: systems
|
||||
date: 2026-04-06
|
||||
layout: column.njk
|
||||
tags: [column, philosophy]
|
||||
permalink: /columns/2026/systems/
|
||||
---
|
||||
|
||||
## Rough Notes
|
||||
- [Reichenbach](https://plato.stanford.edu/archives/fall2025/entries/reichenbach/)
|
||||
> "The greatest empiricist of the 20th century!"
|
||||
- [Boulding?](asdf) - Spaceship Earth, General Systems Theory
|
||||
- Meta-laws, generalizations, that leap from one discipline to another.
|
||||
- We group a body of knowledge together under an umbrella of a domain, and in that box, the roots of connection may grow, but we miss the cross-cutting relationships that may have been realized, had we not kept knowledge as a house plant, bundled in a box.
|
||||
- Teaching the truth vs. teaching a model. Do we confuse the latter for the former, or is it misrepresented from the get-go?
|
||||
|
||||
|
||||
## Questions
|
||||
- Have we kneecapped ourselves through rigid classification, and artificial specialization?
|
||||
- Are these contradictions? Discuss:
|
||||
- The Composition Law:
|
||||
> The whole is more than the sum of its parts.
|
||||
- The Decomposition Law:
|
||||
> The part is more than a fraction of the whole
|
||||
- Are they? If the "part" were less than the fraction of the whole, how could the whole possibly be more than the sum of its parts?
|
||||
- But then, all the parts together must be greater than the whole
|
||||
- But the whole is greater than the sum of all parts...
|
||||
- I say it's flawed to distinguish whole and part. The arithmetic doesn't work out because they're necessarily interconnected.
|
||||
|
||||
|
||||
## Quotes
|
||||
> The power of reason must be sought \
|
||||
> not in the rules that reason dictates to our imagination, \
|
||||
> but in the ability to free ourselves \
|
||||
> from any kind of rules to which we have been conditioned \
|
||||
> through experience and tradition
|
||||
- Reichenbach
|
||||
|
||||
## ToDo
|
||||
- Max Planck Scientific Autobiography
|
||||
Reference in New Issue
Block a user