31 lines
1.6 KiB
Markdown
31 lines
1.6 KiB
Markdown
|
---
|
||
|
|
title: Panel of Judges
|
||
|
|
date: 2026-07-16
|
||
|
|
layout: column.njk
|
||
|
|
tags: [column, computing]
|
||
|
|
permalink: /columns/2026/panel-of-judges/
|
||
|
|
---
|
||
|
|
|
||
|
|
1. How could quantum computing impact LLMs, or other probablistic, non-deterministic models?
|
||
|
|
|
||
|
|
2. Can we put type safety, reliability features around LLM invocations? Can we formalize it, as its own kind of process: sequential or concurrent invocations of asynchronous processes.
|
||
|
|
|
||
|
|
We do that already with RPCs, web requests, etc.
|
||
|
|
|
||
|
|
what are some safety patterns around `async await`?
|
||
|
|
* timeouts
|
||
|
|
- can we explicitly prompt the LLM: You have 500ms to respond or we move on?
|
||
|
|
- Like an ad network bid: "winning" response wins
|
||
|
|
- How are they compared? Panel of judges
|
||
|
|
- At first I thought you could single "identical" judges in front of each submission
|
||
|
|
- that's counting on each to be trustworthy
|
||
|
|
- Then I thought about having a panel of judges, quorum style
|
||
|
|
- Then I thought about a "ranking funnel" a shrinking pool, maybe bracket style (is that a merge-sort elimination?)
|
||
|
|
- or you grade pools of responses as a collection, instead of individually (can't tell if this would be a fidelity loss or win - would that matter?)
|
||
|
|
- this sounds really expensive: time and resources
|
||
|
|
- 1/N success rate, and what does it take to eliminate the other N-1?
|
||
|
|
- it sounds much less efficient than developing / evolving a single model session (some kind of hybrid) as long as it survives (and maybe each session can determine that for itself).
|
||
|
|
- LLM (lives) are not precious. They can off themselves.
|
||
|
|
* exception handling // what's exceptional when prompting an LLM?
|
||
|
|
* validations / sanitation
|