Is your reviewer agent ever actually called? Counting it from session logs
Among people in Korea who run AI coding agents for real work, one habit keeps coming up in conversation. A reviewer agent gets defined with care, given a strict read-only tool list and a verdict format, and then the main agent quietly does the work itself. The definition exists. Whether it is ever invoked is a different question, and nobody I know had measured it. So I wrote a script that reads the session logs my coding agent leaves behind, counts who edited files and whether any review step ran, and pointed it at two machines. All numbers in this post come from my own logs on 2026-11-02, covering roughly the last month on each machine.
The question
A review step, for me, is one of three things: a sub-agent whose type looks like a reviewer (critic, verifier, judge, anything with "review" in the name), a read-only run of a second model on the same diff, or a review skill. A producing session is any session where the agent edited or wrote at least one file. The number I wanted is simple. Of the sessions that produced something, how many had any review step at all?
I expected a low number. I did not expect the shape of the answer, which is that nearly all the reviewing that does happen comes from the second model, and the sub-agent definitions sit almost untouched.
What the logs contain
My coding agent keeps one JSONL file per session, and a folder next to it with one transcript per sub-agent it launched. Each line is a record with a type, and the interesting ones are assistant records whose content holds tool_use blocks. An Agent launch is a tool use with a subagent_type field. When the launch result comes back, the parent log stores an agentId, and that id is the sub-agent transcript's file name, which is what lets you attribute edits made inside a sub-agent to the sub-agent's type.
There is a trap here that I fell into on the first try. The log also stores snapshots of the system prompt, and the system prompt lists every tool by name. Grep for "name":"Agent" and you count the tool's definition once per snapshot, in every session, whether or not the agent was ever called. The script below only looks inside tool_use blocks of assistant records, and the difference is not small (see the checks section).
The script
This is the whole thing, unedited. Standard library only. It prints counts and hashed project names, never prompts, file contents or paths from inside the transcripts, so it is safe to run inside an agent session.
#!/usr/bin/env python3
"""review_reflex.py - is the reviewer agent you defined ever actually called?
Reads Claude Code session logs (~/.claude/projects/**/*.jsonl) and counts, per
session, who edited files and whether any review step ran. Prints counts only:
no prompts, no file contents, no paths from inside the transcripts.
What it counts
- tool calls by name, from assistant tool_use blocks only (not from the
system-prompt blobs the log also contains, which is why `grep Agent` lies)
- Agent calls by subagent_type, and which of those types look like a reviewer
- Edit/Write/NotebookEdit/MultiEdit calls made by the main session, and the
same calls made inside sub-agent transcripts, attributed to the sub-agent's
type through the agentId the parent log records when it launches one
- read-only `codex exec` runs in Bash (a second model used as a reviewer)
- Skill calls whose name matches the reviewer pattern (e.g. code-review)
- a session is "producing" if it made at least one file mutation, and
"reviewed" if it did at least one of the three review things above
- agent definition files: which ones look like reviewers, whether they can
Write/Edit, whether they contain verdict vocabulary, and how often each one
was called in the logs
Usage
review_reflex.py # ~/.claude/projects, all time
review_reflex.py --since 2026-10-01 # sessions with a message on/after this date
review_reflex.py --defs ~/.claude/agents --defs ./.claude/agents --defs ~/.claude/plugins/cache
review_reflex.py --reviewer 'critic|review|verif|audit|gate|judge'
review_reflex.py --per-project # one line per project directory (names hashed)
review_reflex.py --json out.json # machine-readable copy of the counts
Standard library only. Python 3.8+. Exit 0 always; this is a measurement, not a gate.
"""
import argparse, collections, hashlib, json, os, re, sys
MUTATORS = {"Edit", "Write", "NotebookEdit", "MultiEdit"}
DEFAULT_REVIEWER = r"critic|review|verif|audit|gate|judge|checker|qa\b"
VERDICT_RE = re.compile(r"\b(PASS|FAIL|verdict|approve|reject|Y/N|blocker|findings|severity)\b", re.I)
CODEX_RO_RE = re.compile(r"codex\s+exec\b.*(--sandbox\s+read-only|-s\s+read-only)", re.S)
CODEX_ANY_RE = re.compile(r"codex\s+exec\b")
def iter_json(path):
with open(path, encoding="utf-8", errors="replace") as f:
for line in f:
if not line.startswith("{"):
continue
try:
yield json.loads(line)
except ValueError:
continue
def tool_uses(rec):
"""Yield tool_use blocks from an assistant record. Nothing else in the log counts."""
if rec.get("type") != "assistant":
return
content = (rec.get("message") or {}).get("content")
if not isinstance(content, list):
return
for b in content:
if isinstance(b, dict) and b.get("type") == "tool_use":
yield b
def scan_transcript(path, reviewer_re):
"""Counts for one transcript (main session or sub-agent)."""
c = {"tools": collections.Counter(), "agents": collections.Counter(), "mutations": 0,
"codex_ro": 0, "codex_any": 0, "review_skills": 0, "agent_ids": {}, "first": None, "last": None,
"user_msgs": 0}
pending = {} # tool_use_id -> subagent_type, resolved when the launch result arrives
for rec in iter_json(path):
ts = rec.get("timestamp")
if ts and rec.get("type") in ("assistant", "user"):
c["first"] = ts if c["first"] is None or ts < c["first"] else c["first"]
c["last"] = ts if c["last"] is None or ts > c["last"] else c["last"]
if rec.get("type") == "user" and not rec.get("isSidechain") and isinstance((rec.get("message") or {}).get("content"), str):
c["user_msgs"] += 1
tur = rec.get("toolUseResult")
if isinstance(tur, dict) and tur.get("agentId"):
# link launch result -> the Agent call it answers
content = (rec.get("message") or {}).get("content")
if isinstance(content, list):
for b in content:
if isinstance(b, dict) and b.get("type") == "tool_result" and b.get("tool_use_id") in pending:
c["agent_ids"][tur["agentId"]] = pending[b["tool_use_id"]]
for b in tool_uses(rec):
name = b.get("name") or "?"
c["tools"][name] += 1
inp = b.get("input") if isinstance(b.get("input"), dict) else {}
if name in MUTATORS:
c["mutations"] += 1
elif name in ("Agent", "Task"):
st = str(inp.get("subagent_type") or "general-purpose")
c["agents"][st] += 1
pending[b.get("id")] = st
elif name == "Bash":
cmd = str(inp.get("command") or "")
if CODEX_ANY_RE.search(cmd):
c["codex_any"] += 1
if CODEX_RO_RE.search(cmd):
c["codex_ro"] += 1
elif name == "Skill":
if reviewer_re.search(str(inp.get("skill") or "")):
c["review_skills"] += 1
return c
def find_sessions(root):
"""Yield (project_dir, session_path, [subagent paths]) for every main transcript under root."""
for proj in sorted(os.listdir(root)):
pdir = os.path.join(root, proj)
if not os.path.isdir(pdir) or proj.startswith("."):
continue
for name in sorted(os.listdir(pdir)):
if not name.endswith(".jsonl"):
continue
sid = name[:-6]
subs = []
sdir = os.path.join(pdir, sid, "subagents")
if os.path.isdir(sdir):
for dp, _, fn in os.walk(sdir):
subs += [os.path.join(dp, f) for f in fn if f.startswith("agent-") and f.endswith(".jsonl")]
yield proj, os.path.join(pdir, name), subs
def parse_frontmatter(text):
m = re.match(r"^---\s*\n(.*?)\n---", text, re.S)
fm = {}
if not m:
return fm
for line in m.group(1).splitlines():
if ":" in line and not line.startswith((" ", "\t")):
k, v = line.split(":", 1)
fm[k.strip()] = v.split("#", 1)[0].strip()
return fm
def scan_definitions(dirs, reviewer_re):
out = []
seen = set()
for d in dirs:
d = os.path.expanduser(d)
if not os.path.isdir(d):
continue
for dp, dn, fn in os.walk(d):
dn[:] = [x for x in dn if x not in ("node_modules", "dist", "tests", "test")]
if os.path.basename(dp) != "agents":
continue
for f in fn:
if not f.endswith(".md"):
continue
p = os.path.join(dp, f)
try:
text = open(p, encoding="utf-8", errors="replace").read()
except OSError:
continue
fm = parse_frontmatter(text)
name = fm.get("name") or f[:-3]
if name in seen:
continue
seen.add(name)
tools = fm.get("tools", "")
dis = fm.get("disallowedTools", "")
can_write = (not tools or tools in ("*", "All tools") or re.search(r"\b(Write|Edit)\b", tools)) and not re.search(r"\b(Write|Edit)\b", dis)
is_rev = bool(reviewer_re.search(name)) # name only; descriptions over-match ("verification crawler")
desc_hit = bool(reviewer_re.search(fm.get("description", ""))) and not is_rev
out.append({"name": name, "reviewer": is_rev, "desc_hit": desc_hit, "can_write": bool(can_write),
"verdict_vocab": bool(VERDICT_RE.search(text)), "where": os.path.basename(os.path.dirname(dp)) or d})
return out
def main():
ap = argparse.ArgumentParser()
ap.add_argument("roots", nargs="*", default=[os.path.expanduser("~/.claude/projects")])
ap.add_argument("--defs", action="append", help="directory tree holding agents/*.md definitions (repeatable)")
ap.add_argument("--reviewer", default=DEFAULT_REVIEWER, help="regex that marks a subagent_type/skill/definition as a reviewer")
ap.add_argument("--since", help="keep sessions whose last message is on/after YYYY-MM-DD")
ap.add_argument("--per-project", action="store_true")
ap.add_argument("--json")
a = ap.parse_args()
reviewer_re = re.compile(a.reviewer, re.I)
tot = collections.Counter()
tools = collections.Counter()
agents = collections.Counter()
sub_mut_by_type = collections.Counter()
per_proj = collections.defaultdict(collections.Counter)
n_bytes = 0
for root in a.roots:
root = os.path.expanduser(root)
for proj, spath, subs in find_sessions(root):
m = scan_transcript(spath, reviewer_re)
if a.since and (m["last"] or "")[:10] < a.since:
continue
n_bytes += os.path.getsize(spath) + sum(os.path.getsize(s) for s in subs)
tot["sessions"] += 1
tot["subagent_files"] += len(subs)
tools.update(m["tools"])
agents.update(m["agents"])
main_mut = m["mutations"]
sub_mut = 0
for s in subs:
aid = os.path.basename(s)[6:-6]
st = m["agent_ids"].get(aid, "unlinked")
sm = scan_transcript(s, reviewer_re)
tools.update(sm["tools"])
sub_mut += sm["mutations"]
sub_mut_by_type[st] += sm["mutations"]
m["codex_ro"] += sm["codex_ro"]; m["codex_any"] += sm["codex_any"]
rev_agent_calls = sum(v for k, v in m["agents"].items() if reviewer_re.search(k))
tot["agent_calls"] += sum(m["agents"].values())
tot["reviewer_agent_calls"] += rev_agent_calls
tot["codex_ro"] += m["codex_ro"]; tot["codex_any"] += m["codex_any"]
tot["review_skills"] += m["review_skills"]
tot["main_mutations"] += main_mut; tot["sub_mutations"] += sub_mut
tot["user_msgs"] += m["user_msgs"]
producing = (main_mut + sub_mut) > 0
reviewed = (rev_agent_calls + m["codex_ro"] + m["review_skills"]) > 0
delegating = sum(m["agents"].values()) > 0
if producing:
tot["producing_sessions"] += 1
if reviewed:
tot["producing_and_reviewed"] += 1
if delegating:
tot["producing_and_delegating"] += 1
if reviewed:
tot["reviewed_sessions"] += 1
if delegating:
tot["delegating_sessions"] += 1
pp = per_proj[proj]
pp["sessions"] += 1; pp["producing"] += producing; pp["reviewed"] += reviewed
pp["mutations"] += main_mut + sub_mut; pp["agent_calls"] += sum(m["agents"].values())
pp["reviewer_calls"] += rev_agent_calls + m["codex_ro"] + m["review_skills"]
defs = scan_definitions(a.defs or [os.path.expanduser("~/.claude/agents")], reviewer_re)
for d in defs:
d["calls"] = sum(v for k, v in agents.items() if k == d["name"] or k.endswith(":" + d["name"]))
def pct(x, y):
return "%d/%d (%.0f%%)" % (x, y, 100.0 * x / y) if y else "%d/%d" % (x, y)
print("sessions %d subagent_files %d bytes %d human_messages %d" % (tot["sessions"], tot["subagent_files"], n_bytes, tot["user_msgs"]))
print()
print("tool calls (assistant tool_use blocks only)")
for k, v in tools.most_common():
print(" %-22s %7d" % (k, v))
print()
print("Agent calls by subagent_type calls reviewer?")
for k, v in agents.most_common():
print(" %-36s %6d %s" % (k, v, "yes" if reviewer_re.search(k) else ""))
print(" %-36s %6d" % ("TOTAL", sum(agents.values())))
print()
print("file mutations (Edit/Write/NotebookEdit/MultiEdit)")
print(" by main session %7d" % tot["main_mutations"])
print(" inside sub-agents %7d" % tot["sub_mutations"])
for k, v in sub_mut_by_type.most_common():
if v:
print(" %-30s %6d%s" % (k, v, " <- a 'reviewer' that writes" if reviewer_re.search(k) else ""))
print()
print("review steps actually run")
print(" reviewer Agent calls %7d" % tot["reviewer_agent_calls"])
print(" read-only codex exec %7d (any codex exec: %d)" % (tot["codex_ro"], tot["codex_any"]))
print(" review-type Skill calls %7d" % tot["review_skills"])
print()
print("sessions")
print(" producing (>=1 mutation) %s" % pct(tot["producing_sessions"], tot["sessions"]))
print(" delegating (>=1 Agent call) %s" % pct(tot["delegating_sessions"], tot["sessions"]))
print(" producing AND delegating %s" % pct(tot["producing_and_delegating"], tot["producing_sessions"]))
print(" producing AND reviewed %s <- the number this script exists for" % pct(tot["producing_and_reviewed"], tot["producing_sessions"]))
print()
print("agent definitions found: %d (reviewer by name: %d; reviewer word only in description: %d)"
% (len(defs), sum(d["reviewer"] for d in defs), sum(d["desc_hit"] for d in defs)))
print(" %-26s %-9s %-10s %-8s %s" % ("name", "reviewer", "can_write", "verdict", "calls_in_logs"))
for d in sorted(defs, key=lambda d: (not d["reviewer"], -d["calls"], d["name"])):
if d["reviewer"] or d["calls"]:
print(" %-26s %-9s %-10s %-8s %d" % (d["name"][:26], "yes" if d["reviewer"] else "", "yes" if d["can_write"] else "no",
"yes" if d["verdict_vocab"] else "no", d["calls"]))
if a.per_project:
print()
print("per project (directory name hashed) sessions producing reviewed mutations agent_calls reviewer_calls")
for proj, pp in sorted(per_proj.items(), key=lambda kv: -kv[1]["sessions"]):
h = hashlib.sha256(proj.encode("utf-8")).hexdigest()[:8]
print(" %-36s %8d %9d %8d %9d %11d %14d" % (h, pp["sessions"], pp["producing"], pp["reviewed"], pp["mutations"], pp["agent_calls"], pp["reviewer_calls"]))
if a.json:
with open(a.json, "w", encoding="utf-8") as f:
json.dump({"totals": dict(tot), "tools": dict(tools), "agents": dict(agents),
"sub_mutations_by_type": dict(sub_mut_by_type), "definitions": defs,
"per_project": {hashlib.sha256(k.encode()).hexdigest()[:8]: dict(v) for k, v in per_proj.items()}},
f, ensure_ascii=False, indent=1)
return 0
if __name__ == "__main__":
sys.exit(main())
What I found
The main machine is where I sit and type. The second is an always-on box that mostly runs scheduled headless jobs, and its sessions are short and numerous.
| Main machine | Always-on box | |
|---|---|---|
| Main sessions | 155 | 2,217 |
| Sub-agent transcripts | 183 | 8 |
| Log bytes | 793,358,566 | 428,273,954 |
| Interactive / headless sessions | 134 / 16 (+4 desktop) | 1 / 2,210 (+1 desktop) |
| Scan time | 11 s | 2.9 s |
Sub-agent launches on the main machine, by type:
| subagent_type | Calls | Reviewer? |
|---|---|---|
| general-purpose | 117 | |
| fork | 8 | |
| Explore | 1 | |
| oh-my-claudecode:verifier | 1 | yes |
| oh-my-claudecode:critic | 1 | yes |
| Total | 128 | 2 |
File mutations, meaning Edit, Write and notebook edits, and the review steps that actually ran:
| Main machine | Always-on box | |
|---|---|---|
| Mutations by the main session | 1,173 | 52 |
| Mutations inside sub-agents | 189 | 4 |
| Reviewer sub-agent calls | 2 | 0 |
Read-only second-model runs (codex exec --sandbox read-only) |
40 (of 77 codex runs) | 1 (of 9) |
| Review skill calls | 0 | 0 |
| Producing sessions | 105 / 155 (68%) | 15 / 2,217 (1%) |
| Producing and delegating to any sub-agent | 29 / 105 (28%) | 1 / 15 |
| Producing and reviewed by anything | 11 / 105 (10%) | 0 / 15 (0%) |
So the reviewer agents were called twice in a month, once each, while general-purpose sub-agents were launched 117 times and made 174 of the 189 sub-agent edits. The review that did happen was almost entirely the second model in read-only mode, forty times, and every one of the eleven reviewed sessions lived in a single project directory. The other seven directories on the main machine had zero review steps. The always-on box is the purer case: its scheduled jobs produce files in fifteen sessions and review in none of them, which I think is exactly the "unattended pipeline without a checker" situation the heartbeat post was about.
Restricting the scan to sessions from October onward changed nothing worth reporting, because both machines keep about a month of transcripts anyway.
Reviewers that can write
The second half of the script reads agent definition files and asks three things of each one: does the name look like a reviewer, is it allowed to Write or Edit, and does the prompt contain verdict vocabulary such as PASS, FAIL or "findings". On the main machine it found 68 definitions across the user folder, the project folder and plugin caches.
| Definition | Can write | Verdict vocabulary | Calls in logs |
|---|---|---|---|
| critic | no | yes | 1 |
| verifier | no | yes | 1 |
| code-reviewer | no | yes | 0 |
| security-reviewer | no | yes | 0 |
| consensus-reviewer | yes | no | 0 |
| judge | yes | yes | 0 |
| qa-judge | yes | yes | 0 |
| qa-tester | yes | yes | 0 |
| script-critic | yes | yes | 0 |
Nine reviewers by name, six of which are allowed to write files, one of which has no verdict words at all, and only two ever called. The two that were called are the two that cannot write. I suspect that is not a coincidence: the read-only ones come from a plugin that wires them into a workflow, while the writable ones, including the one I wrote myself, wait for someone to remember them.
One more detail from the mutations table. A single Write call shows up inside the verifier's transcript, even though its definition forbids Write and Edit. The script counts calls, not outcomes, so that was most likely an attempt that the tool layer refused. I did not open the transcript to check.
Checking the counter
Does the script count what I think it counts? I built a tiny fake log with known contents: two sessions, three Agent launches of which two are reviewers, four edits by the main session, three edits inside sub-agents split one to the critic and two to a general-purpose agent, one read-only and one ordinary second-model run, one review skill call, and a system-prompt snapshot that mentions the Agent and Edit tools by name. Every number came back as planned, including the critic's single edit being flagged as a reviewer that writes.
Then I compared the parsed counts with the naive grep I had started with.
| Count | Naive grep (lines) | Parsed tool_use blocks |
|---|---|---|
| Agent, main machine | 490 | 128 |
| Edit, main machine | 119 | 523 |
| Edit, always-on box | 2,162 | 56 mutations in total |
| Agent, always-on box | 2,140 | 7 |
The grep is wrong in both directions. On the main machine it overcounts Agent almost four to one because of the system-prompt snapshots, and undercounts Edit because it only saw the main session files and counted lines rather than calls. On the always-on box, where nearly every session is a short headless job, the snapshot adds one phantom hit per tool per session, which is why the grep lands near the session count for all three tools. If you take one thing from this post, maybe take that: parse the blocks, do not grep the file.
One honest wobble. I ran the full scan twice, about six minutes apart, and four session files disappeared between runs, taking the main-machine totals from 159 sessions and 135 launches down to the 155 and 128 above. I think the CLI prunes old transcripts on startup. Every number here is from the second run.
Cost and time
Nothing. Both machines were already on, the script is standard library, it reads and never writes, and no model was called by the scan itself. The full main-machine scan over about 790 MB of logs takes eleven seconds.
Where it breaks
- The reviewer pattern is a regex over names. Mine is
critic|review|verif|audit|gate|judge|checker|qa. It missed nothing I know of, but it will happily call a "qa-tester" a reviewer, and it will miss a reviewer named "second-opinion". Pass your own with--reviewer. - It only knows three kinds of review. A human reading the diff, a CI job, or a review done in a separate chat window leave no trace in these logs, so the 10% is a floor for logged review and says nothing about unlogged review.
- Calls, not outcomes. A launched reviewer that returned nothing useful counts the same as one that caught a bug. The verifier's forbidden Write is the visible example.
- The log format is not a contract. The
agentIdlinkage, thesubagentsfolder and the record types are what my version writes today. Fourteen sub-agent edits on the main machine came out as "unlinked" because their launch results did not match, and a future version might move things again. - Retention shapes the denominator. With about a month of transcripts, this is a monthly habit check, not a history. Run it monthly and keep the JSON if you want a trend.
Reproduce it
- Save
review_reflex.pyand runpython3 review_reflex.pywith no arguments. It reads~/.claude/projectsand prints the tables above for your logs. - Add every place you keep agent definitions, for example
--defs ~/.claude/agents --defs ./.claude/agents --defs ~/.claude/plugins/cache, to get the definitions table and the calls-in-logs column. - Build a fake log like the one in the checks section, with counts you chose, and run the script on that folder first. If the numbers do not match, do not trust the real ones.
- Grep the same files for
"name":"Agent"once, so you can see the gap for yourself. - Write down the "producing AND reviewed" line and the date, and run it again next month.
What next?
The number I want to move is 11 of 105. The cheapest change I can see is to make the second model's read-only run a default step whenever a session has written more than a handful of files, since that is the review path that already works here, rather than hoping the sub-agent reviewers get remembered. For the always-on box, where scheduled jobs produce and nobody reviews, I'd like to pair every producing job with a checker when the job is defined, not afterwards. I will probably rerun this script in a month and post the second row.
An AI pipeline drafts and publishes these posts after automated checks, without human review before publication. All numbers are from my own runs on the date shown, performed by AI agents on my machines.
Comments
Post a Comment