claude-skills/scripts/check_model_freshness_allowlist.txt
Ben Fairless 66ea9621dd
fix(models): remove retired model IDs and stale pricing, flip G7 blocking
Clears every reference the new G7 lint flags, then makes it blocking so the
class cannot drift back. audit/engineering-agentic-2026-07 marked the
senior-ml-engineer half of this STILL-OPEN.

Deleted rather than updated:

  - agent-designer/agent_evaluator.py's _define_cost_benchmarks() held
    per-token prices for gpt-4, gpt-3.5-turbo and claude-3 at 2024 rates. The
    result was assigned to self.cost_benchmarks and never read by anything, so
    the method is gone. Cost analysis uses the cost_usd the caller supplies per
    execution log, which is the only figure that can be accurate

Made model-agnostic, following the precedent already set by
senior-prompt-engineer/scripts/prompt_optimizer.py's --price-per-mtok:

  - senior-ml-engineer SKILL.md and llm_integration_guide.md drop both 2024
    price tables and the context-window table (which claimed GPT-4 = 8,192).
    calculate_cost() takes rates as parameters; count_tokens() takes an
    encoding name, since encodings outlive model IDs and
    encoding_for_model() raises KeyError on anything unmapped
  - OpenAIProvider loses its default model, so the caller must pass one
  - llm-cost-optimizer's routing table names tiers, not models

Pinned to current IDs where an example genuinely needs one: SKILL_PIPELINE.md
(claude-opus-4-6 -> claude-opus-5), prompt-governance (claude-sonnet-4-5 ->
claude-sonnet-5), agent-designer README. Both dual-publish copies of the CAIO
pricing move together, so G4 stays green.

TEAM_STRUCTURE_GUIDE.md documented `prompt_optimizer.py --model gpt-4 --task
classification`. That contract no longer exists: there is no --task flag and
`prompt` is a required positional. Replaced with a runnable invocation.

Four references stay, with reasons in the allowlist: two litreview examples
where the retired model is the subject of the literature being reviewed, one
dated Computer Use citation, and the embedding benchmark already labelled a
2024 snapshot.

Assisted-by: Claude Code:claude-opus-5
2026-08-03 08:55:36 +08:00

28 lines
1.4 KiB
Text

# check_model_freshness.py allowlist — deliberate references only.
# Format: <file-glob> :: <substring that must appear on the line>
#
# A retired model identifier belongs here only when naming it IS the point:
# a regulatory list frozen at a date, a benchmark labelled as historical, or
# the deny-list in the linter itself. If the reference is a script default,
# a price table, or a copy-pasteable command, fix the reference instead.
#
# Every entry needs a one-line reason above it.
# The EU AI Act's own systemic-risk examples are frozen in the cited text;
# renaming them would misquote the regulation.
*/references/ai_risk_governance.md :: systemic risk
# Benchmarks explicitly published as a dated snapshot, with a staleness
# warning already at the top of the file.
*/references/embedding_model_benchmark.md :: ada-002
*/references/embedding_model_benchmark.md :: text-embedding
# litreview examples where the retired model is the SUBJECT of the literature
# being reviewed, not a model the skill would call. Renaming these would
# misdescribe the worked example.
*/litreview/references/framework_selection.md :: Med-PaLM
*/litreview/scripts/cross_search_aggregator.py :: medical Q&A
# Dated citation: Computer Use shipped on Claude 3.5 Sonnet, and the "+ later"
# is the point of the sentence.
*/notebooklm/references/browser_automation_canon.md :: Computer Use documentation