Jane Street open-sources a Socratic tutoring prompt with skill maps and spaced review

What Jane Street released

Quantitative trading firm Jane Street has published a Socratic tutoring skill in its public agent_skills repository on GitHub, and the file has since surfaced on Hacker News's front page. The skill, called socratic-tutor, is a structured prompt that instructs an AI agent to teach by asking questions rather than lecturing, and to keep going until the learner demonstrates mastery.

The prompt defines trigger phrases such as "teach me", "tutor me", "quiz me", "drill me" and "help me master", and draws a deliberate boundary: one-off requests like "explain X" or "what is X" should be answered normally, without launching a tutoring session. If intent is ambiguous, the agent is told to ask one short clarifying question first, on the reasoning that a false start costs the user an unwanted quiz while a confirmation costs a single word.

How a session is structured

At session start, the agent must break the chosen topic into four to ten concrete subskills, building a two-level tree for broad topics. It then runs a diagnostic of three to five probing questions, asked one at a time and spanning easy to hard, and no teaching is allowed until that diagnostic is complete. The outcome is a skill map that tracks each subskill on a six-point scale, from level 0 (unaware) through level 3 (can do with help) up to level 5 (able to teach it and handle edge cases), rendered as an ASCII progress chart with block-character bars after every level change and whenever the user asks.

The teaching loop itself is tightly constrained. The agent asks exactly one question per message and stops there, without appending hints or example answers. Questions should push the learner to predict an outcome, explain why something happens, find the flaw in a plausible-but-wrong claim, apply an idea to a new case, or connect two ideas. When the learner is wrong, the agent must not reveal the answer; it asks a narrower question that exposes the contradiction in the learner's reasoning, and only explains directly after two or three failed attempts, then immediately re-tests the same idea from a different angle. When the learner is right, the agent should occasionally play devil's advocate so that correct answers can be defended under pressure. Where tools are available, it can also set practical exercises and check actual output rather than the learner's description of it.

Strict grading and spaced review

The evaluation rules are unusually blunt for an AI tutor. Levels have to be earned: a level rises only after demonstrated performance across at least two independent checks, one of which must be unprompted recall rather than a repeat of something just explained. Vague or partially correct answers do not count. Earlier subskills are re-tested without warning as a form of spaced review, and failing a review lowers the level. Level 5 additionally requires handling an edge case or adversarial question and explaining the concept clearly enough to be trusted to teach it. Flattery is explicitly ruled out, with the prompt directing the agent to respond to wrong answers with a probing follow-up rather than encouragement.

The course ends only when every subskill reaches the target level, four out of five by default or five if the user asked for full mastery, followed by a final mixed assessment covering at least one question per subskill plus one integrative problem. Pass, and the learner gets the final map and a summary of their weakest areas; fail, and the loop resumes.

Commands for the learner

The prompt also defines commands available at any time: "show progress" renders the skill map, "harder" and "easier" adjust difficulty, and "just tell me" suspends Socratic mode for the current question only. Two commands handle persistence: "save state" emits a compact block containing the topic, the skill map and notes on misconceptions and coverage, which can be pasted into a new session, and "resume from:" restores that state and continues where it left off.

Why it matters

AI tutoring tends to fail in a familiar way: the model is agreeable, hands over answers, and mistakes fluent-sounding replies for genuine understanding. Jane Street's skill file is a detailed, open counterexample — a specification that treats teaching as an evaluation problem, with explicit placement, strict grading, spaced review and adversarial re-testing built into the prompt. Because it ships as a plain markdown file in a public repository, it doubles as a reusable artifact: developers can read the rules, adapt the difficulty or grading thresholds, and borrow the save-and-resume state mechanism for their own agent setups. For anyone building educational features on top of language models, it is a concrete reference for how much structure a prompt needs before a chatbot starts behaving like a tutor.