An Empty Rule That Passed the Schema

When four LLM calls feed into each other, the hardest failure to catch is a step that returns something plausible but wrong. I don't notice it until three steps later when the pipeline is already building a validation rule from it.

After rebuilding everything, each step has a Pydantic schema. The model's response goes right into Pydantic with model_validate_json. If step 1 returns a classification that does not match its schema, the edit fails there and step 2 never runs.

Step 3 generates the rule logic and its schema has a lot to check. The system can produce three kinds of rules: VALIDATE, FOREACH, and WHEN. A FOREACH rule names a loop and can carry accumulations while a WHEN rule has trigger conditions. The model can also give a fourth answer, UNKNOWN, for an edit the rule format cannot represent yet. That led to modelling all four as a discriminated union on a rule_type field:

Step3LogicResult = Annotated[
    Union[ValidateLogic, ForeachLogic, WhenLogic, DeclineLogic],
    Field(discriminator="rule_type"),
]

The model has to pick a rule type and then fill in that type's required fields. If it says FOREACH but leaves out the loop, Pydantic rejects it and names the missing field.

I added UNKNOWN later after a failure the schema missed. Some edits need something the rule format does not support yet. Before UNKNOWN existed, the model returned an empty VALIDATE rule for those edits. That passed the schema, then broke when the rule was built and the edit was lost. The model can now decline but it has to say which part of the format is missing and the edit is saved as a known gap. If the model declines and names something the format already supports, I log it as a failure.

A failed parse stops the edit without a retry and output that fits the schema gets a second check when the rule is built and run through the rule validator. If the validator finds errors, they are fed back as guidance and the pipeline runs again. If the rule still doesn't validate, the edit is reported as failed and no rule is written.

Each request now carries the step's schema. On Anthropic it goes in output_config.format and on OpenAI it's a strict json_schema response format so the model can only return JSON that fits it. Pydantic still parses every response because the schema would let an empty VALIDATE rule through.

In the earlier LangChain version only step 3 was checked and only for required fields. Now every step is one line of Pydantic and the empty VALIDATE case that used to slip through is rejected before the next step runs.