Multi-step questions: decompose answer slots and dependencies

Last updated: 7 Oct 20265 min read

tutorial

A compound question can require several facts from different passages. Represent the needed slots and their dependency order before writing an answer.

Identify the missing facts

“Which rollback rule applies to the service that failed after the second retry?” requires an event-to-service fact and then a service-to-rule fact. Searching only the whole sentence can retrieve a plausible rule for the wrong service. Define answer slots for event identity, service ID, active runbook revision and rollback rule. Mark which slot depends on another. Dialogue slots record user-provided state; a multi-step question plan also records evidence needed from documents.

Bound each subquestion

Each step should state its input facts, retrieval scope and output type. The first may resolve a log event to gateway-west; the second may search gateway-west runbook passages under the active revision. Keep inferred facts as proposed until a source span verifies them. Do not let an early guess silently become a hard filter that prevents finding the correct later evidence. Log event identity can supply the first link in an operational case.

Detect cycles and missing steps

A plan with a missing prerequisite cannot produce a supported final answer. A cycle such as “resolve service from rule” and “resolve rule from service” needs a new anchor or a broader search, not repeated blind retrieval. Keep plan revisions and reviewer edits so failures are attributable. A fixed maximum number of steps limits cost, but it should not convert an incomplete answer into a confident one. The code below checks whether every declared dependency is present and orders steps when possible.

Test the decomposition

Evaluate slot coverage, dependency correctness, source support for each step and final answer accuracy separately. Include one-hop distractors that look answer-ready and cases where the second passage contradicts the first. Multi-passage conflict review handles disagreements among retrieved facts; the chain contract ensures that a seemingly complete path actually joins on the same entity and revision.

Implementation

def ordered_question_steps(dependencies):
    remaining = {step: set(parents) for step, parents in dependencies.items()}
    unknown = set().union(*remaining.values()) - remaining.keys() if remaining else set()
    if unknown:
        raise ValueError("unknown prerequisite: " + ", ".join(sorted(unknown)))
    ordered = []
    while remaining:
        ready = sorted(step for step, parents in remaining.items() if not parents)
        if not ready:
            raise ValueError("cyclic question plan")
        for step in ready:
            ordered.append(step)
            del remaining[step]
        for parents in remaining.values():
            parents.difference_update(ready)
    return ordered

plan = {"resolve_event": set(), "identify_service": {"resolve_event"},
        "lookup_rule": {"identify_service"}}
assert ordered_question_steps(plan) == [
    "resolve_event", "identify_service", "lookup_rule"]

Performance and operating cost

This compact implementation scans remaining steps repeatedly and can take O(s² + e) time for s steps and e dependencies, with O(s + e) space. Plans are usually short; a queue-based topological sort is preferable if hundreds of steps are expected. The order check does not prove that any subquestion has correct evidence.

Common Mistakes

Searching a compound question as if one passage must answer every part.
Using an unverified first-step guess to filter later search.
Treating a cyclic plan as evidence that the question was answered.
Reporting a final answer while a required slot remains unresolved.