Agents That Actually Learn

The only benchmark that measures tool-use quality at the call level — arg_quality, sequence, recovery, and call_efficiency, all programmatically with no LLM-as-judge. 15 multi-turn scenarios across coding, data, and MCP domains. CannyForge lifts arg_quality from 0.837 to 1.000.

Try It Now

FSI-80 Benchmark

15 multi-turn scenarios across coding, data, and MCP domains. 5 scoring dimensions, 6 failure-mode detectors, 4 ablation conditions, Pass^k reliability.

Automatic Learning

Observes tool errors, clusters failures by tool and type, generates scoped corrections. EIR/ECR tracking with stability gate and auto-pruning.

LangGraph Middleware

3-line integration. before_model injects corrections. after_model records errors. finalize_task tracks effectiveness.

Framework Support

LangGraph (full correction loop), LangChain + CrewAI (skill wrappers), MCP (tool protocol). OpenAI Agents SDK planned.

Programmatic Scoring

No LLM-as-judge. Deterministic scoring across tool_selection, arg_quality, sequence, recovery, and call_efficiency.

Pass^k Reliability

Measures probability of succeeding K consecutive times. The honest metric for agents that should get more reliable, not just peak once.

Quick Start

                
# Install
pip install cannyforge

# Run the demo
python benchmark/scenario_harness.py \
    --model deepseek-v4-flash --no-think \
    --domains coding data mcp --passk 3

# Or use in code (3-line LangGraph integration)
from cannyforge import CannyForge
from cannyforge.adapters.langgraph import CannyForgeMiddleware
from langgraph.prebuilt import create_react_agent

forge = CannyForge()
middleware = CannyForgeMiddleware(forge)
agent = create_react_agent(model, tools,
    pre_model_hook=middleware.before_model,
    post_model_hook=middleware.after_model)
# Agent learns from tool errors automatically
                
            