Gorilla Deep Dive

Series · 1 posts

← Back to Paper Reading

  • Gorilla: Turn a Large API Catalog into Retrievable Tools, but APIBench Does Not Establish MCP Product Behavior

    Intermediate Agent runtime, safety, and evaluation
    Gorilla Deep Dive: Part 1 , Note: Aug 27, 2026 , Paper: 2024 , NLP

    A source-grounded reading of Patil et al., NeurIPS 2024: retriever-aware finetuning of LLaMA-7B on APIBench (TorchHub / TensorHub / HuggingFace) so catalog-scale API calls can be retrieved and checked. Zero-shot overall accuracy and hallucination beat prompted GPT-4 on that table—this is not a ReAct loop, MidTool mid-training, or RAG-MCP product routing.

    Understand it in 90 seconds
    Problem
    LLMs writing API calls often hallucinate names, arguments, and usage. The real world is not five fixed tools; it is a huge API catalog whose documentation changes frequently.
    Core insight
    Treat tool use as retrieve then call. Build instruction–API pairs on APIBench with self-instruct, then finetune LLaMA-7B in a retriever-aware way (RAT) so the model learns to read the documentation after Use this API documentation for reference: and emit a correct call.
    Strongest evidence
    NeurIPS Table 1. Gorilla zero-shot overall on TorchHub / HuggingFace / TensorFlow Hub is 59.13% / 71.68% / 83.79%, with hallucination 6.98% / 10.95% / 5.40%. GPT-4 zero-shot on the same table is 38.70% / 19.80% / 18.20% overall and 36.55% / 37.16% / 78.65% hallucination. Figure 6 shows that when documents change at test time, a RAT-trained model changes its call.
    Main boundary
    The corpus is ML-hub model-card / API JSON, not an arbitrary REST product catalog. Evaluation is single-call AST subtree matching, not a multi-step agent loop. A weak retriever can hurt (Table 2). Do not write APIBench numbers into MidTool or RAG-MCP.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact