SKU AGT-4003 · Sold by External
Gorilla APIBench
Product specifications
| SKU | AGT-4003 |
|---|---|
| Data type | Agent traces |
| Volume | 17003 instruction-API-call pairs (15,218 train + 1,785 eval across HuggingFace/TorchHub/TensorHub), over 1,726 reference APIs |
| Size on disk | ~43 MB |
| Format | JSON Lines (one JSON object per line) - {hub}_train.json / {hub}_eval.json for instruction pairs; {hub}_api.jsonl for the reference API database |
| Access model | PUBLIC LICENSE |
| Pricing | Free · open-source dataset (Apache-2.0) |
| Quality score | — |
| License | Apache License 2.0 |
APIBench is the dataset introduced with Gorilla (UC Berkeley & Microsoft Research) to teach and evaluate an LLM's ability to select and correctly invoke machine-learning APIs. It pairs a natural-language instruction with a single reference API call plus explanation and runnable code, drawn from three model hubs: Hugging Face, TorchHub, and TensorFlow Hub (TensorHub). Each hub has separate train and eval splits; the reference API database (the 'API Store') lists the canonical call signature, arguments, environment requirements, example code, and performance metrics for every API. Instructions were synthesized with GPT-4 via self-instruct (10 instructions per API). Model outputs are graded by parsing the generated call into an abstract syntax tree and checking whether the reference API's AST is a sub-tree of it (functional correctness), while calls matching no known API are counted as hallucinations. Code, data, and models are released under Apache-2.0 at github.com/ShishirPatil/gorilla.