Benchmark summary
A BFCL-based tool-calling benchmark ported to the SC# evaluation runtime. It tests whether the model selects the correct function, uses the right arguments, and avoids calling tools that do not match the request. Each test can see all tools declared in its script, so the model often has more tools to choose from than in the original case.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Tool Calling - Irrelevance: Architecture & Artwork
8 passed1 failed |
88.9% |
IN39 614 OUT6 994 | |
| ✓ | NoToolForApartmentRentCostBFCL/irrelevance_158: apartment rental cost asked; only a new-construction cost estimator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 461 OUT270 |
| ✓ | NoToolForBestJazzSingerBFCL/irrelevance_164: best Jazz singer asked; only a top-instrument-players search (singers are not instrument players) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 456 OUT828 |
| ✓ | NoToolForCathedralCeilingDesignBFCL/irrelevance_157: how to design a cathedral ceiling asked; only a lookup of facts about an existing building is available (retrieves data, does not advise on design) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 457 OUT700 |
| ✓ | NoToolForHolyBookDefinitionBFCL/irrelevance_144: definition of a holy book asked; only a search-within-a-named-holy-book tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 455 OUT379 |
| ✓ | NoToolForLectureTeacherBFCL/irrelevance_167: teacher of upcoming lectures asked; only an upcoming-concert details lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 458 OUT299 |
| ✓ | NoToolForMostFamousSculptureBFCL/irrelevance_153: most famous sculpture asked; only a sculpting-tools lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 463 OUT1 061 |
| ✓ | NoToolForPianoMaintenanceTipsBFCL/irrelevance_166: piano maintenance tips asked; only an instrument rental-prices lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 458 OUT519 |
| × | NoToolForScreamArtistBFCL/irrelevance_159: artist behind 'The Scream' asked. NOTE: the only tool (artwork_search) requires a museum_location that the user did not provide, so it cannot be correctly called; BFCL labels this irrelevant. Kept faithfully with ground truth = no tool call (flagged as borderline).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 943 OUT1 876 |
| ✓ | NoToolForSculptureRockTypeBFCL/irrelevance_152: which rock suits a garden sculpture asked; only a sculpture 3D-model creator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 463 OUT1 062 |
Tool Calling - Irrelevance: Buildings, Sculpture & Music
6 passed1 failed1 skipped |
85.7% |
IN26 291 OUT5 125 | |
| – | NoToolForCelloInstrumentTypeBFCL/irrelevance_165: instrument classification of a cello asked. NOTE: BFCL labels this irrelevant, but the only tool (get_instrument_info with detail=type) arguably answers it; flagged for review as a borderline BFCL item. Kept faithfully with ground truth = no tool call.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_instrument_info' with detail='type' answers 'what type of instrument is a cello' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForDeclarationSignersBFCL/irrelevance_137: who signed the declaration asked; only a historical-event DATE lookup is available (returns a date, not the signatories) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 282 OUT619 |
| ✓ | NoToolForEiffelTowerHeightBFCL/irrelevance_156: Eiffel Tower height asked; only an architecture-plan generator (creates a new design, does not retrieve facts) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 286 OUT674 |
| ✓ | NoToolForFamousDancersBFCL/irrelevance_162: famous dancers asked; only a museum-visual-artists lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 288 OUT753 |
| × | NoToolForReformationInitiatorBFCL/irrelevance_145: who initiated the Reformation asked; only a religious-event YEAR lookup is available (returns a year, not a person) -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 583 OUT1 573 |
| ✓ | NoToolForScientistDefinitionBFCL/irrelevance_143: definition of "scientist" asked; only a named-historical-figure info lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 278 OUT479 |
| ✓ | NoToolForSellingGuitarBFCL/irrelevance_163: how to sell a guitar asked; only an instrument-tuning tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 283 OUT707 |
| ✓ | NoToolForSimilarBlueColorBFCL/irrelevance_148: a color similar to blue asked; only a complementary-color tool (opposite, not similar) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 291 OUT320 |
Tool Calling - Irrelevance: Cell Biology & Genetics
7 passed1 failed2 skipped |
87.5% |
IN34 151 OUT9 211 | |
| ✓ | NoToolForClottingBloodCellsBFCL/irrelevance_32: blood cell type responsible for clotting asked; only a nucleus-count-based cell-type classifier is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 790 OUT421 |
| ✓ | NoToolForCommonAncestorBFCL/irrelevance_41: common ancestor of two animals asked (phylogenetics knowledge); only a fibonacci-sequence calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 789 OUT677 |
| ✓ | NoToolForEvolutionaryAlgorithmInGameAIBFCL/irrelevance_43: how to apply an Evolutionary Algorithm in game AI asked (advice); only a creature-evolution simulator is available (wrong context).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 790 OUT984 |
| ✓ | NoToolForGeneVariantFrequencyBFCL/irrelevance_36: gene-variant frequency in a population asked; only a dominant-trait-from-alleles tool is available (the true-match variant-frequency tool is deliberately isolated in A4).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 795 OUT1 028 |
| – | NoToolForIdentifyingGeneticSequenceBFCL/irrelevance_33: interpret a genetic code sequence asked; only a species-identification-from-sequence tool is available (returns a species, not what the sequence means).0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'identify_species' (identifies a species from a genetic sequence) plausibly answers the query about a DNA sequence, so a competent model may call it - BFCL-mislabeled/borderline-relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForIncreasingDeerPopulationBFCL/irrelevance_39: how-to advice on growing an animal population asked; only a population-growth CALCULATOR (needs numeric rates) is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 789 OUT609 |
| ✓ | NoToolForLionMatingProcessBFCL/irrelevance_35: mating process of a species asked; only a dominant/recessive genetic-traits lookup is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 786 OUT1 641 |
| ✓ | NoToolForMitochondriaCountBFCL/irrelevance_29: number/count of mitochondria asked; only a cell-part FUNCTION lookup is available (wrong query direction).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 790 OUT1 449 |
| × | NoToolForPandaEvolutionaryHistoryBFCL/irrelevance_42: evolutionary history asked (a narrative fact); only a biodiversity-index calculator is available.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN7 622 OUT2 402 |
| – | NoToolForWhaleGeneSequenceBFCL/irrelevance_44: the actual gene sequence of a species asked; only a gene-sequence GENERATOR (needs numeric mutation rate + duration) is available.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'gene_sequencer' (generate gene sequences for evolutionary changes) directly answers the query - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Irrelevance: Color, Geometry & Population
11 passed |
100.0% |
IN40 836 OUT6 865 | |
| ✓ | NoToolForColorInComputerVisionBFCL/irrelevance_131: color meaning in computer vision asked; only a color-in-personality-psychology analyzer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 712 OUT1 283 |
| ✓ | NoToolForDeclarationSigningDateBFCL/irrelevance_135: signing date asked; only date-arithmetic (add days) and age tools are available, neither retrieves an event date -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 710 OUT594 |
| ✓ | NoToolForElectricityDiscovererBFCL/irrelevance_140: who discovered electricity asked; only an electric-current calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 706 OUT352 |
| ✓ | NoToolForHexColorMixBFCL/irrelevance_146: mixing two hex colors asked; only a prophet-details lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 722 OUT1 194 |
| ✓ | NoToolForIndependenceSigningDateBFCL/irrelevance_138: signing date asked; only an age calculator (needs birthdate + current date) and a date-adder are available, neither retrieves an event date -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 710 OUT437 |
| ✓ | NoToolForMostVisitedMarketBFCL/irrelevance_161: most visited market asked; only a museum visitation-statistics lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 712 OUT592 |
| ✓ | NoToolForNbaFinalWinnerBFCL/irrelevance_133: NBA final winner asked; only a historical-battle details lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 713 OUT320 |
| ✓ | NoToolForPicassoPaintingYearBFCL/irrelevance_151: year of a Picasso painting asked; only a sculpture-dimensions lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 711 OUT1 180 |
| ✓ | NoToolForRelativityAuthorBFCL/irrelevance_142: who proposed special relativity asked; only a chemical-element properties lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 715 OUT188 |
| ✓ | NoToolForSculptureCreatorBFCL/irrelevance_155: sculpture creator asked; only a max-seating-capacity calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 711 OUT399 |
| ✓ | NoToolForWhichColorsToMixBFCL/irrelevance_150: which colors to mix for a shade asked; only an RGB-to-Pantone converter is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 714 OUT326 |
Tool Calling - Irrelevance: Cooking, Currency & Markets
6 passed2 failed |
75.0% |
IN32 465 OUT7 898 | |
| × | NoToolForBakerySourdoughBFCL/irrelevance_223: a BAKERY selling sourdough asked; only a grocery-shop product finder available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 506 OUT1 325 |
| ✓ | NoToolForChickenRecipeBFCL/irrelevance_211: the RECIPE to cook chicken asked; only a cook-TIME calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 241 OUT1 069 |
| ✓ | NoToolForChocolateCakeIngredientsBFCL/irrelevance_220: cake INGREDIENTS asked; only a cooking-TEMPERATURE tool available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 244 OUT472 |
| ✓ | NoToolForCurrencyToGbpBFCL/irrelevance_233: CURRENCY conversion asked; only a WEATHER tool available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 246 OUT631 |
| ✓ | NoToolForHowToBoilAnEggBFCL/irrelevance_212: best WAY to boil an egg asked; only a boiling-TIME calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 243 OUT958 |
| ✓ | NoToolForLasagnaRecipeBFCL/irrelevance_219: the RECIPE for lasagna asked; only a recipe-CALORIE calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 241 OUT891 |
| ✓ | NoToolForLegExercisesBFCL/irrelevance_221: leg EXERCISES asked; only a grocery-list generator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 241 OUT561 |
| × | NoToolForPetStoreBFCL/irrelevance_224: a PET store asked; only a GROCERY-store locator available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 503 OUT1 991 |
Tool Calling - Irrelevance: Dates & Elements
10 passed |
100.0% |
IN36 270 OUT6 331 | |
| ✓ | NoToolForHydrogenPropertiesBFCL/irrelevance_141: chemical properties of Hydrogen asked; only a scientist-contributions lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 626 OUT633 |
| ✓ | NoToolForLargestPlanetBFCL/irrelevance_139: largest planet asked; only a star-info lookup (needs a star name) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 626 OUT818 |
| ✓ | NoToolForMostImportantProphetBFCL/irrelevance_147: most important prophet asked; only a two-color mixer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 626 OUT482 |
| ✓ | NoToolForMuseumVisitFrequencyBFCL/irrelevance_160: museum member visit frequency asked; only a single most-frequent-visitor lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 631 OUT921 |
| ✓ | NoToolForPantoneOfSkyBlueBFCL/irrelevance_149: Pantone code of sky blue asked; only a paint-proportions calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 628 OUT1 181 |
| ✓ | NoToolForPersonalityTypeBFCL/irrelevance_130: user asks what personality type they are; only a big-five calculator (which needs the user's rated answers) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 624 OUT727 |
| ✓ | NoToolForStadiumSeatingCapacityBFCL/irrelevance_154: stadium seating capacity asked; only a sculpture-creator lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 627 OUT336 |
| ✓ | NoToolForVicePresidentBFCL/irrelevance_136: Vice President asked; only a US-president-by-year lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 626 OUT371 |
| ✓ | NoToolForWaterlooCasualtiesBFCL/irrelevance_132: casualty count of a battle asked; only a historical-event date lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 628 OUT543 |
| ✓ | NoToolForWorldCupWinnerBFCL/irrelevance_134: sports World Cup winner asked; only a historical-battle outcome predictor is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 628 OUT319 |
Tool Calling - Irrelevance: Electromagnetism, Thermo & Data
9 passed1 failed |
90.0% |
IN37 489 OUT7 431 | |
| ✓ | NoToolForElectronRestMassBFCL/irrelevance_23: mass of an electron asked (a knowledge fact); only magnetic-field / current tools are available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 397 OUT266 |
| ✓ | NoToolForFlightDurationBFCL/irrelevance_20: flight duration asked; only a currency converter is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 400 OUT409 |
| ✓ | NoToolForFreezingPointBFCL/irrelevance_25: freezing point asked; only boiling-point calculators are available (wrong phase transition).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 407 OUT1 501 |
| × | NoToolForGasAmountGeneratedBFCL/irrelevance_26: amount of gas generated asked; only a gas-PRESSURE calculator is available (wrong quantity).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 864 OUT1 221 |
| ✓ | NoToolForGiniCoefficientBFCL/irrelevance_48: Gini coefficient asked; only a GDP-fetch tool is available (different economic indicator).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 400 OUT286 |
| ✓ | NoToolForHeatingWaterBFCL/irrelevance_27: heat energy to warm water asked. NOTE: the original tool (calculate_heat) is a fair match for this query (only the well-known specific heat is missing), which would make "no call" ambiguous; per human-baseline fairness it is deliberately NOT shown here (kept as a distractor in A3), so no visible tool fits and "no call" is unambiguously correct.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 411 OUT640 |
| ✓ | NoToolForHexagonSidesBFCL/irrelevance_28: number of sides of a hexagon asked (basic geometry knowledge); only a boiling-point tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 398 OUT179 |
| ✓ | NoToolForSymbolicMagneticFieldBFCL/irrelevance_22: magnetic field asked with SYMBOLIC values (distance 'r', current 'I'); the two magnetic-field tools require concrete numeric distance + current, so there is nothing to pass -> no fit.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 411 OUT2 234 |
| ✓ | NoToolForTimeDifferenceBFCL/irrelevance_49: time-zone difference asked; only a geo-coordinate lookup tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 400 OUT492 |
| ✓ | NoToolForWorldSeriesWinnerBFCL/irrelevance_53: sports trivia asked; only a SQL database-query runner is available (no database/query provided).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 401 OUT203 |
Tool Calling - Irrelevance: Entertainment, Sports & Knowledge
9 passed |
100.0% |
IN31 410 OUT3 833 | |
| ✓ | NoToolForBasketballResultBFCL/irrelevance_84: basketball game result asked; only a stock-price retrieval tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 490 OUT405 |
| ✓ | NoToolForBestSoccerPlayerBFCL/irrelevance_79: best current soccer player asked; only a stock-market prediction tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 488 OUT411 |
| ✓ | NoToolForCaliforniaGovernorBFCL/irrelevance_87: current Governor of California asked; only a court-case filing tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 486 OUT290 |
| ✓ | NoToolForCrimeThrillerMovieListBFCL/irrelevance_88: best crime-thriller movies of 2020 asked; only a document forgery detector is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 495 OUT1 226 |
| ✓ | NoToolForFifaWorldCupWinnerBFCL/irrelevance_80: FIFA World Cup 2010 winner asked; only a stock-ticker lookup is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 491 OUT261 |
| ✓ | NoToolForFrenchTranslationBFCL/irrelevance_61: French translation of a word asked; only an SVM-model tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 490 OUT236 |
| ✓ | NoToolForHorrorMovieListBFCL/irrelevance_77: horror movie recommendations asked; only an investment future-value tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 489 OUT630 |
| ✓ | NoToolForPresidentialElectionWinnerBFCL/irrelevance_85: 2020 presidential election winner asked; only a criminal-case detail lookup is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 491 OUT192 |
| ✓ | NoToolForSportsNewsBFCL/irrelevance_76: latest sports news asked; only a portfolio investment-trend analyzer is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 490 OUT182 |
Tool Calling - Irrelevance: Finance & Economics
7 passed1 failed |
87.5% |
IN30 189 OUT5 093 | |
| ✓ | NoToolForAverageHousePriceBFCL/irrelevance_69: market-wide average house price asked; only a regression predictor (needs a trained model + feature vector, not provided) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 346 OUT277 |
| ✓ | NoToolForCompoundInterestBFCL/irrelevance_70: compound-interest computation asked; only a mortgage-payment calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 372 OUT1 130 |
| ✓ | NoToolForGoldPriceBFCL/irrelevance_78: today's gold spot price asked; only a bond-price calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 348 OUT441 |
| ✓ | NoToolForHorrorMovieCatalogBFCL/irrelevance_81: horror movie recommendations asked; only a historical stock-price fetcher is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 349 OUT350 |
| × | NoToolForLossProjectionBFCL/irrelevance_73: a loss projection asked; only a revenue-prediction tool is available (revenue != loss).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 714 OUT1 947 |
| ✓ | NoToolForPoundToKilogramBFCL/irrelevance_75: a mass unit conversion asked; only a portfolio-returns calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 347 OUT266 |
| ✓ | NoToolForProfitMarginBFCL/irrelevance_71: profit margin asked; only an ROI calculator is available (ROI != margin) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 369 OUT489 |
| ✓ | NoToolForStockNewsBFCL/irrelevance_82: stock news asked; only a capital-gains calculator is available (gains != news) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 344 OUT193 |
Tool Calling - Irrelevance: Finance Near-Matches
9 passed |
100.0% |
IN30 921 OUT4 766 | |
| ✓ | NoToolForBitcoinPriceBFCL/irrelevance_56: current bitcoin price asked; only a generic database record-select tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 426 OUT289 |
| ✓ | NoToolForBookstoreLookupBFCL/irrelevance_55: which online bookstore sells a title asked; only an add-to-cart tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 431 OUT521 |
| ✓ | NoToolForBusinessRateOfReturnBFCL/irrelevance_74: rate of return asked; only a net-profit calculator is available (profit != rate of return).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 447 OUT1 281 |
| ✓ | NoToolForCoinFlipProbabilityBFCL/irrelevance_66: probability of a coin landing heads asked; only a standard-deviation tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 431 OUT333 |
| ✓ | NoToolForCurrentInterestRateBFCL/irrelevance_83: the current market interest rate asked; only a mortgage-payment calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 425 OUT305 |
| ✓ | NoToolForExternalRateOfReturnBFCL/irrelevance_72: EXTERNAL rate of return asked; only an internal-rate-of-return (IRR) tool is available (ERR != IRR, even though cash flows are given) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 455 OUT1 076 |
| ✓ | NoToolForFinalPriceWithoutBasePriceBFCL/irrelevance_60: final price after discount+tax asked, but the original price is never given, so the (domain-matching) calculateFinalPrice cannot be correctly called -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 445 OUT126 |
| ✓ | NoToolForFrontendDevelopmentBFCL/irrelevance_62: how to build an e-commerce frontend asked; only a recommender-model builder is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 431 OUT583 |
| ✓ | NoToolForHighestGrossingMovieBFCL/irrelevance_54: highest-grossing movie of all time asked; only a movie search (requires a specific title + year, not provided) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 430 OUT252 |
Tool Calling - Irrelevance: Food, Time & Distance
2 passed2 failed1 skipped |
50.0% |
IN16 304 OUT4 245 | |
| × | NoToolForCakeBakingMachineBFCL/irrelevance_218: a baking MACHINE/appliance asked; only a recipe retriever available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN5 458 OUT2 211 |
| ✓ | NoToolForDrivingTravelTimeBFCL/irrelevance_215: driving TRAVEL TIME asked; only a restaurant-bill tip calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 710 OUT524 |
| × | NoToolForLondonTimezoneNameBFCL/irrelevance_226: the NAME of London's timezone asked; only a local-TIME-by-timezone tool available (it needs the timezone as input and returns a time, not the timezone name) -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN5 422 OUT1 076 |
| – | NoToolForPizzaInBostonBFCL/irrelevance_213: a place for pizza asked; a cuisine+location restaurant_finder is available. REVIEW FLAG (fairness): restaurant_finder(cuisine=pizza/Italian, location=Boston) plausibly answers this, so a competent human might legitimately call it. BFCL labels it irrelevant; kept as such and marked Hard because the intended "no call" is debatable. No OTHER query here is a restaurant search, so the tool cannot wrongly satisfy the file's other cases.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'restaurant_finder' (cuisine+location) answers 'good place for pizza in Boston' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForTriangleHypotenuseBFCL/irrelevance_238: the HYPOTENUSE of a triangle asked; only a geo lat/long distance calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 714 OUT434 |
Tool Calling - Irrelevance: Geo, Travel & Sentiment
8 passed1 skipped |
100.0% |
IN27 171 OUT4 047 | |
| ✓ | NoToolForAverageTemperatureBmiBFCL/irrelevance_110: an average December temperature asked; only a BMI calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 396 OUT734 |
| ✓ | NoToolForCaseLawyerBFCL/irrelevance_95: the lawyer of a case asked; only a case-outcome retriever is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 399 OUT352 |
| ✓ | NoToolForCurrentTimeSunriseBFCL/irrelevance_102: the current time asked; only a sunrise/sunset time calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 395 OUT170 |
| ✓ | NoToolForFamousQuoteBFCL/irrelevance_119: a famous quote asked; only a sentiment-analysis tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 393 OUT342 |
| ✓ | NoToolForFastestRouteBFCL/irrelevance_114: the fastest route asked; only a closest-airport finder is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 397 OUT1 009 |
| – | NoToolForRestaurantsInLondonBFCL/irrelevance_113: restaurants in London asked; only a generic points-of-interest locator is available -> no tool fits.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'find_pois' (locate points of interest by category) answers 'restaurants in London' (restaurants are POIs) - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForSportsBrainImpactBFCL/irrelevance_120: the neurological impact of sports asked; only a caffeine neurological-impact tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 396 OUT302 |
| ✓ | NoToolForTrafficConditionBFCL/irrelevance_112: the current traffic condition asked; only an address geocoder is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 395 OUT265 |
| ✓ | NoToolForTravelTimeByCarBFCL/irrelevance_115: travel time by car asked; only a coordinate-distance calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 400 OUT873 |
Tool Calling - Irrelevance: Legal & Case Lookup
9 passed1 failed |
90.0% |
IN39 073 OUT5 076 | |
| ✓ | NoToolForAppleCaloriesBFCL/irrelevance_100: calories in an apple asked; only a litigation-cost calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 548 OUT177 |
| ✓ | NoToolForBrainWeightBFCL/irrelevance_122: average human brain weight asked; only a per-patient neural-activity tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 550 OUT480 |
| ✓ | NoToolForCaseJudgementBFCL/irrelevance_93: a case judgement asked; only a law-firm impactful-cases lister is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 548 OUT406 |
| ✓ | NoToolForChineseRestaurantBFCL/irrelevance_97: a Chinese-restaurant recommendation asked; only a lawsuit-filing tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 551 OUT507 |
| ✓ | NoToolForMovieRecommendationBFCL/irrelevance_116: a movie recommendation asked; only a word-count tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 549 OUT346 |
| × | NoToolForNbaChampionBFCL/irrelevance_118: the 1996 NBA champion asked; only an NBA playoff-brackets display tool is available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN7 117 OUT860 |
| ✓ | NoToolForPaintingEiffelSettlementBFCL/irrelevance_98: how long to paint the Eiffel Tower asked; only a lawsuit-settlement estimator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 552 OUT740 |
| ✓ | NoToolForSecurityGuardSearchBFCL/irrelevance_92: finding a security guard asked; only a lawyer-search tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 558 OUT258 |
| ✓ | NoToolForSphereVolumeBFCL/irrelevance_104: sphere volume asked; only a circular-park area tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 552 OUT1 016 |
| ✓ | NoToolForTrafficLawsBFCL/irrelevance_99: traffic laws asked; only a lawsuit-search tool is available (lawsuits, not laws) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 548 OUT286 |
Tool Calling - Irrelevance: Math & Science Distractors
8 passed |
100.0% |
IN25 236 OUT5 034 | |
| ✓ | NoToolForDefiniteIntegralBFCL/irrelevance_9: definite integral asked; only an income-tax calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 157 OUT1 070 |
| ✓ | NoToolForIntegralOfCubeBFCL/irrelevance_8: definite integral asked; only a string-to-int converter is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 150 OUT1 375 |
| ✓ | NoToolForLatestMoviesBFCL/irrelevance_15: latest movie releases asked; only a physics final-velocity tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 144 OUT281 |
| ✓ | NoToolForPerpendicularSlopeBFCL/irrelevance_3: slope of a perpendicular line asked; only a critical-points finder is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 162 OUT540 |
| ✓ | NoToolForPrimeFactorsBFCL/irrelevance_13: prime factorization asked; only a compound-interest calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 147 OUT380 |
| ✓ | NoToolForRectangleAreaBFCL/irrelevance_6: rectangle area asked; only a circle-drawing tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 158 OUT369 |
| ✓ | NoToolForTriangleAreaBFCL/irrelevance_0: triangle area asked; only a BMI tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 158 OUT393 |
| ✓ | NoToolForUpwardAccelerationBFCL/irrelevance_14: acceleration asked (it is just gravity); the tempting max-height and final-velocity tools compute neither the acceleration nor what is asked -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 160 OUT626 |
Tool Calling - Irrelevance: Math & Statistics
5 passed1 failed |
83.3% |
IN21 146 OUT5 287 | |
| ✓ | NoToolForBinomialMeanBFCL/irrelevance_67: the MEAN of a binomial experiment asked; only p-value tools are available (they return a p-value, not the mean) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 026 OUT541 |
| ✓ | NoToolForCivilLawComponentsBFCL/irrelevance_90: the components/categories of Civil Law asked; only a noise-complaint filing tool is available (filing a complaint does not explain law) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN2 998 OUT957 |
| ✓ | NoToolForCoinTossMaxHeadsBFCL/irrelevance_63: how many heads CAN be obtained from tossing 3 coins (a count/maximum, not a probability) asked; only a probability-of-an-event calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 002 OUT328 |
| × | NoToolForNullHypothesisStdDevBFCL/irrelevance_68: the STANDARD DEVIATION of a null-hypothesis test asked; only p-value tools are available (they return a p-value, not a standard deviation) -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 112 OUT2 204 |
| ✓ | NoToolForPowerOfTwoBFCL/irrelevance_59: raising a number to a power asked; only a linear-equation solver is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 001 OUT297 |
| ✓ | NoToolForQuadraticEquationBFCL/irrelevance_57: solving a quadratic equation asked; only a genetic-algorithm optimizer and a LINEAR equation solver are available (a quadratic is not linear) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 007 OUT960 |
Tool Calling - Irrelevance: Medical, Social & Personality
6 passed3 skipped |
100.0% |
IN20 452 OUT3 671 | |
| ✓ | NoToolForBigMacCaloriesBFCL/irrelevance_123: calories of a Big Mac asked; only a BMI calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 407 OUT378 |
| ✓ | NoToolForEconomicHappinessEffectBFCL/irrelevance_126: the effect of economic status on happiness asked; only a happiness-index lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 409 OUT1 231 |
| ✓ | NoToolForFriendMbtiBFCL/irrelevance_129: a friend's MBTI asked; only a personality-type evaluator (needs trait percentages) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 407 OUT464 |
| – | NoToolForLatestTechTrendBFCL/irrelevance_124: the latest technology trend asked; only a trending-topics-by-category tool is available -> no tool fits.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_social_trends' (trending topics by category) answers 'latest trend in technology' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForMajorAirportsBFCL/irrelevance_117: major US airports asked; only a coordinate-distance calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 410 OUT483 |
| – | NoToolForMotorNeuronDiseaseBFCL/irrelevance_121: motor neuron disease info asked; only a generic disease-info retriever is available -> no tool fits.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'medical_records_get_disease_info' directly answers 'information on motor neuron diseases' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForPopularBooksBFCL/irrelevance_125: popular books by an author asked; only a recent-tweets retriever is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 409 OUT603 |
| – | NoToolForTwitterMoodBFCL/irrelevance_127: overall twitter mood asked; only a twitter sentiment-analysis tool is available -> BFCL ground truth is no tool call.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'sentiment_analysis_twitter' directly answers 'general mood of twitter regarding the new iPhone' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForVegetableServingsBFCL/irrelevance_128: daily vegetable servings asked; only a personality-test scorer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 410 OUT512 |
Tool Calling - Irrelevance: Mixed Distractors
8 passed |
100.0% |
IN25 992 OUT6 833 | |
| ✓ | NoToolForAirQualityBFCL/irrelevance_40: current air quality asked; only a plant-biomass calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 250 OUT303 |
| ✓ | NoToolForFavoriteRestaurantBFCL/irrelevance_31: a favorite restaurant recommendation asked; only a cell-division simulator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 246 OUT190 |
| ✓ | NoToolForGlobalWarmingDocumentaryBFCL/irrelevance_38: a documentary (video) asked; only a scientific-research-paper retriever is available (wrong medium).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 247 OUT898 |
| ✓ | NoToolForLasagnaIngredientsBFCL/irrelevance_47: recipe ingredients asked; only a flight-timetable tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 247 OUT522 |
| ✓ | NoToolForLionDominantTraitBFCL/irrelevance_34: dominant genetic trait of a species asked; only a gene-variant-FREQUENCY lookup is available (different genetic output).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 249 OUT1 008 |
| ✓ | NoToolForMultinucleatedCellNameBFCL/irrelevance_30: name of a cell type with multiple nuclei asked; only a blood-cell classifier (by shape/size) is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 253 OUT586 |
| ✓ | NoToolForPicnicSpotBFCL/irrelevance_37: a picnic spot asked; only a local-fauna information tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 246 OUT3 031 |
| ✓ | NoToolForTranslationBFCL/irrelevance_52: natural-language translation asked; only a file-compression tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 254 OUT295 |
Tool Calling - Irrelevance: Motion, Electricity & Misc
10 passed |
100.0% |
IN35 408 OUT6 918 | |
| ✓ | NoToolForCarProjectileRangeBFCL/irrelevance_16: projectile-style "how far will a car travel" asked with symbolic v/t/theta; the range tool needs concrete numeric velocity + angle -> nothing to pass -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 551 OUT1 058 |
| ✓ | NoToolForConeVolumeBFCL/irrelevance_19: cone volume asked; only an investment-return calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 545 OUT618 |
| ✓ | NoToolForCurrentClockTimeBFCL/irrelevance_17: current clock time asked; only a travel-time (distance/speed) calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 534 OUT538 |
| ✓ | NoToolForElectronMassBFCL/irrelevance_24: mass of an electron asked (a knowledge fact); only an electric-current tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 535 OUT290 |
| ✓ | NoToolForFibonacciOf20BFCL/irrelevance_51: fibonacci number asked; only a cryptocurrency-price tool is available (the fibonacci calculator lives in a different file so it cannot satisfy this query).1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 536 OUT579 |
| ✓ | NoToolForForceAngleFromMomentumBFCL/irrelevance_18: angle of a force from momentum asked; only a vector-angle-from-XY-components tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 541 OUT1 754 |
| ✓ | NoToolForHorrorMovieListBFCL/irrelevance_50: movie recommendations asked; only a Celsius-to-Fahrenheit converter is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 541 OUT826 |
| ✓ | NoToolForMagneticFieldNearWireBFCL/irrelevance_21: magnetic field near a current-carrying wire asked; only an EM-wave-amplitude tool is available (magnetic-field calculators are deliberately kept in a different file).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 550 OUT736 |
| ✓ | NoToolForSineOf45DegreesBFCL/irrelevance_45: sine of an angle asked; only a polygon-drawing tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 536 OUT305 |
| ✓ | NoToolForTeslaPriceBFCL/irrelevance_46: product price of a car asked; only a currency-exchange-rate tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 539 OUT214 |
Tool Calling - Irrelevance: Music Theory & Lookup
8 passed |
100.0% |
IN25 572 OUT6 274 | |
| ✓ | NoToolForAncientGreekLawmakersBFCL/irrelevance_171: who established laws in Ancient Greece asked (history); only a musical-piece composer search is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 194 OUT593 |
| ✓ | NoToolForBasketballGameAtVenueBFCL/irrelevance_169: who is playing a basketball game at a venue asked; only concert-locating tools exist.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 197 OUT409 |
| ✓ | NoToolForBluesScalePatternBFCL/irrelevance_176: the interval pattern of a blues scale asked; only composer lookups are available (get_blues_scale lives in E2) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 199 OUT519 |
| ✓ | NoToolForDonQuixoteAuthorBFCL/irrelevance_172: author of the novel Don Quixote asked; only music-composition lookups exist -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 192 OUT696 |
| ✓ | NoToolForFamousUsComposersBFCL/irrelevance_170: most famous US composers asked (general knowledge, no specific piece); only a chord-progression creator + piece-specific composer lookups exist -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 195 OUT2 700 |
| ✓ | NoToolForGrammyBestAlbumBFCL/irrelevance_177: Grammy Award winner asked; only a song chord-progression lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 200 OUT448 |
| ✓ | NoToolForPrimaryTriadsOfKeyBFCL/irrelevance_173: primary triads of a key asked; only a genre-common-chords finder is available (primary_triads lives in E2) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 198 OUT620 |
| ✓ | NoToolForUniversityClassAvailabilityBFCL/irrelevance_168: availability of a university class asked; only concert-availability / concert-search tools are available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 197 OUT289 |
Tool Calling - Irrelevance: Probability & Legal
5 passed |
100.0% |
IN13 997 OUT4 400 | |
| ✓ | NoToolForFaceCardProbabilityBFCL/irrelevance_64: probability of drawing a face card from a deck asked; only a COIN-toss heads probability tool is available (coins, not cards) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 796 OUT391 |
| ✓ | NoToolForHomeElectricityNeedBFCL/irrelevance_58: home electricity demand (kWh) asked; only a solar-panel COUNT calculator is available (it sizes panels, not electricity need, and needs sunlight hours not given) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN2 799 OUT1 475 |
| ✓ | NoToolForNoiseComplaintProcessBFCL/irrelevance_91: how/whether to report a noise complaint asked; only a law-category listing tool is available (it lists legal categories, it does not file or advise on complaints) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 796 OUT628 |
| ✓ | NoToolForRedMarbleCountBFCL/irrelevance_65: number of red marbles asked; only a determine-population tool is available (it returns a total population from probability + sample size, not a sub-category count) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN2 811 OUT668 |
| ✓ | NoToolForTenantRightsBFCL/irrelevance_89: explaining tenant rights asked; only a contract-generation tool is available (generating a contract does not explain rights) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN2 795 OUT1 238 |
Tool Calling - Irrelevance: Recipes & Cooking
8 passed1 skipped |
100.0% |
IN27 642 OUT5 195 | |
| ✓ | NoToolForBestGameCharacterBFCL/irrelevance_206: best game character vs a dragon asked; only a best-WEAPON finder available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 459 OUT1 018 |
| ✓ | NoToolForBestMovie2020BFCL/irrelevance_235: best MOVIE of 2020 asked; only a currency-exchange calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 456 OUT982 |
| ✓ | NoToolForBuildingAPcBFCL/irrelevance_209: how to build a PC asked; only a cooking-recipe retriever available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 451 OUT948 |
| ✓ | NoToolForFamousPlaceInParisBFCL/irrelevance_210: most famous PLACE in Paris asked; only a restaurant-by-dish search available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 453 OUT451 |
| ✓ | NoToolForGameShortestPathBFCL/irrelevance_208: shortest in-game path asked; only a taxi-fare calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 461 OUT307 |
| ✓ | NoToolForInchToMeterBFCL/irrelevance_234: LENGTH conversion (inches to metres) asked; only a CURRENCY converter available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 452 OUT306 |
| ✓ | NoToolForLowestGameScoreBFCL/irrelevance_207: LOWEST game score asked; only a HIGHEST-score retriever available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 456 OUT864 |
| ✓ | NoToolForNewYorkPopulationBFCL/irrelevance_225: current POPULATION of New York asked; only a time-zone converter available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 454 OUT319 |
| – | NoToolForQuickestFlightLondonTokyoBFCL/irrelevance_236: quickest way to fly London->Tokyo asked; a flight-duration tool is available. REVIEW FLAG (fairness): get_flight_duration plausibly answers this ("quickest ... by plane" ~= quickest flight duration), so a competent human might legitimately call it. Kept as BFCL labels it (irrelevant), but this item is borderline. Marked Hard because the intended "no call" is debatable.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_flight_duration' (with flight_type non-stop) plausibly answers 'quickest way ... by plane' - BFCL-mislabeled/borderline-relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Irrelevance: Restaurants & Travel
7 passed1 failed |
87.5% |
IN29 862 OUT3 288 | |
| ✓ | NoToolForBeijingSunsetBFCL/irrelevance_227: SUNSET time asked; only a SUNRISE calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 313 OUT238 |
| ✓ | NoToolForClosestItalianRestaurantBFCL/irrelevance_216: FIND the closest Italian restaurant asked; only a currency converter available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 312 OUT183 |
| ✓ | NoToolForFindSushiRestaurantBFCL/irrelevance_214: FIND a sushi restaurant asked; only a tip calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 314 OUT186 |
| ✓ | NoToolForNearestPharmacyBFCL/irrelevance_237: FIND the nearest pharmacy asked; only a route-to-a-known-location planner available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 314 OUT372 |
| ✓ | NoToolForPetFriendlyTrainStationBFCL/irrelevance_230: a pet-friendly TRAIN STATION asked; only hotel search tools available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 313 OUT338 |
| ✓ | NoToolForSushiRestaurantsTokyoBFCL/irrelevance_229: FIND sushi RESTAURANTS asked; only hotel-booking tools available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 314 OUT334 |
| ✓ | NoToolForThaiRestaurantChicagoBFCL/irrelevance_231: FIND a Thai RESTAURANT asked; only a hotel-availability checker available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 315 OUT314 |
| × | NoToolForYogaRetreatBFCL/irrelevance_232: a yoga RETREAT asked; only a HOTEL search tool available. REVIEW FLAG (fairness): a retreat is not a hotel, but hotel_search_find_hotels accepts arbitrary amenities (spa, vegetarian) and a location, so it is somewhat tempting. BFCL labels this irrelevant (retreat != hotel); kept as such and marked Hard because the boundary is borderline.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 667 OUT1 323 |
Tool Calling - Irrelevance: Sports & Games (set 1)
10 passed |
100.0% |
IN35 688 OUT5 361 | |
| ✓ | NoToolForAssistAverageBFCL/irrelevance_185: assist average asked; only an average-SCORING tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 570 OUT1 034 |
| ✓ | NoToolForBestChessMoveBFCL/irrelevance_194: best next move asked; only a board-game-STATISTICS tool is available (the move-suggesting board_analyser lives in E4) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 572 OUT514 |
| ✓ | NoToolForChessArrangementCountBFCL/irrelevance_196: total theoretical arrangements asked; only a final-score calculator is available (possible_moves lives in E4) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 572 OUT612 |
| ✓ | NoToolForFootballTeamRankingBFCL/irrelevance_186: ranking of a team asked; only a season-MVP tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 568 OUT390 |
| ✓ | NoToolForIceCreamStoreBFCL/irrelevance_201: find a physical store asked; only a video-game guide is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 565 OUT227 |
| ✓ | NoToolForMostHeartsInPokerBFCL/irrelevance_199: who holds most of a suit asked; only a poker-dealing tool is available (get_highest_card_holder lives in E4) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 572 OUT382 |
| ✓ | NoToolForPlayerBiographyBFCL/irrelevance_189: a biographical "who is X" asked; only a top-ranked-athlete lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 565 OUT397 |
| ✓ | NoToolForTeamRosterBFCL/irrelevance_192: full roster asked; only a top-scorer tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 567 OUT613 |
| ✓ | NoToolForUnoRulesBFCL/irrelevance_198: the rules of a game asked; only a deck-shuffling tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 569 OUT754 |
| ✓ | NoToolForWorldCupWinnerBFCL/irrelevance_191: who won a tournament asked (no team named); only a per-team match-stats tool exists.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 568 OUT438 |
Tool Calling - Irrelevance: Sports & Games (set 2)
8 passed1 failed |
88.9% |
IN34 675 OUT5 770 | |
| ✓ | NoToolForBlackjackAceRuleBFCL/irrelevance_200: a rules question asked; only a poker highest-card-holder tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 467 OUT425 |
| ✓ | NoToolForBookAuthorBFCL/irrelevance_205: author of a book asked; only a video-game score calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 471 OUT289 |
| ✓ | NoToolForChessTournamentWinnerBFCL/irrelevance_195: who won a tournament asked; only a board-position move-suggester is available (the game-history fetch_game_stats lives in E3) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 468 OUT521 |
| ✓ | NoToolForCreateJigsawBFCL/irrelevance_204: CREATE a puzzle asked; only a puzzle-SOLVING tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 461 OUT471 |
| × | NoToolForHaloPlayerRankBFCL/irrelevance_203: a player's RANK asked; only a player-SCORE lookup is available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 948 OUT1 805 |
| ✓ | NoToolForLakersTopScorerBFCL/irrelevance_193: top scorer of a team asked; only a team-details tool (roster/results/upcoming, not scorers) is available (sports_team.get_top_scorer lives in E3) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 466 OUT1 155 |
| ✓ | NoToolForMonopolyWinnerBFCL/irrelevance_197: who won a casual game asked; only a possible-moves calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 466 OUT324 |
| ✓ | NoToolForTopTennisPlayerBFCL/irrelevance_190: top ranked PLAYER asked; only a TEAM standing/ranking tool is available (the athlete-ranking get_top_ranked lives in E3) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 465 OUT575 |
| ✓ | NoToolForWorldSeriesWinnerBFCL/irrelevance_202: who won a real game asked; only a score calculator (from raw points) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 463 OUT205 |
Tool Calling - Irrelevance: Sports Statistics
9 passed |
100.0% |
IN31 560 OUT6 838 | |
| ✓ | NoToolForBasketballSeasonMvpBFCL/irrelevance_187: who won the MVP asked; only a team-ranking tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 509 OUT724 |
| ✓ | NoToolForClippersRosterBFCL/irrelevance_179: roster of a team asked; only a two-team game-results tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 502 OUT866 |
| ✓ | NoToolForCommonChordsInPopBFCL/irrelevance_174: most common chords in a pop song asked; only a key-signature primary-triads tool is available (find_common_chords lives in E1) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 507 OUT1 195 |
| ✓ | NoToolForCricketMatchParticipantsBFCL/irrelevance_180: who is playing in cricket matches today asked; only a cricket-schedule tool (dates/matches, not participants) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 506 OUT883 |
| ✓ | NoToolForLaLigaPlayersBFCL/irrelevance_181: who played in a whole league asked; only a two-named-team last-match-result tool exists.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 502 OUT927 |
| ✓ | NoToolForMoonlightSonataComposerBFCL/irrelevance_175: composer of Moonlight Sonata asked; only a blues-scale generator is available here (all composer-lookup tools are isolated in E1) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 505 OUT297 |
| ✓ | NoToolForNbaMvpWinnerBFCL/irrelevance_184: who won an MVP award asked (no player named); only a per-player stats tool that requires a player name is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 511 OUT409 |
| ✓ | NoToolForTopAssistPlayerBFCL/irrelevance_178: top ASSIST player asked; only a top-GOAL-scorer tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 506 OUT713 |
| ✓ | NoToolForWimbledonChampionBFCL/irrelevance_183: winner of a tournament asked (no person named); only a celebrity-info tool that requires a name is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 512 OUT824 |
Tool Calling - Irrelevance: Time, Weather & Misc
2 passed1 failed1 skipped |
66.7% |
IN10 342 OUT2 264 | |
| × | NoToolForCityDistanceKmBFCL/irrelevance_239: DISTANCE (km) between two cities asked; only a get_date tool that returns the TIME difference between two locations is available (its 'unit' arg mentions distance, but it computes time) -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN5 188 OUT1 182 |
| – | NoToolForCurrentTimeSydneyBFCL/irrelevance_228: current time in Sydney asked; a local-time-by-city tool is available. REVIEW FLAG (fairness): get_local_time(location='Sydney') directly answers this, so a competent human would legitimately call it. BFCL nonetheless files it under irrelevance; kept as such to preserve the source label, but this item is effectively RELEVANT and should be reviewed. Marked Hard because the "no call" expectation is contradicted by the tool itself.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_local_time' directly answers 'current time in Sydney' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForTomatoCaloriesBFCL/irrelevance_222: CALORIES in a tomato asked; only a grocery-item-details lookup available. REVIEW FLAG (fairness): item_details at 'detailed' level could arguably surface nutrition info, and it requires a store_location the user never gave; BFCL labels it irrelevant (it is a grocery-store item lookup, not a nutrition-facts service). Kept as such; marked Medium as borderline.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 578 OUT789 |
| ✓ | NoToolForWriteABookBFCL/irrelevance_217: "can you write a book" asked; only a recipe-CREATION tool available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN2 576 OUT293 |
Tool Calling - Irrelevance: Weather, Environment & Services
10 passed |
100.0% |
IN36 617 OUT7 021 | |
| ✓ | NoToolForBestCo2ReductionBFCL/irrelevance_108: the best way to reduce CO2 asked; only a numeric emission-reduction estimator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 660 OUT974 |
| ✓ | NoToolForBestMonthToVisitBFCL/irrelevance_101: best month to visit Hawaii asked; only an average-monthly-temperature tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 658 OUT622 |
| ✓ | NoToolForCactusNutrientsBFCL/irrelevance_109: a plant's nutrient needs asked; only a weekly watering-needs calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 665 OUT658 |
| ✓ | NoToolForCurrentTimeWeatherBFCL/irrelevance_103: the current time asked; only a current-weather forecast tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 659 OUT340 |
| ✓ | NoToolForGardenSpeciesCompositionBFCL/irrelevance_107: garden species composition asked; only a soil sand/silt/clay composition analyzer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 662 OUT545 |
| ✓ | NoToolForGmoYogaMatBFCL/irrelevance_111: a GMO yoga mat asked; only an eco-friendly product locator is available (GMO is not eco-friendly) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 661 OUT328 |
| ✓ | NoToolForImpactfulCasesBFCL/irrelevance_94: a firm's impactful cases asked; only a single-case detail lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 669 OUT390 |
| ✓ | NoToolForPaintingEiffelCaseResultBFCL/irrelevance_96: how long to paint the Eiffel Tower asked; only a law-case result lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 661 OUT695 |
| ✓ | NoToolForPollinationFlowersBFCL/irrelevance_105: top pollination flower species asked; only a route elevation-profile plotter is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 663 OUT1 758 |
| ✓ | NoToolForTomatoFertilizerBFCL/irrelevance_106: best fertilizer for tomatoes asked; only a soil-type analyzer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 659 OUT711 |
Tool Calling - Parallel Calls: Art, Music & Local Services
9 passed1 failed1 skipped |
90.0% |
IN75 705 OUT15 059 | |
| – | CalculateDrivingDistancesShortestRouteBFCL/parallel_38: two calc_distance calls — New York→Washington DC and Los Angeles→San Francisco. Tightened per methodology: the prompt explicitly enables "shortest route", so shortest_route == true is required (schema default is false).0 AI queries
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Skipped: Ambiguous scope (item flaw, not model behavior): the trailing 'with optional parameter shortest route enabled' can reasonably apply to only the second leg (its nearest clause) or to both; the BFCL ground truth requires it on both, but a competent human would not answer deterministically. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | CheckInstrumentAvailabilityInTwoCitiesBFCL/parallel_42: two check_instrument_availability calls — Yamaha P125 in Berlin and Madrid.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 484 OUT1 333 |
| ✓ | FindMusicStoresInTwoCitiesBFCL/parallel_41: two find_music_instrument_store calls — San Francisco and Chicago; instruments {Yamaha acoustic guitar, Kawai piano} in any order and either casing.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 511 OUT966 |
| ✓ | FindRockAndJazzConcertsInTwoCitiesBFCL/parallel_43: four concert_finder calls — {San Francisco, New York} x {rock, jazz} over the next month. time_period may be omitted (defaults to 30 days ≈ "next month"). Isolated from concert_find_nearby (parallel_44) into a separate script: both locate concerts by location+genre and would otherwise create an ambiguous tool choice.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 808 OUT1 280 |
| ✓ | GetExhibitionDetailsForTwoMuseumsBFCL/parallel_40: two museum_exhibition_detail calls — 'Wonder of Nature' at the Louvre and 'Age of Reptiles' at the British Museum. visitor_type {child, adult} (unordered). Tightened per methodology: the prompt explicitly asks for cost "for children and adult", so both visitor types are required (schema default is ['adult'] only).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 548 OUT1 985 |
| ✓ | GetMostPlayedPopAndRockSongsBFCL/parallel_45: two musicCharts_getMostPlayed calls — most played Pop and Rock song in Australia. Optional duration omitted (ground truth accepts the schema default of 0).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 470 OUT1 706 |
| ✓ | GetMuseumInfoForTwoCitiesBFCL/parallel_39: two museum_info_get_info calls — Washington D.C. and Paris; details {Opening hours, Adult tickets, Child tickets} in any order.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 499 OUT3 385 |
| ✓ | GetRgbAndHexForThreeColorsBFCL/parallel_37: three color_converter_get_color_info calls — navy, purple, maroon; each with {RGB, HEX} (unordered).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 628 OUT1 249 |
| × | GetTeamRankingsInTwoLeaguesBFCL/parallel_47: two get_team_ranking calls — Barcelona in UEFA Champions League and Manchester United in La Liga.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN7 471 OUT650 |
| ✓ | MixPaintColorsForTwoRequestsBFCL/parallel_36: two paint_color_mixture calls — watercolor magenta and acrylic navy.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 520 OUT1 374 |
| ✓ | WinningPercentageForTwoTeamsTwoSeasonsBFCL/parallel_46: four calculate_winning_percentage calls — {Lakers, Bulls} x {2018, 2020}.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 766 OUT1 131 |
Tool Calling - Parallel Calls: Data, Geo & Life Sciences (set 1)
12 passed1 failed |
92.3% |
IN111 702 OUT24 747 | |
| ✓ | AverageFourNumberSetsBFCL/parallel_132: four calculate_average calls — one per list of numbers (element order preserved).1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 764 OUT1 538 |
| × | CalculateBmiFourPeopleMetricBFCL/parallel_134: four calculate_bmi calls (weight kg, height cm). Tightened: the prompt states "All measurements are in the metric system", so unit == "metric" is required (also the default).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 754 OUT1 598 |
| ✓ | CalculateCityDistancesInKilometersBFCL/parallel_135: three geo_distance_calculate calls — NY->LA->Miami->NY, in kilometers. Tightened: the tool defaults to miles but the prompt asks "in kilometers", so units == "kilometers" is REQUIRED (omission would return miles, contradicting the request — the ground truth's "omit" option is unsafe here).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 523 OUT2 265 |
| ✓ | CalculateGpsDistancesAlongRouteBFCL/parallel_133: three calculate_distance calls over GPS coordinate pairs (Paris->Rome->Athens-> Giza), each in kilometers. Uses the coordinate-based tool (coord1/coord2), not the city-name distance tools that are also visible.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 787 OUT4 120 |
| ✓ | CompareAverageTemperatureNyAndLaBFCL/parallel_128: two average_temperature calls — New York 7 days in Fahrenheit, Los Angeles 7 days in Celsius. Tightened: the prompt explicitly requests Fahrenheit for NY, so temp_unit == "Fahrenheit" is required (also the default); Celsius is non-default and required for LA.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 358 OUT2 187 |
| ✓ | CreateTwoHistogramsWithFiveBinsBFCL/parallel_129: two create_histogram calls — one per data set, 5 bins each (data order preserved).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 822 OUT1 812 |
| ✓ | EstimateElephantPopulationThreeScenariosBFCL/parallel_125: three elephant_population_estimate calls — current 500 at 2%, 1.5%, 2.5% over 10y.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 699 OUT1 845 |
| ✓ | FindItalianRestaurantsInNyAndLaBFCL/parallel_130: two find_restaurants calls — New York and Los Angeles, Italian, 4 results each, dietary_requirements [vegan, gluten-free]. Distinct from restaurant_find_nearby (this tool carries food_type + number, which the prompt supplies, so it is the correct choice here).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 422 OUT1 433 |
| ✓ | FindRestaurantsByDietInThreeCitiesBFCL/parallel_127: three restaurant_find_nearby calls — New York, Los Angeles, Chicago, each with dietary_preference [Vegan, Gluten-free, Dairy-free]. List order follows the prompt (matches ground truth).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 545 OUT1 596 |
| ✓ | PredictEvolutionDarwinAndLamarckBFCL/parallel_126: two prediction_evolution calls — African Elephant over 5000y, Darwin then Lamarck. Tightened per methodology: the prompt explicitly says "using the Darwin model", so model == "Darwin" is required for the first call (ground truth also accepts omission via the 'Darwin' default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 381 OUT1 088 |
| ✓ | ProjectPopulationGrowthJapanAndIndiaBFCL/parallel_124: four population_projections calls — Japan/10y and India/20y, each at the current (default) growth rate and at an explicit rate. "current growth rate" maps to the default (growth_rate omitted, accepts null or 0.01); the explicit 1.5%/2.1% rates are required.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 706 OUT2 113 |
| ✓ | RouteThreeLegsAvoidingTollsBFCL/parallel_131: three map_routing_fastest_route calls — SF->Palo Alto->San Jose->home, all avoiding tolls. Tightened: the prompt explicitly says "avoiding toll roads", so avoid_tolls == true is required for every leg (non-default value stated in the prompt).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 551 OUT1 343 |
| ✓ | ShortestCityDistanceBusWithAndWithoutTransferBFCL/parallel_136: two city_distance_find_shortest calls — NY->LA by bus, first without transfer then allowing transfer. Tightened: the prompt explicitly says "using a bus", so transportation == "bus" is required for both (also the default). allow_transfer: false/omitted for the first, true for the second.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 390 OUT1 809 |
Tool Calling - Parallel Calls: Data, Geo & Life Sciences (set 2)
14 passed |
100.0% |
IN125 790 OUT19 770 | |
| ✓ | CalculateAreaOfThreeTrianglesBFCL/parallel_143: three calc_area_triangle calls — (10,5), (15,7), (20,10).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 958 OUT1 053 |
| ✓ | CalculateBinomialProbabilityBasketballBFCL/parallel_161: three calculate_binomial_probability calls — basketball shooter at 60%: (10,7), (15,10), (20,15). probability_of_success == 0.6 is required (prompt states 60%). ISOLATED here (away from prob_dist.binomial in ToolCalling-Parallel-D3): both tools compute binomial probability, and this prompt does not name a function, so having both visible would make the choice genuinely ambiguous. Kept apart so exactly one correct tool is available.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 037 OUT2 822 |
| ✓ | CalculateBmiInMetersForThreePeopleBFCL/parallel_138: three calculate_BMI calls (weight kg, height meters) — John, Sarah, Mike.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 019 OUT1 501 |
| ✓ | CalculateCelestialDistancesInMilesBFCL/parallel_149: two calculate_distance calls between celestial bodies (Mars-Venus, Mars-Jupiter), in miles. Uses the body-name distance tool (body1/body2). Tightened: prompt asks "in miles", so unit == "miles" is required (tool default is kilometers, so omission would be wrong).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 764 OUT1 556 |
| ✓ | CalculateClockAnglesAtThreeTimesBFCL/parallel_145: three calculate_clock_angle calls — 3:15, 8:20, 11:50. Tightened: the prompt asks for results "rounded to 2 decimal places", so round_to == 2 is required (also the default).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 939 OUT2 251 |
| ✓ | CalculateFactorialOfFourNumbersBFCL/parallel_144: four math_factorial calls — 5, 3, 4, 2.1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 968 OUT696 |
| ✓ | CalculateLightTravelTimeThreeLegsBFCL/parallel_147: three light_travel_time calls — 4.22, 6.1, 5.88 light years. Tightened: the prompt explicitly states the speed of light (299792458 m/s), so speed_of_light == 299792458 is required (also default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 025 OUT1 628 |
| ✓ | CalculateSpeedForCarBicycleTrainBFCL/parallel_148: three calculate_speed calls — car 500m/25s in km/h, bicycle 1000m/200s in m/s, train 10000m/600s in km/h. Each output unit is explicitly named in the prompt, so to_unit is required.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 040 OUT1 806 |
| ✓ | FetchAllDataFieldsForTwoEmployeesBFCL/parallel_139: two employee_fetch_data calls — Tech Solutions, employees 12345 and 67890, each fetching all four data fields (order follows the prompt / ground truth).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 872 OUT1 007 |
| ✓ | FindDiCaprioDramaAndComedyMoviesBFCL/parallel_140: two imdb_find_movies_by_actor calls — Leonardo DiCaprio, Drama in 2010 and Comedy in 2012. Tightened: the prompt explicitly names each category, so category is required for both.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 760 OUT849 |
| ✓ | GetImaxAndTwoDTheaterReleasesBFCL/parallel_141: two get_theater_movie_releases calls — New York IMAX over 7 days, Los Angeles 2D over 14 days. Tightened: the prompt explicitly names each format, so format is required for both.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 789 OUT952 |
| ✓ | PlotTwoSineWavesBFCL/parallel_146: two plot_sine_wave calls with all five parameters supplied by the prompt.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 895 OUT488 |
| ✓ | SortFourListsAscendingAndDescendingBFCL/parallel_137: eight array_sort calls — four lists, each sorted ascending AND descending (list element order preserved as given).1 AI query
8 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 844 OUT2 188 |
| ✓ | UpdateTwoCustomersNameAndEmailBFCL/parallel_142: two update_user_info calls — users 12345 and 67890, both set name "John" and email "example@.com". update_info is a nested object. Tightened: the prompt names the 'CustomerInfo' database, so database == "CustomerInfo" is required (also the default).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 880 OUT973 |
Tool Calling - Parallel Calls: Data, Geo & Life Sciences (set 3)
10 passed1 failed |
90.9% |
IN87 936 OUT16 021 | |
| ✓ | AreaUnderCurveForTwoPolynomialsBFCL/parallel_150: two mathematics_calculate_area_under_curve calls — polynomial [3,-2,1] over [-1,2] and [1,0,-1] over [0,3] (coefficient/limit order preserved).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 831 OUT1 829 |
| ✓ | BinomialProbabilitiesThreeGamesBFCL/parallel_160: three prob_dist_binomial calls — (20,5), (50,15), (100,30), each with p == 0.3 (prompt states a 0.3 win probability). The prompt explicitly names 'prob_dist.binomial'; the colliding calculate_binomial_probability tool is kept in a separate script (ToolCalling-Parallel-D2).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 001 OUT1 896 |
| ✓ | CalculateBmiSystemFourPeopleBFCL/parallel_154: four calculate_bmi calls (weight kg, height cm). Tightened: the prompt says "use the metric system for all calculations", so system == "metric" is required (also the default).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 128 OUT1 163 |
| ✓ | DiceRollProbabilitiesThreeScenariosBFCL/parallel_159: three probability_dice_roll calls — roll 4 three times and roll 2 twice on a six-sided die (die_sides default 6, accepts omission), then roll 7 twice on an eight-sided die (die_sides == 8 required, non-default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 958 OUT1 440 |
| ✓ | GenerateNormalVariatesTwoConfigsBFCL/parallel_158: four random_normalvariate calls — two at (mu 5, sigma 2) and two at (mu 10, sigma 3). Duplicate calls: each configuration must appear exactly twice (multiplicity 2).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 064 OUT1 707 |
| ✓ | PredictPricesForFourHousesBFCL/parallel_157: four predict_house_price calls — New York (3/2/1500), Los Angeles (4/3/2000), Chicago (2/1/1200), Miami (3/2/1800).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 252 OUT2 169 |
| ✓ | RaiseTwoNumbersToPowersBFCL/parallel_152: two math_power calls — 3^5 and 2^3, no modulus (mod omitted / null).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 684 OUT797 |
| ✓ | RunLinearRegressionWithAndWithoutStandardizationBFCL/parallel_155: two run_linear_regression calls — same predictors [Age, Income, Education] and target "Spending Score", first without standardization then with. standardize is explicitly requested (false then true), so it is required for both.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 876 OUT1 010 |
| ✓ | TrainRandomForestFourConfigsBFCL/parallel_156: four random_forest_train calls on "data_random_forest" — (100,10), (200,20), (300,30), (400,40). The prompt explicitly names this function, disambiguating it from train_random_forest_classifier visible in this script.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 310 OUT1 685 |
| ✓ | TrainRfClassifierTwoConfigsBFCL/parallel_153: two train_random_forest_classifier calls — dataset1 (depth 10, 100 trees) and dataset2 (depth 20, 200 trees). The prompt explicitly names this function, disambiguating it from the other random-forest training tool (random_forest_train) visible in this script.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 874 OUT949 |
| × | TriangleAreasInDifferentUnitsBFCL/parallel_151: three geometry_area_triangle calls in different units — (15,20) square meters, (25,30) square feet, (35,40) square inches. First unit is the default (accepts omission / square meters); feet and inches are non-default and required.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN7 958 OUT1 376 |
Tool Calling - Parallel Calls: Finance, Stats & Legal
13 passed |
100.0% |
IN110 699 OUT19 715 | |
| ✓ | BuildTwoDecisionTreeClassifiersBFCL/parallel_12: two model_DecisionTreeClassifier calls — gini/depth 5 and entropy/depth 10, both random_state 1.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 440 OUT480 |
| ✓ | CalculateFutureValueForThreeTermsBFCL/parallel_18: three financials_calculate_future_value calls — years 1, 5, 10 (same 1000 @ 0.05).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 602 OUT1 654 |
| ✓ | CalculatePresentValueForThreeTermsBFCL/parallel_14: three calculate_present_value calls — years 10, 20, 30 (same payment 1000, rate 0.05).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 599 OUT2 249 |
| ✓ | CapitalGainsTaxForTwoStatesBFCL/parallel_15: two calculate_capital_gains_tax calls — CA (15000/25000) and FL (20000/50000).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 492 OUT1 778 |
| ✓ | ComputePythagorasForTwoTrianglesBFCL/parallel_10: two math_pythagoras calls — sides (3,4) and (5,12).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 383 OUT1 104 |
| ✓ | ConfidenceIntervalForTwoSampleSizesBFCL/parallel_13: two confidence_interval_calculate calls — sample sizes 50 and 150. Tightened per methodology: the prompt explicitly requests "95% confidence interval", so confidence_level == 0.95 is required (schema default is 0.9, but the prompt states the value).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 495 OUT2 126 |
| ✓ | CourtCaseStatusAndTrialDatesBFCL/parallel_22: four court_info_get_case_status calls — cases XY1234 (LA) and GH5678 (Orange County), each queried for status (details may be omitted via the 'status' default) and for trial_date.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 738 OUT2 594 |
| ✓ | GetStockPriceAndVolumeForThreeCompaniesBFCL/parallel_17: three get_stock_data calls — AAPL, GOOG/GOOGL, MSFT; each with data_points {price, volume} (unordered).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 534 OUT1 612 |
| ✓ | LoanEligibilityForTwoInstitutionsBFCL/parallel_20: two loan_eligibility_check calls — HSBC (500k/100k) and Wells Fargo (700k/120k).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 507 OUT629 |
| ✓ | MortgagePaymentForThreeLoanTermsBFCL/parallel_19: three calculate_mortgage_payment calls — loan terms 15, 20, 30 (400000 @ 0.04).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 627 OUT1 579 |
| ✓ | PredictHousePriceInTwoCitiesBFCL/parallel_11: two ml_predict_house_price calls — New York 3000 sq ft, Los Angeles 4000 sq ft.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 414 OUT1 038 |
| ✓ | ReturnOnInvestmentGainAndLossBFCL/parallel_16: two calculate_return_on_investment calls — gain 500 on 2000, loss -1000 on 5000.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 445 OUT1 440 |
| ✓ | SearchMoneyLaunderingConvictionsBFCL/parallel_21: two law_crimes_search calls — money laundering in San Francisco 2019 and Texas 2018.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 423 OUT1 432 |
Tool Calling - Parallel Calls: Legal, History & Environment
11 passed2 failed1 skipped |
84.6% |
IN122 454 OUT20 773 | |
| ✓ | BookTwoConnectingFlightsBFCL/parallel_30: two book_flight calls — San Francisco→Tokyo (2022-05-03) and Tokyo→Sydney (2022-05-18).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 057 OUT622 |
| ✓ | CalculateAlimonyForTwoDurationsBFCL/parallel_23: two alimony_calculator_ca_calculate calls — durations 10 and 20 years (10000/3000 income).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 120 OUT1 372 |
| × | CalculateWasteForFamilyAndBachelorBFCL/parallel_29: two waste_calculation_calculate calls — LA family (2 adults, 2 children) and New York bachelor (1 single). The population argument is a nested object.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match1 == 1
Values:
match1 = 0 |
Failed | IN9 070 OUT1 268 |
| ✓ | FetchTwoHistoricalFactsBFCL/parallel_31: two history_fact_fetch calls — Treaty of Paris and Magna Carta. Optional depth/year omitted (ground truth accepts the schema defaults).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 976 OUT1 187 |
| ✓ | FindClassicalConcertsWithParkingBFCL/parallel_44: two concert_find_nearby calls — classical concerts near Berlin and Paris next Friday. Tightened per methodology: the prompt explicitly asks for concerts "with available parking", so amenities == ['Parking'] is required (this also matches the schema default, but the prompt states it). Isolated from concert_finder (parallel_43) into a separate script: both locate concerts by location+genre and would otherwise create an ambiguous tool choice.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 034 OUT1 991 |
| – | FindCourtCasesByNumberAndTypeBFCL/parallel_26: four court_case_find calls — the cross product of case numbers {67813, 71249} and types {Civil, Criminal}; case_number is a single-element array; Civil may be omitted via its default. FLAGGED as borderline (see report): the prompt "case numbers '67813','71249' for type 'Civil' and 'Criminal'" is ambiguous about the pairing; BFCL ground truth expects all four combinations, kept faithful.0 AI queries
0 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL Skipped: Ambiguous call decomposition (item flaw, not model behavior): 'case numbers 67813, 71249 for type Civil and Criminal' has no single deterministic reading - two calls (both numbers per type, case_number as a list) is as valid as the BFCL 2x2 = four single-number calls; a competent human would not answer deterministically. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | FindNatureReservesNearTwoCitiesBFCL/parallel_27: two nature_reserve_find_nearby calls — Berkeley (10km, picnic tables + restrooms) and Tokyo (5km, playgrounds + biking trails). Amenity list order is unordered.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 058 OUT1 002 |
| ✓ | GetCaseDetailsWithHistoryAndLitigantsBFCL/parallel_24: two law_case_get_details calls — cases 28473 and 64725. Tightened per methodology: the prompt explicitly asks for "their history and details of litigants", so include_history == true and include_litigants == true are required (schema defaults are false).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 040 OUT1 418 |
| ✓ | GetEventsForTwoPresidenciesBFCL/parallel_32: two us_history_events_by_presidency calls — Abraham Lincoln and George Washington. Optional start_year/end_year omitted (ground truth accepts the schema defaults).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 971 OUT1 517 |
| ✓ | GetPresidentsAndVicePresidentsByYearBFCL/parallel_33: four get_president_and_vp calls — president for 1980 and 2016, vice president for 1975 and 2011.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 326 OUT1 315 |
| ✓ | GetReligionInfoForTwoEmpiresBFCL/parallel_35: two ancient_empires_get_religion_info calls — Persian Empire and Mauryan Empire. Tightened per methodology: the prompt explicitly asks for "religious history and influences", so include_influences == true is required (schema default is false).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 011 OUT1 944 |
| ✓ | GetTemperatureForTwoCitiesBFCL/parallel_28: two get_current_and_future_temperature calls — Seattle and Los Angeles, next 3 hours.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 983 OUT1 306 |
| × | LookupLawsuitsForTwoYearsBFCL/parallel_25: two lawsuit_lookup calls — Dara Inc for years 2019 and 2018.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 3
expectedCount = 2 |
Failed | IN13 754 OUT2 073 |
| ✓ | TrackChristianityInTwoRegionsBFCL/parallel_34: two religion_history_track calls — Christianity in Egypt and Turkey, 100–1500 A.D.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 054 OUT3 758 |
Tool Calling - Parallel Calls: Markets & Environment
11 passed2 failed |
84.6% |
IN114 544 OUT23 190 | |
| ✓ | CalculateCagrForTwoInvestmentsBFCL/parallel_176: two calculate_cagr calls — (5000,7000,5) and (8000,12000,3).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 665 OUT1 090 |
| ✓ | CalculateEmissionSavingsForSolarAndWindBFCL/parallel_186: two calculate_emission_savings calls — solar(12 months) and wind(8 months), Midwest region. Prompt states the Midwest region explicitly -> region required (accept "Midwest"/"Midwest region").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 558 OUT2 886 |
| ✓ | CompareFutureValueWithAndWithoutContributionsBFCL/parallel_182: two finance_calculate_future_value calls — (5000,0.07,10, no contribution) and (3000,0.06,10, contribution 200). First scenario states "no additional contributions" -> null or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 725 OUT1 669 |
| × | ComputeEuclideanNormForThreePointsBFCL/parallel_183: three math_hypot calls — (5,7), (10,15), (20,25). z omitted or 0 (2D points).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 2
expectedCount = 3 |
Failed | IN8 620 OUT1 409 |
| ✓ | EstimatePopulationsForTwoSpeciesAcrossTwoYearsBFCL/parallel_185: four estimate_population calls — Bengal Tigers/India and African Elephants/Kenya for 2021 and for 2023. NOTE: upstream ground truth's third entry lists year "" (omitted -> default 2020), which is an upstream error: the prompt explicitly asks for 2023 for "both these species". We tighten both 2023 estimates to year == 2023 (fair deterministic answer).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 904 OUT1 797 |
| ✓ | FindRootsForTwoQuadraticEquationsBFCL/parallel_184: two algebra_quadratic_roots calls — (3,7,2) and (5,-4,1).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 678 OUT1 923 |
| ✓ | GetClosingPricesForTwoCompaniesOnTwoDatesBFCL/parallel_178: four get_stock_price (by-date variant) calls — the 2x2 cross product of {Microsoft, Apple} x {2022-01-01, 2022-02-01} on NASDAQ. Upstream ground truth's first entry loosely lists company_name as [Microsoft, Apple]; the deterministic human answer is the four distinct (company, date) combinations, which we require. exchange NASDAQ (default) may be omitted.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 862 OUT1 982 |
| ✓ | GetDetailedAirQualityForThreeDaysBFCL/parallel_187: three get_air_quality calls — New York City, detail true, for 2023-05-05, 05-04, 05-03. Prompt asks for PM2.5/PM10/ozone/sources -> detail == true; "today/yesterday/day before" with today = May 5, 2023.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 764 OUT2 336 |
| ✓ | GetFourPriceTypesForTwoCompaniesBFCL/parallel_180: eight stock_price calls — {Microsoft, Apple} x {Open, Close, High, Low}, days 30. Prompt explicitly lists all four data types, so data_type must be set explicitly (required) to distinguish the eight calls; omission (default 'Close') cannot represent the four requested types.1 AI query
8 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL |
Passed | IN9 438 OUT2 174 |
| ✓ | GetPricePerOunceForFourMetalsBFCL/parallel_177: four get_metal_price calls — gold, silver, platinum, palladium, measure 'ounce'. Prompt asks "price per ounce" -> measure == "ounce" required.1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 785 OUT1 883 |
| × | GetStockPricesForFourCompaniesAndDurationsBFCL/parallel_181: four get_stock_prices calls — Apple(1 week), Microsoft(2 weeks), Amazon(3 weeks), Tesla(1 month). Each 'companies' argument is a single-element list.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 878 OUT946 |
| ✓ | GetTrafficInfoForThreeLegsWithModesBFCL/parallel_188: three get_traffic_info calls — NY->LA (driving), LA->SF (bicycling), SF->NY (transit). First leg explicitly says "driving" -> mode == "driving" required (tightened; also the default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 759 OUT1 038 |
| ✓ | ProjectFutureValueForThreeStocksBFCL/parallel_175: three portfolio_future_value calls — AAPL(5000,0.07,5), MSFT(8000,0.06,7), AMZN(10000,0.08,10).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 908 OUT2 057 |
Tool Calling - Parallel Calls: Mathematics & Physics (set 1)
12 passed1 failed |
92.3% |
IN103 975 OUT18 546 | |
| × | AreaOfTwoRightTrianglesInMetersBFCL/parallel_68: two calculate_area calls — right triangles base/height 12x15 and 18x24 in meters. unit required (tool default is 'cm'; the prompt states meters/square meters).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN7 970 OUT1 004 |
| ✓ | CheckPlayerStatusAcrossThreeSeasonsBFCL/parallel_49: three player_status_check calls — player 3142 in RocketLeague for seasons 2017/2018/2019. Seasons are explicitly given, so each is required (default 2023 not applicable).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 071 OUT1 397 |
| ✓ | CircumferenceOfFourCirclesInCmBFCL/parallel_65: four calculate_circumference calls — radii 5/10/15/20. Tightened per methodology: the prompt gives every radius in cm (e.g. "5cm") and the tool default unit is 'm', so a correct call must pass unit == 'cm'/'centimeter' (ground truth requires it on the first call and merely allows omission on the rest; we require it consistently since it is stated).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 213 OUT1 950 |
| ✓ | ConvertEuroAndPoundsToDollarsBFCL/parallel_59: two get_conversion_cost calls — 15000 Euro->dollars and 200 pounds->dollars.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 900 OUT1 459 |
| ✓ | FinalVelocityForFourDropsBFCL/parallel_81: four calculate_final_velocity calls — heights 10/20/15/25, initial_velocity 0 (stated), gravity 9.81 (default, may be omitted).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 354 OUT1 666 |
| ✓ | FindItalianAndJapaneseRestaurantsBFCL/parallel_52: two restaurant_finder calls — Italian in New York (preferences ['Vegetarian']) and Japanese in Los Angeles (delivery; ['Delivery'] or omitted since default is ['Delivery']).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 888 OUT1 150 |
| ✓ | FindRootsOfTwoQuadraticsAlgebraBFCL/parallel_62: two algebra_quadratic_roots calls — 3x^2+4x+2 and 5x^2-7x+3.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 050 OUT1 368 |
| ✓ | FindSoupAndVeganSaladRecipesBFCL/parallel_51: two recipe_search_find calls — Chicken Noodle Soup (diet omitted/default 'Keto') and a Vegan Salad (diet == 'Vegan', dish name accepts several phrasings).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 831 OUT1 168 |
| ✓ | GcdOfJohnAndMaryNumbersBFCL/parallel_80: two math_gcd calls — gcd(36,48) and gcd(60,96). Same tool as parallel_76.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 946 OUT2 023 |
| ✓ | GcdOfTwoPairsBFCL/parallel_76: two math_gcd calls — gcd(45,60) and gcd(81,27).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 878 OUT1 109 |
| ✓ | GetPokemonMovesAndCheckSolarBeamBFCL/parallel_48: two PokemonGO_get_moves calls — Pikachu's move list (move omitted/default 'Run') and a check whether Bulbasaur can learn 'Solar Beam'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 868 OUT1 624 |
| ✓ | PrimeFactorsFormattedAndArrayBFCL/parallel_74: four get_prime_factors calls — numbers 4567 and 7890, each formatted true and false (user wants "a formatted string as well as an array").1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 160 OUT1 979 |
| ✓ | SaveGameProgressEasyAndHardBFCL/parallel_50: two game_save_progress calls — stage 7 easy and stage 3 hard; level omitted/default 'user'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 846 OUT649 |
Tool Calling - Parallel Calls: Mathematics & Physics (set 2)
13 passed |
100.0% |
IN111 965 OUT21 400 | |
| ✓ | AreaOfThreeCirclesInMetersBFCL/parallel_66: three geometry_area_circle calls — radii 5/10/15 in meters (default is meters, so units may be omitted).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 335 OUT1 728 |
| ✓ | BookTwoHotelRoomsBFCL/parallel_57: two hotel_booking_book calls — Sheraton (NY, 2 adults/1 child) and Marriott (LA, 1 adult/2 children).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 383 OUT764 |
| ✓ | CheckAvailabilityForTwoHotelsBFCL/parallel_56: two hotel_booking_check_availability calls — Queens Hotel (Berlin) and Royal Hotel (Paris), each with check-in/out dates (several date formats accepted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 315 OUT796 |
| ✓ | CircumferenceOfFourCirclesInMetersBFCL/parallel_70: four geometry_circumference calls — radii 5/10/15/20. Tightened per methodology: the prompt asks for output in meters and gives radii in m, but the tool default unit is 'cm', so a correct call must pass units == 'm'/'meters' (ground truth requires it on the first call and only allows omission on the rest; we require it consistently since it is explicitly stated).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 511 OUT2 027 |
| ✓ | DerivativeOfPolynomialAndItsDerivativeBFCL/parallel_71: two calculate_derivative calls — 3x^3-2x^2+5x-7 at x=4, then its derivative 9x^2-4x+5 at x=2.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 278 OUT2 669 |
| ✓ | FinalVelocityForThreeDragRaceRoundsBFCL/parallel_83: three final_velocity calls — (v0=0,a=5,t=10), (v0=10,a=7,t=8), (v0=20,a=4,t=12).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 512 OUT1 316 |
| ✓ | FindGroceryStoresByCriteriaBFCL/parallel_55: two grocery_store_find_by_criteria calls — NYC ['24 hours'] and San Diego ['Home Delivery'].1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 148 OUT1 372 |
| ✓ | GetTwoCookingRecipesBFCL/parallel_53: two get_cooking_recipe calls — Lasagne Bolognese for 4 and Caesar Salad for 2.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 166 OUT1 371 |
| ✓ | GetTwoExchangeRatesBFCL/parallel_58: two get_exchange_rate calls — USD->AUD and USD->CAD.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 130 OUT1 155 |
| ✓ | HighestCommonFactorOfFourPairsBFCL/parallel_77: four math_hcf calls — (45,60), (90,120), (36,48), (72,96). FLAGGED (see final report): the prompt is phrased as chaining ("use that result to find the HCF with...") but the ground truth treats it as four INDEPENDENT parallel calls on the literal pairs; a human might interpret it as sequential composition. Kept BFCL-faithful (four independent literal-pair calls) since parallel semantics preclude feeding one call's result into another.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 549 OUT3 818 |
| ✓ | OrderSameItemsFromTwoWholeFoodsLocationsBFCL/parallel_54: two whole_foods_order calls — same two items (order-independent) at downtown and uptown, size large.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 196 OUT1 041 |
| ✓ | PrimeFactorizeAsDictionaryBFCL/parallel_79: two prime_factorize calls — 4567 and 7890, return_type 'dictionary' (explicitly requested).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 172 OUT1 864 |
| ✓ | SolveTwoQuadraticEquationsBFCL/parallel_63: two solve_quadratic_equation calls — (5,6,1) and (3,2,1). Prompt names this function explicitly.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 270 OUT1 479 |
Tool Calling - Parallel Calls: Mathematics & Physics (set 3)
12 passed |
100.0% |
IN98 262 OUT18 321 | |
| ✓ | AreaOfTwoCirclesInMetersBFCL/parallel_67: two geometry_calculate_area_circle calls — radii 5 and 10 in meters. unit required (tool default is 'cm'; the prompt states meters).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 019 OUT1 264 |
| ✓ | AreaOfTwoTrianglesBFCL/parallel_69: two calculate_triangle_area calls — base/height 10x5 and 8x6.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 079 OUT1 103 |
| ✓ | DerivativeOfTwoFunctionsAtValuesBFCL/parallel_73: two calculus_derivative calls — 3x^2+2x-1 at 5 (variable x, default) and 4y^3-3y^2+2y-1 at 3 (variable y).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 124 OUT1 836 |
| ✓ | DisplacementForThreeCarsBFCL/parallel_84: three calculate_displacement calls — (15,7,accel 3.5), (20,10,accel 2.0), (25,8,no acceleration). The third car travels "without any acceleration" so acceleration == 0 (also the default; omission accepted).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 349 OUT1 697 |
| ✓ | EuclideanNormForThreePointsBFCL/parallel_61: three math_hypot calls — (3,4) and (6,8) with z omitted/0, and (9,12,15).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 254 OUT1 741 |
| ✓ | FactorialOfThreeNumbersBFCL/parallel_60: three math_factorial calls — 5, 7, 9.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 081 OUT989 |
| ✓ | FinalSpeedForTwoFreeFallObjectsBFCL/parallel_85: two calculate_final_speed calls — initial_speed 0 for 10 s and initial_speed 5 for 7 s; gravity -9.81 (default, may be omitted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 149 OUT2 887 |
| ✓ | GcdViaNumberTheoryBFCL/parallel_78: two number_theory_gcd calls — (45,60) and (81,63). Prompt names this function explicitly.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 022 OUT1 213 |
| ✓ | IntegrateTwoFunctionsWithTwoMethodsBFCL/parallel_72: four integrate calls — x**3 on [2,5] and 2x**2+3x-1 on [-1,3], each with trapezoid (default, may be omitted) and simpson (explicit).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 468 OUT1 934 |
| ✓ | PrimeFactorsOfThreeNumbersBFCL/parallel_75: three number_analysis_prime_factors calls — 45, 100, 150.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 087 OUT878 |
| ✓ | SolveFourQuadraticsWithRootTypeBFCL/parallel_64: four solve_quadratic calls — (2,5,3) all roots, (1,-3,2) real, (4,-7,3) all roots, (1,2,1) real. 'all' is the default so it may be omitted; 'real' is explicitly requested and required.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 518 OUT1 694 |
| ✓ | VelocityForTwoCyclingDaysBFCL/parallel_82: two calculate_velocity calls — 120 km in 5 h and 150 km in 6 h; unit km/h (default, may be omitted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 112 OUT1 085 |
Tool Calling - Parallel Calls: Mixed Domains
7 passed3 failed |
70.0% |
IN73 342 OUT14 847 | |
| ✓ | CalculateBmiForTwoPeopleBFCL/parallel_4: two calculate_bmi calls — 6 ft / 80 kg and 5.6 ft / 60 kg.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 278 OUT712 |
| ✓ | CalculateEmForceForTwoTimeIntervalsBFCL/parallel_1: two calculate_em_force calls — same field/area, d_time 4 then 10.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 299 OUT1 283 |
| ✓ | CalculateFactorialForThreeNumbersBFCL/parallel_7: three math_factorial calls — 5, 10, 15.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 338 OUT779 |
| ✓ | CalculateResistanceForCopperAndAluminumBFCL/parallel_2: two calculate_resistance calls — copper then aluminum, same length/area. Tightened per methodology: the prompt explicitly names "resistivity of copper", so the first call must pass resistivity == "copper" (ground truth also accepts omission via the 'copper' default, but an explicitly-stated value must be supplied). Aluminum is non-default → required.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 286 OUT2 366 |
| ✓ | CalculateSalesTaxForThreeCitiesBFCL/parallel_6: three calculate_sales_tax calls — Chicago/IL $30.45, Sacramento/CA $52.33, Portland/OR $11.23.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 470 OUT1 472 |
| ✓ | FetchPopulationForCitiesStateAndCountryBFCL/parallel_8: four database_us_census_get_population calls — NYC (city), LA (city), Alaska (state), USA (country); year defaults to 2000 (may be omitted). Requires area-type inference.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 554 OUT1 268 |
| × | FindMovieShowingsForTwoMoviesBFCL/parallel_9: two find_movie_showing calls near San Diego — Tenet at 5 pm and No Time To Die at 7:30 pm.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 1
expectedCount = 2 |
Failed | IN7 144 OUT1 809 |
| × | GetSequenceAnd3DForThreeProteinsBFCL/parallel_3: three protein_info_get_sequence_and_3D calls — HbA1c, normal + rat hemoglobin. Tightened per methodology: the prompt explicitly asks for "their 3D models", so model_3d == true is required (ground truth also accepts omission via the default of true).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN7 418 OUT3 492 |
| × | ListShowRatingsOnNetflixAndHuluBFCL/parallel_5: two streaming_services calls — Netflix['Friends'] and Hulu['The Office','Stranger Things'] (Hulu show order unordered), both sorted by rating. Must group shows per service.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN7 293 OUT1 296 |
| ✓ | PlayTwoArtistsForGivenDurationsBFCL/parallel_0: two spotify_play calls — Taylor Swift for 20 min and Maroon 5 for 15 min.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 262 OUT370 |
Tool Calling - Parallel Calls: Physical & Life Sciences (set 1)
11 passed1 failed |
91.7% |
IN120 512 OUT23 617 | |
| ✓ | ElectricFieldForTwoChargesBFCL/parallel_94: two calculate_electric_field calls — charge 5/distance 2 and charge 3/distance 4. "Conducted in a vacuum" = the default permitivity → may be omitted or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 410 OUT848 |
| ✓ | ElectromagneticForceInTwoMediaBFCL/parallel_96: two electromagnetic_force calls — same charges 5,10 and distance 2, differing by medium_permittivity. Vacuum call (8.854e-12) equals the default → may be omitted or 8.854e-12; the second medium (5e-12) is non-default → required.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 566 OUT3 909 |
| × | EstimateTravelTimeForThreeRoutesBFCL/parallel_93: three route_estimate_time calls — (1) NYC->Miami via Philadelphia/Washington D.C./Atlanta, (2) NYC->Miami skipping Atlanta (via Philadelphia/Washington D.C.), (3) Philadelphia-> Miami via Washington D.C. Stop orderings from ground truth accepted. FLAG parallel_93: BORDERLINE. Ground truth allows many stop-list variants, some including the start city itself as a stop; a competent human's stop set is not fully deterministic. Kept BFCL-faithful by accepting the natural stop orderings (start city excluded from stops).1 AI query
12 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 12
expectedCount = 3 |
Failed | IN27 055 OUT4 287 |
| ✓ | FinalSpeedForTwoDroppedObjectsBFCL/parallel_88: two calculate_final_speed calls — v0 0/height 10 and v0 5/height 20. gravity stated as 9.8 (the default) so it may be omitted or 9.8 per ground truth.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 416 OUT1 127 |
| ✓ | FinalVelocityForTwoCarsBFCL/parallel_87: two calculate_final_velocity calls — Car A (v0 0, a 6, t 10), Car B (v0 20, a 4, t 15).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 433 OUT954 |
| ✓ | FinalVelocityFromDistanceForTwoObjectsBFCL/parallel_86: two kinematics_final_velocity_from_distance calls — accel 5/dist 100 and accel 10/dist 200. "Both objects start from rest" → initial_velocity may be omitted or 0 (default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 405 OUT1 321 |
| ✓ | FindVeganRestaurantsInThreeCitiesBFCL/parallel_91: three vegan_restaurant_find_nearby calls — LA, SF, Seattle, each with operating_hours 22 (prompt: "open until at least 22:00", a non-default value → required).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 462 OUT1 750 |
| ✓ | GenerateFourTravelItinerariesBFCL/parallel_90: four travel_itinerary_generator calls — Tokyo/7/200/urban, Paris/10/150/history, Sydney/5/100/nature, Rome/12/180/culture. Tokyo's "urban" equals the default → omit or "urban".1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 787 OUT1 241 |
| ✓ | GetDirectionsForRoundTripBFCL/parallel_89: three get_directions calls — SF->Palo Alto (fastest), Palo Alto->Golden Gate Bridge (scenic), Golden Gate Bridge->SF (fastest). route_type explicitly stated each time.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 512 OUT1 879 |
| ✓ | MagneticFieldForTwoLoopsBFCL/parallel_95: two calculate_magnetic_field calls — current 10/radius 0.5 and current 15/radius 1.0. "Same as in free space" = default permeability → may be omitted or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 516 OUT1 451 |
| ✓ | ResonantFrequencyForTwoCircuitsBFCL/parallel_97: two calculate_resonant_frequency calls — L 0.005/C 1e-07/round 3 and L 0.007/C 2e-07/round 4. Rounding is stated explicitly → required.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 480 OUT2 851 |
| ✓ | ShortestDrivingDistanceRoundTripBFCL/parallel_92: three get_shortest_driving_distance calls — NYC->LA, LA->Miami, Miami->NYC, all in miles (prompt explicitly "in miles" → non-default unit required).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 470 OUT1 999 |
Tool Calling - Parallel Calls: Physical & Life Sciences (set 2)
12 passed1 failed |
92.3% |
IN112 035 OUT26 997 | |
| ✓ | AbsolutePressureAtTwoAltitudesBFCL/parallel_102: two calc_absolute_pressure calls — gauge 2.5 with atm 1 (default; may be omitted) and gauge 2.5 with atm 0.85 (non-default → required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 521 OUT2 948 |
| × | BoilingMeltingPointsAcrossSeaLevelsBFCL/parallel_100: four get_boiling_melting_points calls — water@0, iron@1000, water@1000, iron@0. FLAG parallel_100: BORDERLINE. The prompt ("water and iron at sea levels of 0 and 1000 meters respectively") reads most naturally as TWO calls (water@0, iron@1000), but the BFCL ground truth is the full 2x2 cross product (FOUR calls). Kept BFCL-faithful to the ground truth; a competent human could reasonably answer with two calls, so this item's expected call count is not fully deterministic.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 4
expectedCount = 2 |
Failed | IN8 700 OUT2 448 |
| ✓ | CellInfoDetailedAndBriefBFCL/parallel_108: two biology_get_cell_info calls — neuron with detailed=true (prompt: "detailed description"), muscle with detailed=false ("less detailed", the default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 408 OUT2 198 |
| ✓ | DensityForTwoSubstancesBFCL/parallel_101: two calculate_density calls (mass/volume schema) — 10 kg / 2 m³ and 15 kg / 3 m³. unit "kg/m³" equals the default → may be omitted or "kg/m³".1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 536 OUT929 |
| ✓ | ElectricFieldStrengthAtSeveralDistancesBFCL/parallel_98: four calculate_electric_field_strength calls — charge 2, distances 0.5/1/2 in vacuum (default medium), plus distance 1 in air. The distance-1 vacuum and distance-1 air calls are distinguished by medium. Vacuum equals the default → may be omitted or "vacuum"; "air" is required.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 831 OUT2 118 |
| ✓ | EntropyChangeForTwoConditionsBFCL/parallel_103: two entropy_change_calculate calls — substance A, mass 2, temps 25->75 and 10->50. pressure 1 equals the default → may be omitted or 1.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 642 OUT1 142 |
| ✓ | FetchDnaSequenceThreeWaysBFCL/parallel_106: three fetch_DNA_sequence calls for id XYZ123 — default fasta/0, then genbank format, then 500 bp upstream. Defaults (fasta, upstream 0) may be omitted.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 613 OUT720 |
| ✓ | FunctionLookupForTwoMoleculesBFCL/parallel_110: two cell_biology_function_lookup calls — ATP in mitochondria and DNA in nucleus, both with specific_function=true (prompt asks for a specific function within the organelle).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 469 OUT1 504 |
| ✓ | HeatCapacityForThreeConditionsBFCL/parallel_105: three calc_heat_capacity calls — 300K/2.5m³, 350K/2.5m³, 300K/1.5m³, all for air. "air" equals the default gas → may be omitted or "air".1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 717 OUT1 712 |
| ✓ | MolecularWeightForTwoCompoundsBFCL/parallel_111: two calculate_molecular_weight calls — Glucose (C6H12O6) and Sucrose (C12H22O11), both in grams/mole (prompt explicitly requests that unit; ground truth also accepts "g/mol").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 470 OUT1 334 |
| ✓ | PhaseChangeEnergyForWaterBFCL/parallel_99: four thermo_calculate_energy calls — 500 g water, phase melting/freezing/ vaporization/condensation. Substance "water" equals the default → may be omitted or "water".1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 879 OUT3 169 |
| ✓ | ProteinSequencesForTwoGenesTwoSpeciesBFCL/parallel_107: four get_protein_sequence calls — BRCA1 and BRCA2 in Homo sapiens (default species, may be omitted) and in Pan troglodytes (non-default → required).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 686 OUT3 618 |
| ✓ | ProteinsInThreeCompartmentsBFCL/parallel_109: three cellbio_get_proteins calls — nucleus/mitochondria/cytoplasm, each with include_description=true (prompt asks for "a brief description of each protein").1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 563 OUT3 157 |
Tool Calling - Parallel Calls: Physical & Life Sciences (set 3)
11 passed2 failed |
84.6% |
IN113 523 OUT27 882 | |
| ✓ | AnalyzeDnaForThreeMutationTypesBFCL/parallel_114: six analyze_dna_sequence calls — two sequences ("AGCTTAGCTA" and "AGCTTAGGCTA") against reference "AGCTTAGCTA", each checked for insertion (default, may be omitted), deletion, and substitution mutations.1 AI query
6 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL |
Passed | IN9 181 OUT3 016 |
| ✓ | DiabetesPredictionForFourPeopleBFCL/parallel_113: four diabetes_prediction calls for four people with distinct weight/height/activity.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 962 OUT1 917 |
| ✓ | EntropyChangeIsothermalAndNotBFCL/parallel_104: two calculate_entropy_change calls — same 300->350 K and heat_capacity 4.18, once isothermal (the default true, may be omitted) and once not isothermal (false, required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 624 OUT1 888 |
| ✓ | ForestGrowthForTwoRegionsBFCL/parallel_120: two forest_growth_forecast calls — Amazon Rainforest over 10 years and Boreal Forests of Canada over 20 years, excluding human impact (false = the default, may be omitted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 515 OUT1 207 |
| ✓ | GenerateFourDnaSequencesBFCL/parallel_123: four generate_DNA_sequence calls — length 500 with a single-nucleotide preference of A, T, C, G respectively.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 822 OUT1 008 |
| × | GeneticSimilarityInTwoFormatsBFCL/parallel_115: four genetics_calculate_similarity calls — human vs chimpanzee and human vs gorilla, each in percentage (default, may be omitted) and fraction. species1 accepts "human"/"Human".1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 812 OUT2 432 |
| ✓ | GenotypeFrequenciesForThreeGenotypesBFCL/parallel_116: three calculate_genotype_frequency calls — allele frequency 0.7, genotypes AA/Aa/aa.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 679 OUT1 407 |
| × | IdentifyFourBirdsBFCL/parallel_119: four identify_bird calls — small blue/forest (default size, may be omitted), large black/lake, medium brown/desert, large green/tropical rainforest.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 921 OUT3 225 |
| ✓ | MutationTypeForTwoSnpsBFCL/parallel_112: two mutation_type_find calls — rs123456 in Homo sapiens (default species, may be omitted) and rs7891011 in Canis lupus familiaris (non-default → required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 537 OUT3 942 |
| ✓ | PopulationDensityForChinaTwoYearsBFCL/parallel_117: two calculate_density calls (country/population schema) — China in 2000 (pop 1.267 billion) and 2010 (pop 1.341 billion), land area 9.597 million km² both times.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 608 OUT1 796 |
| ✓ | PrecipitationStatsForThreeTimeFramesBFCL/parallel_118: three ecology_data_precipitation_stats calls — Amazon rainforest for six_months, year, five_years.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 610 OUT1 863 |
| ✓ | TurtlePopulationForTwoYearsBFCL/parallel_121: two ecology_get_turtle_population calls — Galapagos Islands in 2015 and 2020, both with species info requested (species=true; years stated explicitly → required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 530 OUT2 368 |
| ✓ | VehicleEmissionsForThreeVehiclesBFCL/parallel_122: three calculate_vehicle_emission calls — 15000 miles each: gas with default factor 1.4 (may be omitted), diesel with factor 2.7, EV with factor 0.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 722 OUT1 813 |
Tool Calling - Parallel Calls: Services & Lookups
11 passed1 failed |
91.7% |
IN98 376 OUT15 892 | |
| ✓ | CalculateShortestAndScenicDistancesForTwoLegsBFCL/parallel_190: four calculate_shortest_distance calls — {NYC->LA, LA->Miami} x {Shortest, Scenic}.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 368 OUT1 808 |
| ✓ | FetchNewsForTwoTopicsInEuropeBFCL/parallel_192: two get_news calls — Climate Change and Artificial Intelligence, 5 articles each, region Europe. Region "Europe" explicitly requested -> required (accept "Europe"/"EU").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 049 OUT1 184 |
| ✓ | FindLibrariesWithFacilitiesInThreeCitiesBFCL/parallel_191: three public_library_find_nearby calls — NY[Reading Room,Fiction], LA[Wi-Fi,Children Section], Chicago[Cafe,Reading Room]. Facilities compared as an unordered pair.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 243 OUT1 132 |
| ✓ | FindParksWithAmenitiesInThreeCitiesBFCL/parallel_189: three parks_find_nearby calls — NY[Tennis Court,Picnic Area], LA[Playground,Running Track], Chicago[Tennis Court,Playground]. Amenities compared as an unordered pair.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 248 OUT2 044 |
| ✓ | FindUpcomingEventsInThreeCitiesBFCL/parallel_194: three event_finder_find_upcoming calls — jazz/LA(14d), rock/Chicago(10d), classical/Boston(7d). All day windows explicitly stated -> days_ahead required (7 for Boston, which also equals the default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 261 OUT1 229 |
| ✓ | GetCurrentWeatherForFourCitiesBFCL/parallel_199: four get_current_weather calls — New York, Los Angeles, London, Tokyo, with both temperature and humidity included. Prompt explicitly asks to include both -> flags == true (also the default).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 343 OUT1 442 |
| ✓ | GetDetailedBriefsForTwoMoviesBFCL/parallel_195: two movie_details_brief calls — Inception and The Dark Knight, extra_info true. Prompt asks for Director/Cast/Awards -> extra_info == true required.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 027 OUT1 432 |
| ✓ | GetLawsuitDetailsForTwoCasesBFCL/parallel_196: two get_lawsuit_details calls — case 12345 (NY Supreme Court, with verdict) and case 67890 (LA Superior Court, no verdict). "would like the verdict" -> true; "do not need" -> null/false.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 176 OUT863 |
| × | GetLawsuitInfoForTwoCasesByYearBFCL/parallel_197: two lawsuit_info calls — 12345ABC (2018, New York) and 67890XYZ (2019, California). Years are explicitly stated -> required. First location "New York" may be omitted (matches the default).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 151 OUT1 315 |
| ✓ | GetStockPricesForTwoCompaniesOverDaysBFCL/parallel_179: two get_stock_price (by-days variant) calls — Apple on NASDAQ for 10 days, Microsoft on NYSE for 15 days. Both exchanges are explicitly named -> required (NASDAQ / NYSE). The tool's 'company' field is a ticker symbol; company names are accepted as synonyms.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 050 OUT1 038 |
| ✓ | SearchLawsuitsForTwoEntitiesInCaliforniaBFCL/parallel_198: two lawsuit_search calls — Google/Santa Clara and Facebook/San Mateo, state California. Prompt states "both in the state of California" -> state required (accept "California"/"CA").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 087 OUT992 |
| ✓ | SendTwoEmailsWithCcAndBccBFCL/parallel_193: two send_email calls — Project Update to john.doe (cc manager, bcc hr) and Meeting Reminder to jane.doe (cc assistant, bcc hr). Bodies matched exactly.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 373 OUT1 413 |
Tool Calling - Parallel Calls: Statistics & Finance
13 passed |
100.0% |
IN117 619 OUT25 231 | |
| ✓ | CalculateBinomialProbabilitiesForBasketballShotsBFCL/parallel_163: three calc_binomial_prob calls — (10,6,0.6), (10,6,0.5), (15,6,0.5).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 157 OUT3 147 |
| ✓ | CalculateCompoundInterestForThreePeriodsBFCL/parallel_170: three calculate_compound_interest calls — principal 5000, rate 0.025, time 2/3/5. Prompt states "compounded quarterly, 4 times in a year" -> n == 4 required (tightened; also ground truth).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 110 OUT2 023 |
| ✓ | CalculateDiscountedCashFlowForTwoBondsBFCL/parallel_169: two calculate_discounted_cash_flow calls — (50,5,0.05) and (60,7,0.04). Prompt explicitly states "face value of the bond is $1000" for both -> face_value == 1000 required (ground truth also accepts omission via the default, but an explicitly-stated value must be supplied).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 994 OUT1 925 |
| ✓ | CalculateEventProbabilitiesForCardDeckBFCL/parallel_162: three probability_of_event calls — Ace 4/52 as decimal, heart 13/52 as decimal, red card 26/52 as ratio. format_as_ratio: "as a decimal" -> false or omitted; "as a ratio" -> true (required).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 124 OUT1 415 |
| ✓ | CalculateQuarterlyDividendForThreeScenariosBFCL/parallel_168: three finance_calculate_quarterly_dividend_per_share calls — (5M,2M), (6M,2M), (6M,2.5M).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 180 OUT1 445 |
| ✓ | CalculateReturnOnEquityForTwoCompaniesBFCL/parallel_171: two calculate_return_on_equity calls — (1M,5M,200k) and (2M,10M, no dividends). Second company "did not pay any dividends" -> dividends_paid null or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 955 OUT1 491 |
| ✓ | CalculateRoiForThreeStocksBFCL/parallel_174: three calculate_return_on_investment calls — (150,180,20), (200,210,30), (250,300,40). Dividends are explicitly stated per stock -> required.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 096 OUT1 383 |
| ✓ | ComputeRSquaredForTwoRegressionModelsBFCL/parallel_167: two linear_regression_get_r_squared calls on the same CSV — swap education/credit_score as dependent variable. Independent-variable order follows the prompt/ground truth.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 930 OUT3 608 |
| ✓ | PredictFutureValueForTwoInvestmentsBFCL/parallel_172: two finance_predict_future_value calls — (5000,0.05,10) and (7000,0.04,15). Prompt states "compound interest annually" -> compounding_periods_per_year == 1 required (tightened; also the default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 964 OUT1 457 |
| ✓ | PredictHousePricesForThreeHousesBFCL/parallel_166: three predict_house_price calls — NY 2500/3/2000, LA 3000/3/2005, Chicago 2000/2/1995. The second house is described as "similar" with only area/year/location changed, so rooms == 3 per ground truth.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 147 OUT2 484 |
| ✓ | PredictProfitForTwoFundsBFCL/parallel_173: two investment_predictProfit calls — (5000,0.07,5) and (8000,0.05,7).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 916 OUT1 175 |
| ✓ | RunChiSquaredTestForTwoContingencyTablesBFCL/parallel_164: two chi_squared_test calls — tables [45,55,35,65] and [30,70,50,50], alpha 0.05. Prompt states "significance level of 0.05 for both" -> alpha == 0.05 required (tightened). Table order is fixed.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 900 OUT1 992 |
| ✓ | RunTTestAtTwoSignificanceLevelsBFCL/parallel_165: two t_test calls — same datasets, alpha 0.05 then 0.01 (both stated -> required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 146 OUT1 686 |
Tool Calling - Simple: Arts, Venues & Music (set 1)
8 passed |
100.0% |
IN50 785 OUT11 464 | |
| ✓ | SimpleConcertDetailsBeyonceBFCL/simple_python_286 (concert.get_details): artist=Beyonce, location=San Diego, date=April 2022.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 358 OUT954 |
| ✓ | SimpleConcertInfoPopNyBFCL/simple_python_284 (concert_info.get): location=New York, date=next month, genre=Pop.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 333 OUT1 810 |
| ✓ | SimpleGenerateMelodyC4BFCL/simple_python_291 (music_generator.generate_melody): key=C, start_note=C4, length=16, tempo=120 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 381 OUT1 068 |
| ✓ | SimpleKeySignatureCsharpBFCL/simple_python_298 (music_theory.key_signature): key='C#', scale_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 318 OUT795 |
| ✓ | SimpleMuseumHoursLouvreBFCL/simple_python_273 (museum.get_hours): name='Louvre Museum', location=Paris, day omitted or Monday.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 325 OUT1 380 |
| ✓ | SimplePaintNeededWall30x12BFCL/simple_python_259 (calculate_paint_needed): coverage_rate=400, length=30, height=12.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 381 OUT1 177 |
| ✓ | SimpleReligionHistoryChristianity16cBFCL/simple_python_252 (get_religion_history): religion=Christianity, century=16, sort_by=importance, count=10.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 358 OUT2 733 |
| ✓ | SimpleSculptureInfoPlensaBFCL/simple_python_263 (get_sculpture_info): artist_name='James Plensa', detail=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 331 OUT1 547 |
Tool Calling - Simple: Arts, Venues & Music (set 2)
6 passed2 failed |
75.0% |
IN53 844 OUT11 157 | |
| ✓ | SimpleComposeMelodyCFGBFCL/simple_python_292 (compose_melody): progression=[C,F,G], measures=4, instrument=Piano or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 319 OUT1 007 |
| ✓ | SimpleConcertFindDetailsWeekndBFCL/simple_python_290 (concert.find_details): artist=The Weeknd, month=December, year=2022 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 297 OUT1 407 |
| × | SimpleFindConcertRockChicagoBFCL/simple_python_285 (find_concert): location=Chicago, price=100, genre=Rock.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["chicago, illinois", "chicago, il"]) |
Failed | IN6 305 OUT1 630 |
| ✓ | SimpleMuseumInfoMetBFCL/simple_python_274 (museum_info): museum_name=Metropolitan Museum of Art, info_type=opening_hours or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 295 OUT1 021 |
| ✓ | SimpleMusicalScaleCsharpBFCL/simple_python_299 (musical_scale): key='C#'/'C sharp', scale_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 277 OUT1 438 |
| ✓ | SimplePaintRequirementWallExclusionBFCL/simple_python_260 (paint_requirement.calculate): area {width=20,height=12}, paint_coverage=350, exclusion {type=window, area=15}.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 402 OUT1 475 |
| ✓ | SimpleReligionHistoryChristianity300To400BFCL/simple_python_254 (get_religion_history): religion=Christianity, start_year=300, end_year=400, event_type=all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 322 OUT952 |
| × | SimpleSculptureDetailsDavidBFCL/simple_python_264 (sculpture.get_details): artist=Michelangelo, title=David, detail=size.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 627 OUT2 227 |
Tool Calling - Simple: Arts, Venues & Music (set 3)
9 passed |
100.0% |
IN60 208 OUT10 195 | |
| ✓ | SimpleChordProgressionKeyBFCL/simple_python_297 (music.theory.chordProgression): progression=[I,V,vi,IV], bools any.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 693 OUT744 |
| ✓ | SimpleCompoundInterest10kBFCL/simple_python_269 (calculate_compound_interest): principle=10000, interest_rate=0.05, time=10, compounds_per_year=1 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 736 OUT1 842 |
| ✓ | SimpleConcertSearchClassicalLaBFCL/simple_python_287 (concert.search): genre=classical, location=Los Angeles/LA, date='this weekend', price_range=cheap.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 692 OUT1 198 |
| ✓ | SimpleCreateMixCMajorBFCL/simple_python_293 (music_composer.create_mix): scale='C Major', note_duration=quarter, track_length=180.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 715 OUT906 |
| ✓ | SimpleGenerateCircleImageRedBFCL/simple_python_256 (generate_circle_image): radius=50, color=Red, background=white or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 684 OUT554 |
| ✓ | SimpleMuseumWorkingHoursLouvreBFCL/simple_python_276 (museum_working_hours.get): museum=Louvre, location=Paris, day=Monday or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 670 OUT1 292 |
| ✓ | SimpleRetrieveReligionInfoBuddhismBFCL/simple_python_253 (retrieve_religion_info): religion_name=Buddhism, detail_level=full.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 665 OUT1 871 |
| ✓ | SimpleSculptureSearchChicago19cBFCL/simple_python_265 (sculpture_search): location=Chicago, time_frame='19th century', material=all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 682 OUT1 216 |
| ✓ | SimpleTimeDifferenceSfSydneyBFCL/simple_python_250 (get_time_difference): place1=San Francisco/SF, place2=Sydney.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 671 OUT572 |
Tool Calling - Simple: Arts, Venues & Music (set 4)
8 passed1 failed |
88.9% |
IN56 906 OUT11 912 | |
| ✓ | SimpleBuildingDimensionsEmpireStateBFCL/simple_python_270 (building.get_dimensions): building_name=Empire State Building, unit=feet.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 320 OUT921 |
| ✓ | SimpleConcertFindNearbySeattleBFCL/simple_python_289 (concert.find_nearby): location=Seattle, genre=jazz.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 311 OUT1 157 |
| ✓ | SimpleCreateChordProgressionCBFCL/simple_python_294 (music_generation.create_chord_progression): key=C, chords=4, progression_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 324 OUT875 |
| ✓ | SimpleEarliestReferenceJesusBFCL/simple_python_251 (get_earliest_reference): name='Jesus Christ', source='historical records'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 318 OUT1 978 |
| ✓ | SimpleFindInstrumentAcousticBFCL/simple_python_280 (find_instrument): budget=1000, type=acoustic, make omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 317 OUT1 268 |
| ✓ | SimpleGuitarPriceLesPaulBFCL/simple_python_283 (guitar_price.find): model='Gibson Les Paul', condition=Excellent, location=Chicago.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 331 OUT1 924 |
| ✓ | SimpleIdentifyColorSeaGreenBFCL/simple_python_257 (identify_color_rgb): color_name='Sea Green', standard=basic or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 315 OUT1 108 |
| × | SimpleMuseumInfoBritishDateBFCL/simple_python_277 (museum_info): museum='The British Museum', date=2023-06-20, information=[opening_hours,ticket_price] (either order).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.information.Count == 2
Values:
args.information.Count = 0 |
Failed | IN6 345 OUT1 777 |
| ✓ | SimpleSculptureValueThinkerBFCL/simple_python_266 (get_sculpture_value): sculpture='The Thinker', artist=Rodin.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 325 OUT904 |
Tool Calling - Simple: Arts, Venues & Music (set 5)
9 passed |
100.0% |
IN57 649 OUT8 900 | |
| ✓ | SimpleCircleDimensionsRadius5BFCL/simple_python_272 (calculate_circle_dimensions): radius=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 384 OUT613 |
| ✓ | SimpleDrawRectangleRedBFCL/simple_python_261 (draw_rectangle): width=20, height=10, color=red.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 412 OUT663 |
| ✓ | SimpleFindFluteYamahaBFCL/simple_python_282 (find_flute): brand=Yamaha, specs=[open hole, C foot, silver headjoint].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 416 OUT1 196 |
| ✓ | SimpleGenerateScaleProgressionCBFCL/simple_python_296 (music_generator.generate_scale_progression): key=C, tempo=80, duration=4, scale_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 419 OUT353 |
| ✓ | SimpleInstrumentDetailsPianoYamahaBFCL/simple_python_278 (get_instrument_details): instrument=piano, manufacturer=Yamaha, features=[price,rating] (either order).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 399 OUT1 559 |
| ✓ | SimpleMetTopArtworksBFCL/simple_python_275 (metropolitan_museum.get_top_artworks): number=5, sort_by=popularity or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 410 OUT1 038 |
| ✓ | SimpleMixPaintYellowBlueBFCL/simple_python_258 (mix_paint_color): color1=yellow, color2=blue, lightness=60.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 406 OUT310 |
| ✓ | SimplePapalBiographyInnocentIIIBFCL/simple_python_255 (religious_history.get_papal_biography): papal_name=Innocent III, include_contributions=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 395 OUT1 735 |
| ✓ | SimpleSculptureLocatorMichelangeloBFCL/simple_python_268 (sculpture_locator.find_by_artist): artist=Michelangelo, material=Marble, location=Rome.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 408 OUT1 433 |
Tool Calling - Simple: Arts, Venues & Music (set 6)
6 passed1 failed |
85.7% |
IN41 874 OUT6 291 | |
| ✓ | SimpleAnalyzeStructureB1004BFCL/simple_python_271 (analyze_structure): building_id=B1004, floors=[2,3,4], mode=dynamic.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 006 OUT533 |
| ✓ | SimpleConcertBookingEminemBFCL/simple_python_288 (concert_booking.book_ticket): artist=Eminem, city=New York City, num_tickets=2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 966 OUT663 |
| × | SimpleFindExhibitionNyBFCL/simple_python_267 (find_exhibition): location='New York City, NY', art_form=sculpture, month omitted, user_ratings=high.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["new york city, ny"]) |
Failed | IN5 972 OUT2 036 |
| ✓ | SimpleInstrumentInfoViolin1721BFCL/simple_python_281 (get_instrument_info): name=Violin, maker=Stradivarius, year=1721.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 000 OUT461 |
| ✓ | SimpleInstrumentPriceStratocasterBFCL/simple_python_279 (instrument_price.get): brand=Fender, model='American Professional II Stratocaster', finish=Rosewood.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 980 OUT885 |
| ✓ | SimpleModifyPaintingOilBFCL/simple_python_262 (modify_painting): size='12x18', medium=oil, dominant_color=red.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 983 OUT546 |
| ✓ | SimpleSongLyricsBohemianBFCL/simple_python_295 (get_song_lyrics): song_title='Bohemian Rhapsody', artist_name=Queen, lang=English or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 967 OUT1 167 |
Tool Calling - Simple: Food, Travel & Shopping (set 1)
7 passed1 failed |
87.5% |
IN51 764 OUT10 748 | |
| ✓ | SimpleCookingConvertPoundsToOuncesBFCL/simple_python_365 (cooking_conversion.convert): 2 pounds->ounces of butter.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 466 OUT987 |
| ✓ | SimpleCurrencyExchangeUsdToCadBFCL/simple_python_388 (currency_exchange.convert): USD->CAD, amount=500.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 466 OUT1 458 |
| × | SimpleGroceryStoreOrganicHoustonBFCL/simple_python_369 (grocery_store.find_nearby): Houston, categories {Organic,Fruits,Vegetables} any order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.categories.Count >= 2
Values:
args.categories.Count = 1 |
Failed | IN6 459 OUT1 193 |
| ✓ | SimpleHotelBookPetFriendlyManhattanBFCL/simple_python_380 (hotel_booking): Manhattan, single, 3 nights, 2023-03-10, pet_friendly.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 529 OUT820 |
| ✓ | SimpleMapsDistanceEiffelLouvreBFCL/simple_python_394 (maps.get_distance_duration): Eiffel Tower -> Louvre Museum, traffic omitted/false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 467 OUT1 293 |
| ✓ | SimpleRecipeFindByDietMealBFCL/simple_python_353 (find_recipes): diet=gluten-free, meal_type=dinner, ingredients omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 447 OUT1 485 |
| ✓ | SimpleRestaurantFinderItalianNYCBFCL/simple_python_361 (restaurant_finder): city=NYC, cuisine=Italian, diet=Gluten-free.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 462 OUT1 522 |
| ✓ | SimpleTimeZoneLondon24hBFCL/simple_python_376 (time_zone_converter): city=London, country=UK, display_format=24h.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 468 OUT1 990 |
Tool Calling - Simple: Food, Travel & Shopping (set 2)
8 passed1 failed |
88.9% |
IN63 435 OUT9 021 | |
| ✓ | SimpleBestSushiPlacesTokyoBFCL/simple_python_362 (get_best_sushi_places): city=Tokyo, top=5, review_rate=4.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 054 OUT821 |
| ✓ | SimpleCurrencyConverterUsdToGbpBFCL/simple_python_389 (currency_converter): USD->GBP, amount=200.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 049 OUT1 412 |
| ✓ | SimpleCurrentTimeSydneyCityBFCL/simple_python_377 (get_current_time): city=Sydney, country=Australia, format omitted/HH:MM:SS.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 020 OUT1 249 |
| ✓ | SimpleDistanceCalcNyToBostonTerrainBFCL/simple_python_397 (distance_calculator.calculate): New York -> Boston, consider_terrain=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 036 OUT1 197 |
| ✓ | SimpleHiltonAvailabilityParisBFCL/simple_python_381 (hilton_hotel.check_availability): Paris, 2023-04-04..08, 2 adults, chain Hilton/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 111 OUT747 |
| ✓ | SimpleRecipeCaloriesBeefLasagnaBFCL/simple_python_355 (recipe_info.get_calories): website=Foodnetwork.com, recipe=Beef Lasagna, meal_time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 035 OUT1 412 |
| ✓ | SimpleRecipeUnitTbspToTspBFCL/simple_python_366 (recipe.unit_conversion): 2 tablespoon->teaspoon, precision omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 041 OUT586 |
| × | SimpleSafewayOrderPaloAltoBFCL/simple_python_370 (safeway.order): Palo Alto, [olive oil, rice], [3,1]. EXPECTED MODEL MISS (kept, do not loosen): "a five pound bag of rice" = quantity 1 (5 is the weight, not the count); models tend to send quantity 5. This is a genuine model error the test correctly catches.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.quantity[1] == 1
Values:
args.quantity[1] = 5 |
Failed | IN7 056 OUT775 |
| ✓ | SimpleVeganSoupRecipeUnder30BFCL/simple_python_354 (get_vegan_recipe): dish_type=soup, cooking_time=30, ingredient_preference omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 033 OUT822 |
Tool Calling - Simple: Food, Travel & Shopping (set 3)
8 passed |
100.0% |
IN51 974 OUT8 814 | |
| ✓ | SimpleBookHotelHiltonChicagoBFCL/simple_python_382 (book_hotel): Hilton Hotel, Chicago, single, 2022-12-10, 2 nights.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 543 OUT921 |
| ✓ | SimpleCookingTimeRoastChickenBFCL/simple_python_368 (calculate_cooking_time): weight_kg=1.5, method omitted/roast, temp omitted/180.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 496 OUT1 262 |
| ✓ | SimpleCurrencyConvertEurToCadBFCL/simple_python_390 (currency_conversion.convert): 150 EUR->CAD.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 483 OUT1 145 |
| ✓ | SimpleCurrentTimeSydneyLocationBFCL/simple_python_379 (get_current_time): location=Sydney, country=Australia, timezone omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 472 OUT1 136 |
| ✓ | SimpleParkingLotCentralParkBFCL/simple_python_395 (parking_lot.find_nearest): Central Park NY, radius=2, type omitted/public.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 488 OUT1 274 |
| ✓ | SimpleRecipeFinderVeganServes2BFCL/simple_python_356 (recipe_finder.find): servings=2, diet=vegan, prep_time=30.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 508 OUT1 113 |
| ✓ | SimpleRestaurantClosestSushiBostonBFCL/simple_python_363 (restaurant_search.find_closest): Boston, Sushi, amenities=[Patio].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 487 OUT1 256 |
| ✓ | SimpleWholeFoodsPriceLABFCL/simple_python_371 (whole_foods.check_price): Los Angeles, [tomatoes, lettuce].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 497 OUT707 |
Tool Calling - Simple: Food, Travel & Shopping (set 4)
7 passed |
100.0% |
IN41 990 OUT8 889 | |
| ✓ | SimpleBookRoomThePlazaBFCL/simple_python_383 (book_room): The Plaza, single, 2 nights.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 000 OUT761 |
| ✓ | SimpleConvertCurrencyJpyToUsdBFCL/simple_python_393 (convert_currency): 20000 JPY->USD.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 006 OUT1 694 |
| ✓ | SimpleFindRestaurantItalianBrooklynBFCL/simple_python_364 (find_restaurant): Brooklyn, Italian, Gluten-free.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 995 OUT719 |
| ✓ | SimpleGetRecipeVeganChocCakeBFCL/simple_python_357 (get_recipe): dish_name=(vegan) chocolate cake, diet_preference=vegan.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 987 OUT1 697 |
| ✓ | SimpleHospitalLocateDenverBFCL/simple_python_396 (hospital.locate): Denver CO, radius=5, department=Pediatrics.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 002 OUT1 269 |
| ✓ | SimpleTimezoneConvertNyToLondonBFCL/simple_python_378 (timezone.convert): 3pm, New York -> London.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 998 OUT1 447 |
| ✓ | SimpleWholeFoodsTopBrandsBananasBFCL/simple_python_372 (whole_foods.find_top_brands): product=bananas, number omitted/5, organic=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 002 OUT1 302 |
Tool Calling - Simple: Food, Travel & Shopping (set 5)
4 passed2 failed |
66.7% |
IN35 447 OUT7 451 | |
| × | SimpleExchangeRateFeeGbpJpyBFCL/simple_python_391 (get_exchange_rate_with_fee): GBP->JPY, fee=0.02.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.base_currency?.ToLower()).AnyOf(["gbp"]) |
Failed | IN5 895 OUT2 117 |
| ✓ | SimpleHotelBookingBookParisBFCL/simple_python_384 (hotel_booking.book): Paris, 07-10-2022..07-20-2022, 2 adults, 1 child, Standard/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 959 OUT647 |
| ✓ | SimpleMuseumHoursMetSaturdayBFCL/simple_python_398 (get_museum_hours): museum=Metropolitan Museum of Art/The Met, day=Saturday.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 873 OUT786 |
| × | SimpleRecipeSearchGfCookieBFCL/simple_python_358 (recipe_search): diet=[gluten free], time_limit=30, dish=cookie.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.dish?.ToLower()).AnyOf(["cookie"]) |
Failed | IN5 884 OUT1 552 |
| ✓ | SimpleRestaurantSearchRatedNYCBFCL/simple_python_399 (restaurant_search): NYC, Italian, rating=4, accepts_credit_cards=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 906 OUT900 |
| ✓ | SimpleWalmartPurchaseSanJoseBFCL/simple_python_373 (walmart.purchase): San Jose, [apples, rice, (bottled) water], pack_size=[1,1,12].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 930 OUT1 449 |
Tool Calling - Simple: Food, Travel & Shopping (set 6)
5 passed |
100.0% |
IN27 812 OUT6 587 | |
| ✓ | SimpleGameHighestScoringPlayerBFCL/simple_python_350 (get_highest_scoring_player): game=Valorant, season=2022.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 554 OUT916 |
| ✓ | SimpleHotelBookRoomKingLABFCL/simple_python_385 (hotel_bookings.book_room): LA, king size, 15-10-2023, 2 nights, rooms omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 601 OUT1 156 |
| ✓ | SimpleLatestExchangeGbpJpyBFCL/simple_python_392 (latest_exchange_rate): GBP->JPY, amount omitted/1.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 540 OUT1 440 |
| ✓ | SimpleNutritionalInfoAvocadoBFCL/simple_python_374 (grocery_info.nutritional_info): Walmart, avocado, [Protein,Calories,Carbohydrates].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 561 OUT1 682 |
| ✓ | SimpleRecipeSearchVegPastaBFCL/simple_python_359 (recipe_search): dietary_restriction=Vegetarian, ingredients=[pasta,cheese], servings=2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 556 OUT1 393 |
Tool Calling - Simple: Food, Travel & Shopping (set 7)
3 passed1 failed |
75.0% |
IN20 847 OUT4 307 | |
| ✓ | SimpleBookHotelLuxuryParadiseBFCL/simple_python_386 (book_hotel): Hotel Paradise, Las Vegas, luxury, 05-12-2022, 3 days, view=city.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 264 OUT1 011 |
| ✓ | SimpleFindRecipeCaloriesCarbonaraBFCL/simple_python_360 (find_recipe): recipeName=pasta carbonara, maxCalories=500.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 193 OUT637 |
| ✓ | SimpleGameMultiplayerFinderBFCL/simple_python_351 (multiplayer_game_finder): platform=Windows 10, rating=4.5, genre omitted/Action.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 196 OUT949 |
| × | SimpleWalmartPriceCheckPumpkinsBFCL/simple_python_375 (walmart.check_price): [pumpkins,eggs], quantities [3,24] or [3,2], store omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.items[1].ToLower()).AnyOf(["eggs", "dozen eggs"]) |
Failed | IN5 194 OUT1 710 |
Tool Calling - Simple: Food, Travel & Shopping (set 8)
1 passed2 failed |
33.3% |
IN14 596 OUT5 387 | |
| × | SimpleFindRecipeVeganBrowniesBFCL/simple_python_367 (find_recipe): dietary_restrictions=vegan, recipe_type=dessert (brownies), time=30.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.recipe_type?.ToLower()).AnyOf(["dessert"]) |
Failed | IN4 848 OUT1 720 |
| ✓ | SimpleGameAverageUserScoreBFCL/simple_python_352 (gamespot.getAverageUserScore): game=Zelda BotW, platform=Nintendo Switch/all platforms.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 858 OUT2 890 |
| × | SimpleHotelBookingPlazaRangeBFCL/simple_python_387 (hotel_booking): Plaza Hotel, New York NY, 2022-06-01..2022-06-04, rooms omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["new york city, ny", "new york, ny"]) |
Failed | IN4 890 OUT777 |
Tool Calling - Simple: Knowledge, Places & History (set 1)
10 passed |
100.0% |
IN69 290 OUT14 057 | |
| ✓ | SimpleAirQualityChicagoBFCL/simple_python_203 (get_air_quality): location=Chicago, detail=true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 923 OUT1 360 |
| ✓ | SimpleEvolutionScientistBFCL/simple_python_242 (get_scientist_for_discovery): discovery=Theory of Evolution.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 914 OUT979 |
| ✓ | SimpleGermanWarEventsBFCL/simple_python_231 (history.get_key_events): country=Germany/DE, 1871-1945, event_type=[War].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 957 OUT2 068 |
| ✓ | SimpleKingOfFranceBFCL/simple_python_230 (historic_leader_search): location=France, date=1510, title omitted or King.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 928 OUT1 453 |
| ✓ | SimplePandaPopulationBFCL/simple_python_201 (estimate_population): species=panda(s), country=China/CN, year omitted or 2024.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 915 OUT2 035 |
| ✓ | SimplePersonalityTraitsTypeBFCL/simple_python_227 (get_personality_traits by type): type=ENFJ, traits=[strengths, weaknesses].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 926 OUT1 421 |
| ✓ | SimplePresidentCivilWarBFCL/simple_python_238 (us_history.get_president): event=American Civil War, year in 1861-1865.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 925 OUT879 |
| ✓ | SimpleRestaurantNearbyBFCL/simple_python_204 (restaurant.find_nearby): location=Seattle[, WA], cuisine=Chinese, max_distance=10.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 934 OUT2 015 |
| ✓ | SimpleTordesillasDateBFCL/simple_python_233 (european_history.get_event_date): event=Treaty of Tordesillas, format=YYYY.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 939 OUT610 |
| ✓ | SimpleTrafficBostonNewYorkBFCL/simple_python_205 (get_traffic_info): start=Boston, end=New York/NYC, mode omitted or driving.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 929 OUT1 237 |
Tool Calling - Simple: Knowledge, Places & History (set 2)
7 passed3 failed |
70.0% |
IN68 953 OUT13 376 | |
| ✓ | SimpleBitcoinNewsBFCL/simple_python_210 (get_news): topic=Bitcoin, quantity=5, region omitted or US.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 890 OUT868 |
| × | SimpleEuropeanWars19cBFCL/simple_python_234 (history_eu.fetch_events): century=19, region=any of N/S/E/W, category=Wars. EXPECTED MODEL MISS (kept, do not loosen): region is an enum (Northern/Southern/Eastern/Western); the prompt says only "European", so any of the four is accepted, but models often send the invalid value "Europe". Failing to respect the enum is a genuine model error the test correctly catches.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.region.ToLower().AnyOf(["northern", "southern", "eastern", "western"]) |
Failed | IN6 905 OUT2 028 |
| ✓ | SimpleLibraryNearbyBFCL/simple_python_209 (public_library.find_nearby): location=Boston MA/Massachusetts, facilities=[Fiction, Wi-Fi] (any order).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 915 OUT1 422 |
| × | SimpleMonarchOfEnglandBFCL/simple_python_232 (monarch.getMonarchOfYear): location=England/ENG, year=1800, fullName=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.location.ToLower().AnyOf(["england", "eng"]) |
Failed | IN6 908 OUT802 |
| ✓ | SimpleNeutronDiscovererBFCL/simple_python_243 (get_discoverer): discovery=neutron, detail=true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 886 OUT923 |
| ✓ | SimpleParksNearbyBFCL/simple_python_206 (parks.find_nearby): location=London[, UK], amenities=[Tennis Court].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 892 OUT1 156 |
| ✓ | SimplePersonalityTraitsHobbyBFCL/simple_python_228 (get_personality_traits by hobby): hobby=jogging, trait_count=3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 889 OUT2 561 |
| ✓ | SimplePopulationGrowthBFCL/simple_python_221 (population_growth_estimate): location=London, years=5, rate omitted or 1.2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 892 OUT1 367 |
| × | SimplePresidentDuringEventBFCL/simple_python_241 (US_President_During_Event): event=Civil War, country omitted or USA.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: args.event.ToLower() == "civil war" |
Failed | IN6 894 OUT1 550 |
| ✓ | SimpleTreatyLisbonDateBFCL/simple_python_235 (get_event_date): event=Treaty of Lisbon (variants), location omitted or Lisbon[, Portugal].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 882 OUT699 |
Tool Calling - Simple: Knowledge, Places & History (set 3)
9 passed1 failed |
90.0% |
IN72 332 OUT9 282 | |
| ✓ | SimpleAppleStockInfoBFCL/simple_python_212 (get_stock_info): company=Apple Inc., detail_level=detailed, market omitted or NASDAQ.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 219 OUT1 018 |
| ✓ | SimpleCivilWarStartDateBFCL/simple_python_236 (us_history.get_event_info): event=American Civil War/Civil War, specific_info=Start Date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 220 OUT853 |
| ✓ | SimpleFlightBookBFCL/simple_python_213 (flight.book): departure=San Francisco/SF, destination=London, date=2022-04-27 (variants), time=afternoon, direct=true.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 274 OUT461 |
| × | SimpleGravitationPublicationYearBFCL/simple_python_244 (publication_year.find): author=Isaac Newton, work=Law of Universal Gravitation (variants), location omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.work_title.ToLower().AnyOf(["law of universal gravitation", "universal law of gravitation", "the law of universal gravitation"]) |
Failed | IN7 235 OUT1 137 |
| ✓ | SimpleMovieBriefBFCL/simple_python_215 (movie_details.brief): title=Interstellar, extra_info omitted or false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 209 OUT1 542 |
| ✓ | SimplePresidentIn1861BFCL/simple_python_239 (US_president.in_year): year=1861, full_name omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 229 OUT985 |
| ✓ | SimpleRadiumDiscovererBFCL/simple_python_245 (discoverer.get): element_name=radium (quote variants), year omitted or 0, first omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 199 OUT652 |
| ✓ | SimpleSendEmailBFCL/simple_python_211 (send_email): to=john.doe@example.com, subject=Meeting, body (opt trailing period), cc/bcc empty.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 276 OUT547 |
| ✓ | SimpleShortestDistanceBFCL/simple_python_207 (calculate_shortest_distance): start=New York (variants), end=Miami (variants), route_preference=Shortest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 243 OUT870 |
| ✓ | SimpleUpcomingConcertsBFCL/simple_python_214 (event_finder.find_upcoming): location=New York (variants), genre=Rock, days_ahead=30.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 228 OUT1 217 |
Tool Calling - Simple: Knowledge, Places & History (set 4)
9 passed1 failed |
90.0% |
IN71 788 OUT10 927 | |
| ✓ | SimpleBmiCalcBFCL/simple_python_222 (calculate_bmi): weight=70, height=180, unit omitted or metric.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 179 OUT952 |
| ✓ | SimpleCarEmissionsBFCL/simple_python_200 (calculate_emissions): distance=12000, fuel_type=gas, fuel_efficiency omitted or 25.0, efficiency_reduction omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 212 OUT1 926 |
| ✓ | SimpleFmriAnalyzeBFCL/simple_python_217 (fMRI.analyze): data_source path, sequence_type=multi-band, smooth=6, voxel_size=2.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 222 OUT491 |
| ✓ | SimpleGetDirectionsBFCL/simple_python_208 (map_service.get_directions): start=New York/NYC, end=Los Angeles/LA, avoid=[highways, tolls] (any order).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 173 OUT1 267 |
| ✓ | SimpleGravityDiscoveryBFCL/simple_python_246 (science_history.get_discovery_details): discovery=Gravity, method_used omitted or default.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 139 OUT1 238 |
| ✓ | SimpleGroupDynamicsBFCL/simple_python_223 (group_dynamics.pattern): total=50, extroverts=15, introverts=35.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 199 OUT853 |
| ✓ | SimpleMostFollowedBFCL/simple_python_224 (social_media_analytics.most_followed): topic=psychology, sub_topics=[behaviour, group dynamics], region omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 170 OUT1 043 |
| × | SimpleNeuronCoordinatesBFCL/simple_python_219 (get_neuron_coordinates): neuron_type=GABA, brain_region=All/all part of the brain/entire brain.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.brain_region?.ToLower()).AnyOf(["all", "all part of the brain", "entire brain"]) |
Failed | IN7 165 OUT1 416 |
| ✓ | SimplePresidentIn1940BFCL/simple_python_240 (history_api.get_president_by_year): year=1940, full_term_only omitted/true/false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 151 OUT757 |
| ✓ | SimpleSentimentAnalysisBFCL/simple_python_216 (sentiment_analysis): text=the review, language=english/en.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 178 OUT984 |
Tool Calling - Simple: Knowledge, Places & History (set 5)
10 passed |
100.0% |
IN73 086 OUT13 345 | |
| ✓ | SimpleBigFiveScoresBFCL/simple_python_229 (get_bigfive_scores): characteristics=[efficient, organized, easy going, compassionate], scale omitted or medium.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 316 OUT1 372 |
| ✓ | SimpleChristianityHistoryBFCL/simple_python_249 (religion.history_info): religion=Christianity, till_century=14, include_people omitted or false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 293 OUT2 396 |
| ✓ | SimpleEinsteinContributionBFCL/simple_python_247 (historical_contrib.get_contrib): scientist=Albert Einstein, date=1915-03-17 (variants), category omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 310 OUT734 |
| ✓ | SimpleHistoricalGdpBFCL/simple_python_237 (get_historical_GDP): country=United States/US, start_year=1960, end_year=2000.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 315 OUT1 048 |
| ✓ | SimpleMriReportBFCL/simple_python_218 (patient.get_mri_report): patient_id=546382, mri_type omitted or brain, status=concluded.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 324 OUT1 410 |
| ✓ | SimpleNeuronalActivityBFCL/simple_python_220 (calculate_neuronal_activity): input_synaptic_rate=200, weight=0.5, decay_rate=0.1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 334 OUT1 360 |
| ✓ | SimpleReadingPreferenceBFCL/simple_python_225 (psych_research.get_preference): category=reading, option_one=digital[reading], option_two=physical book(s), demographic omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 297 OUT1 244 |
| ✓ | SimpleRelativityInventionBFCL/simple_python_248 (science_history.get_invention): invention_name=theory of relativity, want_year=true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 287 OUT1 004 |
| ✓ | SimpleRenewableEmissionSavingsBFCL/simple_python_202 (calculate_emission_savings): energy_type=renewable, usage_duration=3, region=California/CA. EXPECTED MODEL MISS (kept, do not loosen): the prompt says "renewable energy sources" generically; models often over-specify a concrete type (e.g. "solar"). Ground truth wants "renewable".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 315 OUT1 432 |
| ✓ | SimpleZodiacCompatibilityBFCL/simple_python_226 (get_zodiac_compatibility): sign1=Aries, sign2=Gemini, scale omitted or percentage.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 295 OUT1 345 |
Tool Calling - Simple: Law, Finance & Records (set 1)
9 passed1 failed |
90.0% |
IN76 818 OUT12 353 | |
| ✓ | SimpleAirQualityByDateBFCL/simple_python_196 (air_quality): location London, date reformatted to 08-16-2022.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 287 OUT1 387 |
| × | SimpleCivilCaseDetailsWithDissentBFCL/simple_python_167 (law.civil.get_case_details): case_title Roe v. Wade, include_dissent true.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 105 OUT1 529 |
| ✓ | SimpleCompoundedInterestQuarterlyBFCL/simple_python_153 (calculate_compounded_interest): principal 5000, rate 0.03, time 5, n 4 (quarterly).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 323 OUT1 397 |
| ✓ | SimpleCrimeRateByCityBFCL/simple_python_164 (get_crime_rate): city San Francisco, state California/CA, type violent/omitted, year 2020.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 297 OUT1 244 |
| ✓ | SimpleCriminalRecordsByPersonBFCL/simple_python_158 (get_criminal_records): name Mr. X, location New York, NY, from_year 2012, to_year 2015.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 318 OUT1 022 |
| ✓ | SimpleDetailedWeatherForecastHoursBFCL/simple_python_185 (detailed_weather_forecast): location New York, duration 72 hours, include_precipitation true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 290 OUT919 |
| ✓ | SimpleFutureValueByDurationBFCL/simple_python_155 (calculate_future_value): initial_investment 1000, interest_rate 0.05, duration 2, compounded omitted or 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 309 OUT1 194 |
| ✓ | SimpleLawsuitDetailsWithExtraFieldsBFCL/simple_python_178 (get_lawsuit_details): case_number LAX2019080202, court_location Los Angeles, additional_details omitted or full default list.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 307 OUT817 |
| ✓ | SimpleLawsuitSearchByFilingDateBFCL/simple_python_168 (lawsuit_search): company Google/GOOG, start_date 01-01-2021, location California, status ongoing/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 307 OUT1 326 |
| ✓ | SimpleLawyerExperienceMonthsBFCL/simple_python_175 (lawyer.get_experience): name John Doe, law_type Bankruptcy.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 275 OUT1 518 |
Tool Calling - Simple: Law, Finance & Records (set 2)
8 passed2 failed |
80.0% |
IN70 819 OUT11 090 | |
| ✓ | SimpleAirQualityIndexByTimeBFCL/simple_python_197 (get_air_quality_index): location San Diego, time 12pm/12:00.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 055 OUT588 |
| ✓ | SimpleCivilCasesRetrieveTheftBFCL/simple_python_165 (civil_cases.retrieve): year 2020, crime_type theft, location Los Angeles.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 070 OUT780 |
| × | SimpleFindNearbyDivorceLawyerBFCL/simple_python_166 (lawyer.find_nearby): city Chicago IL, specialty [Divorce], fee 400. NOTE: single-call contract kept intentionally; the model occasionally emits a spurious second call (no other visible tool fits this query), which is a model-reliability signal, not a test defect.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.city?.ToLower()).AnyOf(["chicago, il.", "chicago, il"]) |
Failed | IN7 080 OUT1 290 |
| ✓ | SimpleGenerateLawContractRentalBFCL/simple_python_162 (generate_law_contract): parties [John, Alice], contract_type Rental Agreement, location California/CA.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 069 OUT927 |
| ✓ | SimpleLawsuitDetailsFindByCompanyBFCL/simple_python_176 (lawsuit_details.find): company_name Apple Inc., year 2010, case_type Patent/IPR.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 081 OUT1 325 |
| ✓ | SimpleLawsuitDetailsWithVerdictBFCL/simple_python_181 (get_lawsuit_details): case_number 123456-ABC, court_location Los Angeles, with_verdict true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 093 OUT1 252 |
| × | SimpleLocalNurseryAnnualPlantsBFCL/simple_python_193 (local_nursery.find): location Toronto, plant_types [Annual].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.plant_types[0]?.ToLower()).AnyOf(["annual", "annuals"]) |
Failed | IN7 059 OUT1 441 |
| ✓ | SimplePropertyRecordsWithOwnerBFCL/simple_python_163 (property_records.get): address 123 main street, parcel_number 1234567890, county Santa Clara, include_owner true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 141 OUT1 259 |
| ✓ | SimpleRoiSavingsAccountBFCL/simple_python_150 (calculate_roi): deposit 1000, annual_interest_rate 0.03, years 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 103 OUT1 181 |
| ✓ | SimpleWeatherForecastDetailedDaysBFCL/simple_python_189 (weather_forecast_detailed): location New York, days 3, details true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 068 OUT1 047 |
Tool Calling - Simple: Law, Finance & Records (set 3)
8 passed2 failed |
80.0% |
IN79 321 OUT14 851 | |
| ✓ | SimpleCaseInfoVictimBFCL/simple_python_160 (get_case_info): docket 2022/AL2562, court California/CA, info_type victim.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 202 OUT1 170 |
| ✓ | SimpleCrimeStatuteTheftBFCL/simple_python_161 (crime_statute_lookup): jurisdiction California/CA, crime theft, detail_level detailed.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 178 OUT1 051 |
| ✓ | SimpleCurrentWeatherCelsiusBFCL/simple_python_186 (current_weather_condition): city Tokyo, country Japan, measurement c/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 178 OUT1 398 |
| ✓ | SimpleEnvironmentalAirQualityDaysBFCL/simple_python_199 (environmental_data.air_quality_index): location San Jose, days 3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 162 OUT1 600 |
| × | SimpleFelonyCrimeRecordBFCL/simple_python_156 (crime_record.get_record): case_number CA123456, county San Diego, details true.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 967 OUT1 107 |
| ✓ | SimpleHighestGrossingBankBFCL/simple_python_151 (highest_grossing_banks): country U.S variants, year 2020, top_n 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 186 OUT2 632 |
| ✓ | SimpleHistoricalFraudCaseBFCL/simple_python_170 (law_case_search.find_historical): subject fraud, from_year 2010, to_year 2015.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 194 OUT723 |
| × | SimpleLawsuitCasesByCompanyYearBFCL/simple_python_177 (get_lawsuit_cases): company_name Facebook, year 2018, status all/omitted. EXPECTED MODEL MISS (kept, do not loosen): "Patent" is the case TYPE, not the status (enum is open/closed/all); status should be omitted/"all". Models often wrongly set status="Patent".1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 925 OUT2 679 |
| ✓ | SimpleLawsuitInfoByCaseNumberBFCL/simple_python_182 (lawsuit_info): case_number XYZ123, year omitted/2023, location omitted/all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 157 OUT255 |
| ✓ | SimplePlantsForSlopeErosionBFCL/simple_python_194 (get_plants_for_slope): slope_type hill/steep/moderate, num_results 3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 172 OUT2 236 |
Tool Calling - Simple: Law, Finance & Records (set 4)
10 passed |
100.0% |
IN73 552 OUT12 583 | |
| ✓ | SimpleActDetailsAmendmentBFCL/simple_python_159 (get_act_details): act_name Criminal Law Amendment Act, amendment_year 2013.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 328 OUT1 858 |
| ✓ | SimpleCheckFeloniesByNameBFCL/simple_python_157 (criminal_history.check_felonies): full_name John Doe, birth_date 01-01-1980, state California/CA.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 368 OUT1 032 |
| ✓ | SimpleCourtCaseSearchNoFullTextBFCL/simple_python_169 (court_case.search): docket_number 123456, location Texas, full_text false/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 355 OUT1 388 |
| ✓ | SimpleCurrentWeatherTempHumidityBFCL/simple_python_187 (get_current_weather): location Seattle, include_temperature true/omitted, include_humidity true/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 319 OUT1 697 |
| ✓ | SimpleLawsuitCheckClosedStatusBFCL/simple_python_184 (lawsuit.check_case): case_id 1234, closed_status true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 341 OUT1 334 |
| ✓ | SimpleLawsuitsSearchByCompanyLocationBFCL/simple_python_180 (lawsuits_search): company_name Google, location California/CA, year 2020, case_type all/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 336 OUT1 244 |
| ✓ | SimpleMutualFundBalanceBFCL/simple_python_152 (calculate_mutual_fund_balance): investment_amount 50000, annual_yield 0.05, years 3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 366 OUT1 037 |
| ✓ | SimpleSlopeGradientCoordinatesBFCL/simple_python_192 (calculate_slope_gradient): point1 [40.7128,-74.006], point2 [34.0522,-118.2437], unit degree/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 431 OUT1 164 |
| ✓ | SimpleStateLawCaseYearRangeBFCL/simple_python_173 (law_case_search): topic land disputes, year_range [2015,2021], location New York, judicial_system state.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 377 OUT750 |
| ✓ | SimpleTopConstitutionalCasesBFCL/simple_python_174 (get_top_cases): field_of_law constitutional law, top_number 10, country China/CN.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 331 OUT1 079 |
Tool Calling - Simple: Law, Finance & Records (set 5)
8 passed2 failed |
80.0% |
IN79 657 OUT12 198 | |
| ✓ | SimpleCarbonFootprintLifestyleBFCL/simple_python_195 (calculate_carbon_footprint): daily_miles 20, meat_meals_per_week 3, annual_trash_weight 500, flights_per_year omitted/0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 263 OUT835 |
| ✓ | SimpleDailyWaterIntakeWeightBFCL/simple_python_198 (calculate_daily_water_intake): weight 70, activity_level omitted/moderate, climate omitted/temperate.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 189 OUT1 464 |
| ✓ | SimpleFetchLawCaseByNumberBFCL/simple_python_171 (fetch_law_case_details): case_number 43403, court New York, year 2018.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 233 OUT676 |
| ✓ | SimpleFutureValueByPresentValueBFCL/simple_python_154 (calculate_future_value): present_value 5000, annual_interest_rate 0.05, years 10, compounds_per_year omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 242 OUT1 706 |
| ✓ | SimpleHumidityForecastDaysBFCL/simple_python_188 (weather.humidity_forecast): location Miami, days 7, min_humidity omitted/0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 198 OUT1 355 |
| × | SimpleLatestCourtCaseCompaniesBFCL/simple_python_179 (find_latest_court_case): company1 Apple, company2 Samsung, country USA/omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 974 OUT1 196 |
| × | SimpleLawsuitSearchByCountyBFCL/simple_python_183 (lawsuit_search): entity Apple, county Santa Clara County, state California/omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 951 OUT1 523 |
| ✓ | SimpleLegalCaseFetchDetailsBFCL/simple_python_172 (legal_case.fetch): case_id R vs Adams, details true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 198 OUT1 594 |
| ✓ | SimpleParkElevationAreaBFCL/simple_python_190 (park_information): park_name Yellowstone, information [Elevation, Area] (order-independent).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 200 OUT931 |
| ✓ | SimpleTallestMountainsRadiusBFCL/simple_python_191 (locate_tallest_mountains): location Denver, radius 50, amount 5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 209 OUT918 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 1)
1 failed |
0.0% |
IN6 258 OUT1 879 | |
| × | SimpleAnswerStringLogistic1BFCL/live_simple_32-9-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L113: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN6 258 OUT1 879 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 2)
1 failed |
0.0% |
IN6 237 OUT3 202 | |
| × | SimpleAnswerStringLogistic2BFCL/live_simple_33-10-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L114: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN6 237 OUT3 202 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 3)
1 skipped |
- |
IN0 OUT0 | |
| – | SimpleAnswerStringValidBFCL/live_simple_34-11-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L115: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Harness-semantic mismatch (item flaw for our runner): the query ('...Valid?') is directly answerable, and our system prompt permits answering in plain text when no tool is needed, so the model reasonably returns 0 tool calls instead of routing to the answer_string meta-tool. Not a deterministic tool-routing test. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 4)
1 passed |
100.0% |
IN4 038 OUT2 360 | |
| ✓ | SimpleAnswerStringWhatIsBFCL/live_simple_35-12-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L116: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 038 OUT2 360 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 1)
2 passed1 failed |
66.7% |
IN14 737 OUT3 848 | |
| ✓ | SimpleChangeFoodBurgerBFCL/live_simple_20-4-0 (change_food): food_item='burger', modification_request='no onions, extra cheese'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 899 OUT407 |
| × | SimpleThinqAirCleanHighBFCL/live_simple_40-17-0 (ThinQ_Connect): AIR_CLEAN mode, HIGH wind, monitoring on, air clean POWER_ON. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. windStrength broadened (high|strongest|strong) for "strongest wind strength". EXPECTED MODEL MISS: 27B sets airConJobMode=AIR_CLEAN but omits airCleanOperationMode=POWER_ON — the prompt asks to "start the air purifying function", so POWER_ON is the fair GT; kept strict (a real tool-use miss).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.airCleanOperationMode?.ToLower()).AnyOf(["power_on"]) |
Failed | IN4 939 OUT1 607 |
| ✓ | SimpleWeatherTelAvivBFCL/live_simple_4-3-0 (get_current_weather): location='Tel Aviv, Israel', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 899 OUT1 834 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 10)
3 passed |
100.0% |
IN13 965 OUT5 492 | |
| ✓ | SimpleChafodOrderPizzaBFCL/live_simple_23-5-1 (ChaFod): TheFod='PIZZA'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 639 OUT3 666 |
| ✓ | SimpleUberEatMcdonaldsPizzaBFCL/live_simple_29-7-2 (uber.eat.order): restaurant="McDonald's", items=[pizza], quantities=[1].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 660 OUT473 |
| ✓ | SimpleWeatherShanghaiBFCL/live_simple_13-3-9 (get_current_weather): location='Shanghai, China', unit=celsius. Prompt is Chinese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 666 OUT1 353 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 11)
2 passed1 failed |
66.7% |
IN15 099 OUT4 509 | |
| ✓ | SimpleChafodSwitchBurgerBFCL/live_simple_24-5-2 (ChaFod): TheFod='BURGER'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 005 OUT640 |
| ✓ | SimpleFindBeerSierraNevadaBFCL/live_simple_48-21-0 (find_beer): brewery='Sierra Nevada', taste=bitter, aroma=hoppy, color=pale, style=lager; defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 065 OUT2 116 |
| × | SimpleWeatherChennaiBFCL/live_simple_15-3-11 (get_current_weather): location='Chennai, Tamil Nadu', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["chennai, tamil nadu"]) |
Failed | IN5 029 OUT1 753 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 12)
3 passed |
100.0% |
IN15 216 OUT3 432 | |
| ✓ | SimpleChafodGreensSaladBFCL/live_simple_25-5-3 (ChaFod): TheFod='SALAD' (greens mapped to salad).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 030 OUT584 |
| ✓ | SimpleFindBeerCreekPorterBFCL/live_simple_49-21-1 (find_beer): brewery='Creek', taste='slightly bitter', aroma=chocolatey, color=dark, style=porter; defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 123 OUT1 279 |
| ✓ | SimpleWeatherQuintanaRooBFCL/live_simple_19-3-15 (get_current_weather): location='Quintana Roo, Mexico', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 063 OUT1 569 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 13)
1 passed1 failed |
50.0% |
IN12 159 OUT4 749 | |
| ✓ | SimpleLexListExportsV2BFCL/live_simple_30-8-0 (aws.lexv2_models.list_exports): botId='my-bot-id', botVersion='v2', sortBy=ASC or omitted, others omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 838 OUT607 |
| × | SimpleWeatherLangSonBFCL/live_simple_16-3-12 (get_current_weather): location='Lang Son, Vietnam', unit=fahrenheit or omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 321 OUT4 142 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 14)
1 passed1 failed |
50.0% |
IN9 603 OUT2 962 | |
| × | SimpleLexListExportsDescBFCL/live_simple_31-8-1 (aws.lexv2_models.list_exports): botId='B12345', botVersion='v1', sortBy=DESC, others omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.sortBy?.ToLower()).AnyOf(["desc"]) |
Failed | IN4 845 OUT2 043 |
| ✓ | SimpleWeatherBostonBFCL/live_simple_17-3-13 (get_current_weather): location='Boston, MA', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 758 OUT919 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 15)
1 passed1 failed |
50.0% |
IN8 712 OUT3 819 | |
| ✓ | SimpleGetUserInfoBlackBFCL/live_simple_0-0-0 (get_user_info): user_id=7890, special='black'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 365 OUT624 |
| × | SimpleWeatherMoscowBFCL/live_simple_18-3-14 (get_current_weather): location='Moscow, Russia', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["moscow, russia"]) |
Failed | IN4 347 OUT3 195 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 16)
3 passed |
100.0% |
IN14 597 OUT4 329 | |
| ✓ | SimpleFetchWeatherCoordsBFCL/live_simple_39-16-0 (fetch_weather_data): latitude=37.8651, longitude=-119.5383; url/units/language defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 879 OUT1 024 |
| ✓ | SimpleGithubStarGorillaBFCL/live_simple_1-1-0 (github_star): repos='ShishirPatil/gorilla,gorilla-llm/gorilla-cli', aligned=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 881 OUT995 |
| ✓ | SimpleWeatherBangkokBFCL/live_simple_14-3-10 (get_current_weather): location='Bangkok, Thailand', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 837 OUT2 310 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 17)
2 passed1 failed |
66.7% |
IN23 443 OUT8 311 | |
| ✓ | SimpleCarbonIntensityGBBFCL/live_simple_50-22-0 (get_latest_carbon_intensity): zone='Great Britain' or 'GB'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 590 OUT1 053 |
| ✓ | SimpleMultiplyThreeTwoBFCL/live_simple_47-20-0 (multiply): a=3, b=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 592 OUT280 |
| × | SimpleWeatherOpenMeteoMariposaBFCL/live_simple_38-15-0 (get_current_weather, open-meteo variant): location='Mariposa, CA', unit=fahrenheit; url default omitted.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 4 |
Failed | IN14 261 OUT6 978 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 2)
3 passed |
100.0% |
IN14 695 OUT4 009 | |
| ✓ | SimpleChangeFoodChickenSpicyBFCL/live_simple_21-4-1 (change_food): food_item='chicken dish', modification_request='extra spicy'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 920 OUT460 |
| ✓ | SimpleThinqAirDryMidBFCL/live_simple_41-17-1 (ThinQ_Connect): AIR_DRY mode, MID wind (or omitted), all else optional. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. windStrength broadened (mid|medium) for "medium wind strength"; air-clean off-state accepts off/power_off.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 889 OUT458 |
| ✓ | SimpleWeatherDivinopolisBFCL/live_simple_5-3-1 (get_current_weather): location='Divinopolis, MG' (verbatim), unit=fahrenheit or omitted. Prompt is Portuguese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 886 OUT3 091 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 3)
3 passed |
100.0% |
IN14 706 OUT3 835 | |
| ✓ | SimpleParseAnswerNoDefineBFCL/live_simple_36-13-0 (parseAnswer): answer='The text does not define logistic regression, therefore I cannot answer this question.'1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 903 OUT1 538 |
| ✓ | SimpleThinqKoreanTarget22BFCL/live_simple_42-17-2 (ThinQ_Connect): coolTargetTemperature=22 required; mode COOL and wind MID optional. Prompt is Korean, copied verbatim. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. Prompt is dehumidify (제습) so GT's optional airConJobMode=COOL is a source oddity — accept cool/dehumidify/air_dry (optional). windStrength MID broadened (mid|medium). The "22도" target-temp is accepted in either coolTargetTemperature or targetTemperature (the two are genuinely ambiguous for "target temperature").1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 920 OUT494 |
| ✓ | SimpleWeatherSanFranciscoBFCL/live_simple_6-3-2 (get_current_weather): location='San Francisco, CA', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 883 OUT1 803 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 4)
3 passed |
100.0% |
IN14 589 OUT5 196 | |
| ✓ | SimpleParseAnswerNoDefineValidityBFCL/live_simple_37-14-0 (parseAnswer): answer='The text does not define logistic regression, therefore I cannot answer this question.'1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 885 OUT1 374 |
| ✓ | SimpleThinqCoolHighWindBFCL/live_simple_43-17-3 (ThinQ_Connect): HIGH wind required; COOL mode and coolTargetTemperature=24 optional. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. air-clean off-state broadened (power_off|off) — the model files the unrequested clean-mode as "off".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 866 OUT1 463 |
| ✓ | SimpleWeatherRigaBFCL/live_simple_7-3-3 (get_current_weather): location='Riga, Latvia'; unit fahrenheit/celsius/omitted all accepted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 838 OUT2 359 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 5)
3 passed |
100.0% |
IN15 192 OUT3 523 | |
| ✓ | SimpleThinqTurnedOnCoolBFCL/live_simple_44-18-0 (ThinQ_Connect): airConOperationMode=POWER_ON required; COOL currentJobMode optional. NOTE: tool flattened to nullable top-level params (see struct comment) so a valid partial call is one tool call, not a forced retry. windStrength/airClean off-state broadened to synonyms (mid|medium, stop|off|power_off).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 041 OUT553 |
| ✓ | SimpleUberComfortAddisonBFCL/live_simple_2-2-0 (uber.ride): loc='2020 Addison Street, Berkeley, CA, USA', type=comfort, time=600.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 103 OUT1 291 |
| ✓ | SimpleWeatherLondonBFCL/live_simple_8-3-4 (get_current_weather): location='London, UK' or 'London, England'; unit fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 048 OUT1 679 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 6)
1 passed2 failed |
33.3% |
IN15 225 OUT3 053 | |
| × | SimpleThinqCoolAirCleanStartBFCL/live_simple_45-18-1 (ThinQ_Connect): airCleanOperationMode=START, airConOperationMode=POWER_ON, coolTargetTemperature=20. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. airCleanOperationMode "air cleaning on" broadened (start|on|power_on). airConOperationMode=POWER_ON and coolTargetTemperature=20 kept strict — both fair GT. EXPECTED MODEL MISS: 27B gets job-mode + air-clean START + temp, but omits airConOperationMode=POWER_ON.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.airConOperationMode?.ToLower()).AnyOf(["power_on"]) |
Failed | IN5 093 OUT1 132 |
| ✓ | SimpleUberPlusBakerBFCL/live_simple_3-2-1 (uber.ride): loc='221B Baker Street, Berkeley, CA, USA', type=plus, time=600.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 105 OUT642 |
| × | SimpleWeatherHyderabadBFCL/live_simple_9-3-5 (get_current_weather): location='Hyderabad, India', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["hyderabad, india"]) |
Failed | IN5 027 OUT1 279 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 7)
3 passed |
100.0% |
IN15 244 OUT4 115 | |
| ✓ | SimpleThinqFlatCoolCleanBFCL/live_simple_46-19-0 (ThinQ_Connect, flat params): COOL, MID, monitoring on, clean START, POWER_ON, power-save off, target 24C.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 144 OUT2 080 |
| ✓ | SimpleUberPlusVietnameseBFCL/live_simple_26-6-0 (uber.ride): loc Vietnamese/English form, type=plus, time=10 (minutes). Prompt is Vietnamese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 067 OUT447 |
| ✓ | SimpleWeatherYosemiteMariposaBFCL/live_simple_10-3-6 (get_current_weather): location='Mariposa, CA', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 033 OUT1 588 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 8)
1 passed2 failed |
33.3% |
IN15 065 OUT5 826 | |
| × | SimpleThinqCoolScheduledBFCL/live_simple_51-23-0 (ThinQ_Connect): COOL, HIGH wind, POWER_ON, power-save on, target 22C, start in 1h30m. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. airConOperationMode=POWER_ON kept strict ("Start in 1h30m" schedules a power-on). EXPECTED MODEL MISS: 27B gets job-mode COOL + wind + power-save + temp + schedule, but omits airConOperationMode=POWER_ON.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.airConOperationMode?.ToLower()).AnyOf(["power_on"]) |
Failed | IN5 096 OUT979 |
| × | SimpleUberEatPitadaBFCL/live_simple_27-7-0 (uber.eat.order): restaurant='uber pitada', items=[burgers, chicken wings], quantities=[5, 6].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.restaurant?.ToLower()).AnyOf(["uber pitada"]) |
Failed | IN5 004 OUT1 264 |
| ✓ | SimpleWeatherNaplesBFCL/live_simple_11-3-7 (get_current_weather): location='Naples, FL' or 'Naples, Florida'; unit fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 965 OUT3 583 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 9)
2 passed1 failed |
66.7% |
IN14 068 OUT3 276 | |
| × | SimpleChafodWhopperBurgerBFCL/live_simple_22-5-0 (ChaFod): TheFod='BURGER' (Whopper mapped to burger).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: call.ToolName == expectedTool
Values:
call.ToolName = "uber_eat_order"
expectedTool = "ChaFod" |
Failed | IN4 657 OUT1 100 |
| ✓ | SimpleUberEatKfcChineseBFCL/live_simple_28-7-1 (uber.eat.order): KFC order, 4 items, quantities [10,50,30,90]. Prompt is Chinese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 713 OUT458 |
| ✓ | SimpleWeatherNewYorkBFCL/live_simple_12-3-8 (get_current_weather): location='New York, NY', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 698 OUT1 718 |
Tool Calling - Simple: Live Media, Weather & Data (set 2)
9 passed1 failed |
90.0% |
IN76 082 OUT9 820 | |
| ✓ | SimpleCreditQuoteAutoCreditoBFCL/live_simple_67-31-0 (obtener_cotizacion_de_creditos): monto=1000000, plazo=12, producto='auto', año=2024, enganche=0.2.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 698 OUT3 502 |
| ✓ | SimpleCurrentLocDmsBFCL/live_simple_87-48-0 (get_current_loc): coordinate_format='DMS', include_altitude=true, timeout default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 583 OUT1 156 |
| ✓ | SimpleGetMoviesMumbaiTodayBFCL/live_simple_59-28-0 (get_movies): city='Mumbai', optionals default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 551 OUT700 |
| ✓ | SimplePlayArtistKaarijaTracksBFCL/live_simple_93-54-0 (play_artist): artist_name='Käärijä'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 566 OUT233 |
| ✓ | SimpleRetrieveAnalysisSuperfancyBFCL/live_simple_69-33-0 (analysis_api.AnalysisApi.retrieve_analysis): project/component/vulnerability UUIDs.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 681 OUT488 |
| ✓ | SimpleSensorAlertsPerPage10BFCL/live_simple_70-34-0 (get_sensor_alerts): perPage=10, sensorSerial='Q3CC-CRT3-SZ2G', rest omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 629 OUT1 453 |
| ✓ | SimpleSitefinityNewsItemJulian12BFCL/live_simple_81-42-0 (sitefinity_create_contentitem): ContentItem='NewsItem', Title='Julian is testing12', rest omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 632 OUT328 |
| ✓ | SimpleTodoAddShopping9pmB25BFCL/live_simple_55-25-1 (todo): type='add', content='go for shopping at 9 pm'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 575 OUT305 |
| × | SimpleTodoDeleteTodoRandomBFCL/live_simple_54-25-0 (todo): type='delete', content='todo random'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.content?.ToLower()).AnyOf(["todo random"]) |
Failed | IN7 577 OUT1 086 |
| ✓ | SimpleWeatherGetLondonUkBFCL/live_simple_77-38-0 (weather.get): city='London', country='GB' (from UK), units/forecast default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 590 OUT569 |
Tool Calling - Simple: Live Media, Weather & Data (set 3)
9 passed1 failed |
90.0% |
IN71 616 OUT7 996 | |
| × | SimpleExtractMillennialsAppleBFCL/live_simple_71-35-0 (extract_parameters_v1): demographics=[millennials], targets=[brand:Apple], metrics=[view], min_date=2022-07-01. NOTE: targets @param clarified to '<attribute>:<entity>' order (was ambiguous 'target:target_type', which made the model emit the reversed 'Apple:brand'); GT 'brand:Apple' kept strict.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.metrics.Count == 1
Values:
args.metrics.Count = 2 |
Failed | IN7 179 OUT2 070 |
| ✓ | SimpleFetchPersonId4BFCL/live_simple_91-52-0 (fetchPerson): id=4.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 137 OUT707 |
| ✓ | SimpleLogFoodChaiTeaBFCL/live_simple_88-49-0 (log_food): food_name='chai tea', portion_amount=16.0, portion_unit='ounces', meal_name='snack'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 205 OUT652 |
| ✓ | SimplePlaySongChaChaChaBFCL/live_simple_94-55-0 (play_song): artist_name='Käärijä', song_name='Cha Cha Cha'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 155 OUT261 |
| ✓ | SimpleSendEmailAndyForecastBFCL/live_simple_78-39-0 (send_email): to='andy@gorilla.ai', subject='Sales Forecast Request', body set, cc/bcc/attachments omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 203 OUT924 |
| ✓ | SimpleSitefinityNewsJulianTestingBFCL/live_simple_82-43-0 (sitefinity_create_contentitem): ContentItem='News', Title='Julian is testing', rest omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 171 OUT385 |
| ✓ | SimpleTemperatureParisFranceBFCL/live_simple_90-51-0 (temperature): location='Paris, France', units/time default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 129 OUT954 |
| ✓ | SimpleTodoManagerAddSleepBFCL/live_simple_57-26-1 (todo_manager.handle_action): type='add', content='go to sleep at 9 pm'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 142 OUT334 |
| ✓ | SimpleTodoManagerDeleteGymBFCL/live_simple_56-26-0 (todo_manager.handle_action): type='delete', content='go to gym'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 132 OUT286 |
| ✓ | SimpleTranslateEnglishFrenchBFCL/live_simple_76-37-0 (language_translator.translate): source='en', target='fr', text='What is your name?'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 163 OUT1 423 |
Tool Calling - Simple: Live Media, Weather & Data (set 4)
10 passed |
100.0% |
IN53 296 OUT6 653 | |
| ✓ | SimpleOpenMeteoBerkeleyBFCL/live_simple_85-46-0 (open_meteo_api.fetch_weather_data): coordinates=[37.8715,-122.273], units default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 344 OUT2 119 |
| ✓ | SimpleSearchHotelsSanDiegoBFCL/live_simple_79-40-0 (search_hotels): location='San Diego, CA'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 384 OUT454 |
| ✓ | SimpleSitefinityNewsSleepProductivityBFCL/live_simple_83-44-0 (sitefinity_create_contentitem): full news article with all metadata.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 478 OUT776 |
| ✓ | SimpleSortArrayAscendingBFCL/live_simple_86-47-0 (sort_array): array=[1,54,3,1,76,2343,21], reverse default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 354 OUT735 |
| ✓ | SimpleTodoAddGoToGoaBFCL/live_simple_64-29-4 (todo): type='add', content='go to Goa'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 281 OUT658 |
| ✓ | SimpleTodoAddGymTomorrowBFCL/live_simple_61-29-1 (todo): type='add', content='go to gym tomorrow.'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 280 OUT405 |
| ✓ | SimpleTodoAddHiCharlieBFCL/live_simple_65-29-5 (todo): type='add', content='Hi charlie'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 291 OUT462 |
| ✓ | SimpleTodoAddSleep9pmBFCL/live_simple_63-29-3 (todo): type='add', content='go to sleep at 9 pm'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 287 OUT277 |
| ✓ | SimpleTodoCompleteFinancialReportBFCL/live_simple_60-29-0 (todo): type='complete', content='Submit monthly financial report'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 304 OUT517 |
| ✓ | SimpleTodoDeleteRaviBFCL/live_simple_62-29-2 (todo): type='delete', content='ravi'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 293 OUT250 |
Tool Calling - Simple: Live Movies, Music & Booking (set 1)
3 passed |
100.0% |
IN14 597 OUT2 142 | |
| ✓ | SimpleMoviesTarantinoWhitakerBFCL/live_simple_208-117-0 (Movies_3_FindMovies): directed_by='Quentin Tarantino', cast='Duane Whitaker', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 886 OUT1 100 |
| ✓ | SimplePlaySongRoarBFCL/live_simple_230-121-0 (play_song): song_id='wjeiruhuq_roar', artist='Katy Perry', quality='high', device_id omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 883 OUT822 |
| ✓ | SimpleSetVolume20BFCL/live_simple_242-126-0 (set_volume): volume=20.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 828 OUT220 |
Tool Calling - Simple: Live Movies, Music & Booking (set 10)
3 passed |
100.0% |
IN15 629 OUT3 562 | |
| ✓ | SimpleHandoverPackageArrivalBFCL/live_simple_257-137-1 (answer_question): statement=package-arrival-time question, urgency/language default. System instructs HANDOVER on question. NOTE: no collider loop observed (single clean call, sibling tests pass); statement is free text -> substring-intent of the key phrase (model drops "Can you tell me").1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 217 OUT601 |
| ✓ | SimpleMoviesNiteshDirectorBFCL/live_simple_217-117-9 (Movies_3_FindMovies): directed_by='Nitesh Tiwari', genre/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 186 OUT2 054 |
| ✓ | SimpleTtsChineseMaleBFCL/live_simple_227-118-1 (text_to_speech.convert): text=Chinese phrase, language='zh-CN', gender='male', format='wav', speed default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 226 OUT907 |
Tool Calling - Simple: Live Movies, Music & Booking (set 11)
3 passed |
100.0% |
IN14 947 OUT3 468 | |
| ✓ | SimpleMoviesColaizzoWatkinsBFCL/live_simple_218-117-10 (Movies_3_FindMovies): directed_by='Paul Downs Colaizzo', cast='Michaela Watkins', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 036 OUT1 333 |
| ✓ | SimpleRequestsGetBuProductsBFCL/live_simple_229-120-0 (requests.get): anchor='user'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 954 OUT991 |
| ✓ | SimpleTtsPrettyGirlDefaultBFCL/live_simple_228-119-0 (text_to_speech.convert): text='I am a pretty girl', language/gender/speed default.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 957 OUT1 144 |
Tool Calling - Simple: Live Movies, Music & Booking (set 12)
3 passed |
100.0% |
IN14 662 OUT4 138 | |
| ✓ | SimpleBookFlightParisBFCL/live_simple_233-123-0 (book_flight): departure_location='Paris, France', departure_time='12/03/2023 15:00', return_time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 923 OUT448 |
| ✓ | SimpleGetVersionBFCL/live_simple_247-129-0 (version_api.VersionApi.get_version): no arguments.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 835 OUT699 |
| ✓ | SimpleMoviesSimonCurtisDramaBFCL/live_simple_219-117-11 (Movies_3_FindMovies): directed_by='Simon Curtis', genre='Drama', cast='McKinley Belcher III'.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 904 OUT2 991 |
Tool Calling - Simple: Live Movies, Music & Booking (set 13)
3 passed |
100.0% |
IN15 208 OUT2 991 | |
| ✓ | SimpleAclRetrieveProjectsBFCL/live_simple_248-130-0 (acl_api.AclApi.retrieve_projects): uuid='12', excludeInactive=true, onlyRoot=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 054 OUT735 |
| ✓ | SimpleBookFlightNewYorkToParisBFCL/live_simple_234-123-1 (book_flight): departure_location='New York, NY', departure_time='20/06/2022 17:00', return_time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 104 OUT488 |
| ✓ | SimpleMoviesMackendrickThrillerBFCL/live_simple_220-117-12 (Movies_3_FindMovies): directed_by='Alexander Mackendrick', genre='Thriller', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 050 OUT1 768 |
Tool Calling - Simple: Live Movies, Music & Booking (set 14)
3 passed |
100.0% |
IN14 821 OUT2 082 | |
| ✓ | SimpleBookFlightJfkBFCL/live_simple_245-127-0 (book_flight): departure_location='JFK', departure_date='16/03/2024', departure_time='14:00'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 995 OUT395 |
| ✓ | SimpleChatWithBunnyBFCL/live_simple_249-131-0 (chat_with_friend): user_message='Hey bunny, How are you doing'. NOTE: no collider loop observed (single clean call, sibling tests pass); user_message is a free-text passthrough -> substring-intent (model may drop the "Hey bunny," greeting).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 910 OUT648 |
| ✓ | SimpleMoviesJamesGrayBFCL/live_simple_221-117-13 (Movies_3_FindMovies): directed_by='James Gray', genre/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 916 OUT1 039 |
Tool Calling - Simple: Live Movies, Music & Booking (set 15)
2 passed |
100.0% |
IN9 291 OUT2 180 | |
| ✓ | SimpleBookFlightNewYorkMarchBFCL/live_simple_246-128-0 (book_flight): departure_location='New York, NY', departure_time='15/03/2024'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 642 OUT400 |
| ✓ | SimpleMoviesVanOrmanBrownBFCL/live_simple_222-117-14 (Movies_3_FindMovies): directed_by='Thurop Van Orman', cast='Sterling K. Brown', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 649 OUT1 780 |
Tool Calling - Simple: Live Movies, Music & Booking (set 16)
1 passed1 skipped |
100.0% |
IN4 598 OUT3 913 | |
| ✓ | SimpleMoviesImaginativeFantasyBFCL/live_simple_223-117-15 (Movies_3_FindMovies): genre='Fantasy' ('Imaginative fiction'), director/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 598 OUT3 913 |
| – | SimpleRescheduleGymSessionBFCL/live_simple_231-122-0 (reschedule_event): event_identifier='456123', new_datetime='2022-10-30T16:30:00Z'.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: source flaw: relative-date arithmetic is inconsistent — prompt states 'Monday 2022-10-20' (that date is actually a Thursday) and the GT 'next Thursday' resolves to 2022-10-30 (a Sunday), so the exact new_datetime cannot be derived deterministically. |
Skipped | IN0 OUT0 |
Tool Calling - Simple: Live Movies, Music & Booking (set 17)
1 passed1 failed |
50.0% |
IN11 812 OUT2 732 | |
| ✓ | SimpleMoviesTimBurtonDeppBFCL/live_simple_224-117-16 (Movies_3_FindMovies): directed_by='Tim Burton', genre='Offbeat', cast='Johnny Depp'.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 589 OUT1 344 |
| × | SimpleRescheduleMedAppointmentBFCL/live_simple_232-122-1 (reschedule_event): event_identifier='med123', new_datetime='2021-04-15T13:45:00Z'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 223 OUT1 388 |
Tool Calling - Simple: Live Movies, Music & Booking (set 18)
2 passed |
100.0% |
IN8 954 OUT911 | |
| ✓ | SimpleMoviesKirillComedyDramaBFCL/live_simple_225-117-17 (Movies_3_FindMovies): directed_by='Kirill Mikhanovsky', genre='Comedy-drama', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 512 OUT697 |
| ✓ | SimpleSetVolume70BFCL/live_simple_244-126-2 (set_volume): volume=70.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 442 OUT214 |
Tool Calling - Simple: Live Movies, Music & Booking (set 2)
3 passed |
100.0% |
IN14 642 OUT1 905 | |
| ✓ | SimpleMoviesLeitchTuisanoBFCL/live_simple_209-117-1 (Movies_3_FindMovies): directed_by='David Leitch', cast='Lori Pelenise Tuisano', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 911 OUT1 032 |
| ✓ | SimpleSetVolume30BFCL/live_simple_243-126-1 (set_volume): volume=30.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 864 OUT512 |
| ✓ | SimpleSpotifyFriendsMarshmelloBFCL/live_simple_235-124-0 (play_spotify_song): query='track:Friends artist:Marshmello', shuffle/volume default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 867 OUT361 |
Tool Calling - Simple: Live Movies, Music & Booking (set 3)
3 passed |
100.0% |
IN15 207 OUT3 046 | |
| ✓ | SimpleFlightStatusIndigo6E123BFCL/live_simple_251-133-0 (flight.status.check): flight_id='6E123', passenger_name='Karan', ticket_number='IND4567'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 102 OUT786 |
| ✓ | SimpleMoviesDramaBFCL/live_simple_210-117-2 (Movies_3_FindMovies): genre='Drama', director/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 052 OUT1 931 |
| ✓ | SimpleSpotifyDilNu124BFCL/live_simple_236-124-1 (play_spotify_song): query='track:Dil Nu artist:Maninder Buttar', shuffle/volume default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 053 OUT329 |
Tool Calling - Simple: Live Movies, Music & Booking (set 4)
3 passed |
100.0% |
IN15 203 OUT3 592 | |
| ✓ | SimpleFlightStatusIndigoAirlineBFCL/live_simple_252-134-0 (flight.status.check): airlines='Indigo', person_name='Karan', ticket_number='IND4567'. NOTE: tool doc says "if only one name mentioned ... use LNU", so "Karan LNU" is equivalent-correct per the documented convention -> accept both.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 096 OUT794 |
| ✓ | SimpleMoviesComedyCordenBFCL/live_simple_211-117-3 (Movies_3_FindMovies): genre='Comedy' (prompt 'Comic'), cast='James Corden', director dontcare. NOTE: args are correct; single-call contract stays strict. The 27B occasionally emits a 2nd call (over-call) -> legit intermittent fail (kept as signal).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 072 OUT2 100 |
| ✓ | SimpleSpotifyWreckingBallBFCL/live_simple_237-125-0 (play_spotify_song): query='track:Wrecking Ball artist:Miley Cyrus', device_id/play default. NOTE: GT accepts only the structured 'track:.. artist:..' form (kept strict). The 27B occasionally drops the format (e.g. "wrecking ball Miley Cyrus") -> legit intermittent miss (kept as signal).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 035 OUT698 |
Tool Calling - Simple: Live Movies, Music & Booking (set 5)
3 passed |
100.0% |
IN15 131 OUT1 203 | |
| ✓ | SimpleMoviesComedyEdgarWrightBFCL/live_simple_212-117-4 (Movies_3_FindMovies): directed_by='Edgar Wright', genre='Comedy', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 028 OUT473 |
| ✓ | SimpleSpotifyDilNu125BFCL/live_simple_238-125-1 (play_spotify_song): query='track:Dil Nu artist:Maninder Buttar', device_id/play default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 037 OUT375 |
| ✓ | SimpleUberEatsBigMacBFCL/live_simple_250-132-0 (uber.eat.order): restaurants='McDonald's', items=[Big Mac, McFlurry], quantities=[2,1].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 066 OUT355 |
Tool Calling - Simple: Live Movies, Music & Booking (set 6)
3 passed |
100.0% |
IN15 151 OUT2 104 | |
| ✓ | SimpleMoviesOffbeatTimBurtonBFCL/live_simple_213-117-5 (Movies_3_FindMovies): directed_by='Tim Burton', genre='Offbeat', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 047 OUT1 126 |
| ✓ | SimpleSpotifyShapeOfYouBFCL/live_simple_239-125-2 (play_spotify_song): query='track:Shape of You artist:Ed Sheeran' or 'Shape of You'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 036 OUT650 |
| ✓ | SimpleUberEatsBurgersWingsBFCL/live_simple_253-135-0 (uber.eat.order): restaurants='McDonald's', items=[burgers, chicken wings], quantities=[5,6].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 068 OUT328 |
Tool Calling - Simple: Live Movies, Music & Booking (set 7)
2 passed1 failed |
66.7% |
IN14 968 OUT3 390 | |
| ✓ | SimpleMoviesComicNiteshTiwariBFCL/live_simple_214-117-6 (Movies_3_FindMovies): directed_by='Nitesh Tiwari', genre='Comedy' or 'Comedy-drama', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 039 OUT1 273 |
| × | SimpleSpotifyBabySharkBFCL/live_simple_240-125-3 (play_spotify_song): query='Baby Shark' (artist unknown).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: (args.query?.ToLower()).AnyOf(["baby shark"]) |
Failed | IN4 957 OUT1 261 |
| ✓ | SimpleUserMandatesPendingBFCL/live_simple_254-136-0 (user.mandates): user_id='U123', status='pending'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 972 OUT856 |
Tool Calling - Simple: Live Movies, Music & Booking (set 8)
2 passed1 failed |
66.7% |
IN14 900 OUT2 641 | |
| ✓ | SimpleMoviesFantasyTonightBFCL/live_simple_215-117-7 (Movies_3_FindMovies): genre='Fantasy', director/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 968 OUT947 |
| × | SimpleSpotifyJohnnyPapaBFCL/live_simple_241-125-4 (play_spotify_song): query='Johnny Johnny Yes Papa' (artist unknown).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: (args.query?.ToLower()).AnyOf(["johnny johnny yes papa"]) |
Failed | IN4 947 OUT425 |
| ✓ | SimpleUserMandatesActiveBFCL/live_simple_255-136-1 (user.mandates): user_id='U123456', status='active' or omitted (default).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 985 OUT1 269 |
Tool Calling - Simple: Live Movies, Music & Booking (set 9)
2 passed1 failed |
66.7% |
IN15 566 OUT5 675 | |
| × | SimpleHandoverComplaintStatusBFCL/live_simple_256-137-0 (answer_question): statement=user complaint-status question, urgency/language default. System instructs HANDOVER on question.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.statement?.ToLower()).AnyOf(["i have had submit a compaint and i like to check its status"]) |
Failed | IN5 207 OUT1 086 |
| ✓ | SimpleMoviesActionLeitchKingBFCL/live_simple_216-117-8 (Movies_3_FindMovies): directed_by='David Leitch', genre='Action', cast='Alex King'.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 176 OUT3 024 |
| ✓ | SimpleTtsPrettyGirlSpanishBFCL/live_simple_226-118-0 (text_to_speech.convert): text='I am a pretty girl' or Spanish translation, language='es-ES', gender/format/speed default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 183 OUT1 565 |
Tool Calling - Simple: Live Services, Weather & Commands (set 1)
4 passed |
100.0% |
IN18 934 OUT4 605 | |
| ✓ | SimpleExtractChesterJaneBFCL/live_simple_189-114-0 (extractor.extract_information): [Chester,42],[Jane,43] (Jane a year older).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 780 OUT891 |
| ✓ | SimpleServiceIdIroningUnitIntBFCL/live_simple_174-100-0 (get_service_id): ironing service -> service_id=2, unit omitted/1. Thai prompt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 727 OUT1 083 |
| ✓ | SimpleWeatherNewYorkBFCL/live_simple_193-116-1 (Weather_1_GetWeather): city=New York, date=2023-03-08.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 713 OUT1 520 |
| ✓ | SimpleWeatherPacificaBFCL/live_simple_192-116-0 (Weather_1_GetWeather): city=Pacifica, date=2023-04-11.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 714 OUT1 111 |
Tool Calling - Simple: Live Services, Weather & Commands (set 10)
9 passed |
100.0% |
IN41 470 OUT8 095 | |
| ✓ | SimpleCmdDockerPsBFCL/live_simple_172-99-6 (cmd_controller.execute): is Docker running -> 'docker ps'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 573 OUT891 |
| ✓ | SimpleCmdDockerStartBFCL/live_simple_168-99-2 (cmd_controller.execute): start Docker -> 'docker start'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 573 OUT308 |
| ✓ | SimpleCmdEchoDateBFCL/live_simple_171-99-5 (cmd_controller.execute): today's date -> 'echo %date%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 582 OUT749 |
| ✓ | SimpleCmdEchoTimeBFCL/live_simple_166-99-0 (cmd_controller.execute): current time -> 'echo %time%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 594 OUT859 |
| ✓ | SimpleCmdListCDriveBFCL/live_simple_170-99-4 (cmd_controller.execute): list C drive -> 'dir C:\'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 563 OUT954 |
| ✓ | SimpleCmdRunPythonScriptBFCL/live_simple_169-99-3 (cmd_controller.execute): run python script with venv interpreter.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 656 OUT1 073 |
| ✓ | SimpleCmdStartCalcBFCL/live_simple_173-99-7 (cmd_controller.execute): open calculator -> 'start calc'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 575 OUT312 |
| ✓ | SimpleCmdTasklistCodeBFCL/live_simple_167-99-1 (cmd_controller.execute): is VS Code running -> tasklist filter.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 606 OUT931 |
| ✓ | SimpleProvidersChonburiDateRangeBFCL/live_simple_186-111-0 (get_service_providers): ironing, Chonburi, rating>=4.5, 12/03/2024 16:00-18:00.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 748 OUT2 018 |
Tool Calling - Simple: Live Services, Weather & Commands (set 11)
2 passed2 failed |
50.0% |
IN19 671 OUT8 350 | |
| ✓ | SimpleItemsCircleLogosBFCL/live_simple_162-96-1 (get_items): 4 circle-logo copyright items.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 826 OUT2 889 |
| ✓ | SimpleItemsLegoMovieBFCL/live_simple_163-96-2 (get_items): 7 Lego-movie copyright items.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 921 OUT1 627 |
| × | SimpleItemsSuperheroLogosBFCL/live_simple_161-96-0 (get_items): 12 superhero-logo copyright items.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.items.Count == 12
Values:
args.items.Count = 1 |
Failed | IN5 055 OUT2 192 |
| × | SimpleProvidersChonburiNoLateBFCL/live_simple_187-112-0 (get_service_providers): ironing, Chonburi, rating>=4.5, 12/03/2024, never late. NOTE: the prompt gives a time range "16:00 - 18:00", so end_available_date is expected; the 27B occasionally omits it (~1/3 of runs) -> a legit model-reliability miss, not an assertion flaw.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.end_available_date ?? "").AnyOfContains(["2024-03-12", "2024-12-03"]) |
Failed | IN4 869 OUT1 642 |
Tool Calling - Simple: Live Services, Weather & Commands (set 12)
2 passed |
100.0% |
IN9 434 OUT1 298 | |
| ✓ | SimpleCopyrightCocaColaBFCL/live_simple_164-97-0 (get_copyright_info): Coca-Cola logo, The Coca-Cola Company, confidence 100.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 863 OUT546 |
| ✓ | SimpleProfessionalCleaningRating2BFCL/live_simple_183-108-0 (getDataForProfessional): cleaning in Bangkok, rating>=2.0 -> avg_rating=2.0, province_id=1, service_id=1.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 571 OUT752 |
Tool Calling - Simple: Live Services, Weather & Commands (set 13)
3 passed |
100.0% |
IN13 690 OUT3 951 | |
| ✓ | SimpleMpnSpecs3pak7BFCL/live_simple_191-115-1 (raptor.mpn.specs): identifier=3pak7, MPN, include_images omitted/false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 559 OUT1 732 |
| ✓ | SimpleMpnSpecsAchdImagesBFCL/live_simple_190-115-0 (raptor.mpn.specs): identifier=ACHD, MPN, include_images=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 569 OUT1 326 |
| ✓ | SimpleProfessionalCleaningRating4BFCL/live_simple_184-109-0 (getDataForProfessional): cleaning in Bangkok, rating>4 -> service_id=1, province_id=1, rating=4.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 562 OUT893 |
Tool Calling - Simple: Live Services, Weather & Commands (set 14)
2 passed |
100.0% |
IN9 771 OUT2 378 | |
| ✓ | SimpleExtractChineseNamesBFCL/live_simple_165-98-0 (extractor.extract_information): Chinese text -> [李雷,18],[李丽,21] (sister 3 yrs older).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 892 OUT1 591 |
| ✓ | SimpleProfessionalIroningChiangMaiBFCL/live_simple_188-113-0 (getDataForProfessional): ironing in Chiang Mai, Sankampang/Ton Pao, rating 4.5.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 879 OUT787 |
Tool Calling - Simple: Live Services, Weather & Commands (set 2)
3 passed |
100.0% |
IN13 110 OUT2 920 | |
| ✓ | SimpleServiceIdIroningUnitSessionBFCL/live_simple_175-101-0 (get_service_id): ironing -> service_id=2, unit omitted/session. Thai prompt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 353 OUT989 |
| ✓ | SimpleWeatherMartinezBFCL/live_simple_194-116-2 (Weather_1_GetWeather): city=Martinez, date=2023-04-25.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 387 OUT677 |
| ✓ | SimpleWeatherPaloAltoBFCL/live_simple_195-116-3 (Weather_1_GetWeather): city=Palo Alto, date=2023-04-25.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 370 OUT1 254 |
Tool Calling - Simple: Live Services, Weather & Commands (set 3)
3 passed |
100.0% |
IN13 072 OUT3 776 | |
| ✓ | SimpleServiceIdIroningUnitQtyBFCL/live_simple_176-102-0 (get_service_id): ironing -> service_id=2, unit omitted/1. Thai prompt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 339 OUT717 |
| ✓ | SimpleWeatherAlamedaWednesdayBFCL/live_simple_196-116-4 (Weather_1_GetWeather): city=Alameda, upcoming Wednesday (today Mon 2023-04-25) -> 2023-04-27.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 369 OUT1 628 |
| ✓ | SimpleWeatherStinsonBeachBFCL/live_simple_197-116-5 (Weather_1_GetWeather): city=Stinson Beach, date=2023-04-05.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 364 OUT1 431 |
Tool Calling - Simple: Live Services, Weather & Commands (set 4)
4 passed |
100.0% |
IN17 401 OUT4 560 | |
| ✓ | SimpleServiceIdCleaningBFCL/live_simple_177-103-0 (get_service_id): cleaning -> service_id=1, unit omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 325 OUT951 |
| ✓ | SimpleServiceIdIroningBFCL/live_simple_178-103-1 (get_service_id): ironing -> service_id=2, unit omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 327 OUT435 |
| ✓ | SimpleWeatherHealdsburgBFCL/live_simple_198-116-6 (Weather_1_GetWeather): city=Healdsburg, date=2023-03-02.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 380 OUT1 339 |
| ✓ | SimpleWeatherMarshallBFCL/live_simple_199-116-7 (Weather_1_GetWeather): city=Marshall (prompt 'Marshall, MN'), date=2023-03-05. NOTE: GT city is bare "Marshall"; model faithfully copies the "Marshall, MN" disambiguator from the prompt -> substring-intent (equivalent-correct).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 369 OUT1 835 |
Tool Calling - Simple: Live Services, Weather & Commands (set 5)
3 passed |
100.0% |
IN13 209 OUT4 399 | |
| ✓ | SimpleServiceIdIroningChiangMaiBFCL/live_simple_179-104-0 (get_service_id): ironing in Chiang Mai -> service_id=2, province_id=2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 374 OUT369 |
| ✓ | SimpleWeatherCampbellBFCL/live_simple_201-116-9 (Weather_1_GetWeather): city=Campbell, date=2023-03-04.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 431 OUT2 156 |
| ✓ | SimpleWeatherFremontBFCL/live_simple_200-116-8 (Weather_1_GetWeather): city=Fremont, date=2023-03-01.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 404 OUT1 874 |
Tool Calling - Simple: Live Services, Weather & Commands (set 6)
3 passed |
100.0% |
IN13 396 OUT3 874 | |
| ✓ | SimpleServiceIdIroningBangkokRatingBFCL/live_simple_180-105-0 (get_service_id): ironing in Bangkok, rating>=4 -> service_id=2, province_id=1, rating=4.0.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 468 OUT892 |
| ✓ | SimpleWeatherFosterCityBFCL/live_simple_202-116-10 (Weather_1_GetWeather): city=Foster City, date=2023-04-25.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 461 OUT1 776 |
| ✓ | SimpleWeatherWashingtonBFCL/live_simple_203-116-11 (Weather_1_GetWeather): city=Washington (prompt 'Washington, DC'), date=2023-03-01. NOTE: GT city is bare "Washington"; model faithfully copies the "Washington, DC" disambiguator from the prompt -> substring-intent (equivalent-correct).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 467 OUT1 206 |
Tool Calling - Simple: Live Services, Weather & Commands (set 7)
1 passed2 failed |
33.3% |
IN16 193 OUT6 512 | |
| × | SimpleServiceIdIroningChatuchakBFCL/live_simple_181-106-0 (get_service_id): ironing, Bangkok(Chatuchak), rating>=4.5 -> service_id=2, province_id omitted/1, district_name=Chatuchak, rating=4.5.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.district_name?.ToLower()).AnyOf(["chatuchak"]) |
Failed | IN4 586 OUT1 111 |
| × | SimpleWeatherBerkeleySaturdayBFCL/live_simple_205-116-13 (Weather_1_GetWeather): city=Berkeley, upcoming Saturday (today Tue 2023-04-25) -> 2023-04-29.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 037 OUT3 087 |
| ✓ | SimpleWeatherRutherfordBFCL/live_simple_204-116-12 (Weather_1_GetWeather): city=Rutherford (prompt 'Rutherford, NJ'), date=2023-04-22. NOTE: GT city is bare "Rutherford"; model faithfully copies the "Rutherford, NJ" disambiguator from the prompt -> substring-intent (equivalent-correct).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 570 OUT2 314 |
Tool Calling - Simple: Live Services, Weather & Commands (set 8)
3 passed |
100.0% |
IN13 820 OUT5 142 | |
| ✓ | SimpleProvidersKhlongSamWaBFCL/live_simple_182-107-0 (get_service_providers): ironing, Bangkok, Khlong Sam Wa / Sai Kong, rating>=4.5.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 639 OUT2 499 |
| ✓ | SimpleWeatherLondonBFCL/live_simple_206-116-14 (Weather_1_GetWeather): city=London (prompt 'London, England'), date=2023-03-05.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 591 OUT1 426 |
| ✓ | SimpleWeatherSacramentoBFCL/live_simple_207-116-15 (Weather_1_GetWeather): city=Sacramento, date=2023-04-22.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 590 OUT1 217 |
Tool Calling - Simple: Live Services, Weather & Commands (set 9)
5 passed1 failed |
83.3% |
IN30 322 OUT4 970 | |
| ✓ | SimpleCmdEchoTimeUnitBFCL/live_simple_157-95-14 (cmd_controller.execute): current time -> 'echo %time%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 619 OUT1 366 |
| × | SimpleCmdListCDriveUnitBFCL/live_simple_156-95-13 (cmd_controller.execute): list C drive -> 'dir C:\'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 124 OUT1 249 |
| ✓ | SimpleCmdStartCameraBFCL/live_simple_160-95-17 (cmd_controller.execute): open camera -> 'start microsoft.windows.camera:'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 614 OUT244 |
| ✓ | SimpleCmdTaskkillCodeBFCL/live_simple_158-95-15 (cmd_controller.execute): kill VS Code -> 'taskkill /IM code.exe /F'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 614 OUT715 |
| ✓ | SimpleCmdTimeout10BFCL/live_simple_159-95-16 (cmd_controller.execute): timer 10s -> 'timeout 10'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 634 OUT578 |
| ✓ | SimpleProvidersChonburiTomorrowBFCL/live_simple_185-110-0 (get_service_providers): ironing, Chonburi, rating>=4.5, tomorrow of 20230929 -> 2023-09-30.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 717 OUT818 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 1)
2 passed1 failed |
66.7% |
IN16 345 OUT5 913 | |
| × | SimpleClassifyGreetingsAndLinksBFCL/live_simple_105-62-0 (classify): clean_hello=['hello'], faq_link_accounts_start=['link my account','connect my accounts'].1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 079 OUT2 030 |
| ✓ | SimpleCmdDockerVersionBFCL/live_simple_141-94-0 (cmd_controller.execute): command='docker --version'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 576 OUT716 |
| ✓ | SimpleReqNodesPekAciBFCL/live_simple_128-83-0 (requests.get): nodes list url with ip 192.168.11.33, fabricName='PEK-ACI'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 690 OUT3 167 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 10)
4 passed |
100.0% |
IN19 605 OUT2 587 | |
| ✓ | SimpleCmdShutdownBFCL/live_simple_150-95-7 (cmd_controller.execute): shutdown the pc via 'shutdown /s /t 0'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 866 OUT398 |
| ✓ | SimpleProcessImageMoondreamBFCL/live_simple_122-78-0 (process_data): image.png + long question + model 'vikhyatk/moondream2'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 978 OUT728 |
| ✓ | SimpleReqDataLimit50BFCL/live_simple_135-88-0 (requests.get): data endpoint url, params limit=50.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 897 OUT588 |
| ✓ | SimpleSumNumbersListBFCL/live_simple_116-72-0 (sum_numbers): numbers_list=[133, 34].1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 864 OUT873 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 11)
4 passed |
100.0% |
IN20 023 OUT7 179 | |
| ✓ | SimpleCmdWifiDisconnectBFCL/live_simple_151-95-8 (cmd_controller.execute): disconnect wifi via 'netsh wlan disconnect'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 893 OUT335 |
| ✓ | SimplePipelineFurnitureImageBFCL/live_simple_123-79-0 (pipeline): analyze thewowstyle image with furniture/minimalist question.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 169 OUT3 297 |
| ✓ | SimpleReqTopologySalesAppBFCL/live_simple_136-89-0 (requests.get): topology url ip 192.168.1.1, filter accountName:AcmeCorp AND applicationName:SalesApp.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 050 OUT1 559 |
| ✓ | SimpleTemperatureHyderabadBFCL/live_simple_118-74-0 (get_temperature): units=Celsius, location='Hyderabad, India', time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 911 OUT1 988 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 12)
4 passed |
100.0% |
IN21 307 OUT6 774 | |
| ✓ | SimpleCmdDirDAndEchoBFCL/live_simple_152-95-9 (cmd_controller.execute): 'dir D:\ && echo testing.txt'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 273 OUT3 862 |
| ✓ | SimpleReqAnomaliesSummaryBFCL/live_simple_139-92-0 (requests.get): anomalies summary url ip 192.168.1.1, params fabric='network222'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 394 OUT1 242 |
| ✓ | SimpleRescheduleAlice8pmBFCL/live_simple_137-90-0 (reschedule): identifier='Alice-One-one-One', 2023-11-01T20:00:00, Europe/London.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 306 OUT386 |
| ✓ | SimpleWeatherForecastParisBFCL/live_simple_120-76-0 (weather.forecast): Paris, France 2023-04-03..05, Celsius, defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 334 OUT1 284 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 13)
3 passed |
100.0% |
IN14 712 OUT3 813 | |
| ✓ | SimpleCmdDelPuttyBFCL/live_simple_153-95-10 (cmd_controller.execute): remove putty shortcut via del with quoted path.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 818 OUT738 |
| ✓ | SimpleDynamicPricingSaaSBFCL/live_simple_113-69-0 (calculate_dynamic_pricing): geolocation='34.0522, -118.2437', base=100, min=90.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 041 OUT1 497 |
| ✓ | SimpleRescheduleBob4pmBFCL/live_simple_138-91-0 (reschedule): identifier='Bob-123', 2023-11-01T16:00:00, Europe/London.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 853 OUT1 578 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 14)
3 passed |
100.0% |
IN14 094 OUT2 727 | |
| ✓ | SimpleCmdStartYoutubeBFCL/live_simple_154-95-11 (cmd_controller.execute): open youtube rickroll via start command.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 711 OUT541 |
| ✓ | SimpleGetTicketsCompanyOpenBFCL/live_simple_124-80-0 (get_tickets): customer (company name in prompt), status=open, limit=10.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 696 OUT972 |
| ✓ | SimpleGreetPartnerJeffBFCL/live_simple_140-93-0 (greet_partner): user='Emily', partner='Jeff', relationship='husband'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 687 OUT1 214 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 15)
1 passed2 failed |
33.3% |
IN14 467 OUT4 364 | |
| ✓ | SimpleCalculateTaxNewYorkBFCL/live_simple_104-61-2 (calculate_tax): purchase_amount=250, state='NY', all optionals omitted/default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 834 OUT2 377 |
| × | SimpleCmdDateBFCL/live_simple_155-95-12 (cmd_controller.execute): what is the date via 'date /t' or 'echo %date%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: args.command.ToLower().AnyOf(["date /t", "echo %date%"]) |
Failed | IN4 803 OUT930 |
| × | SimpleConcatJohnDoeBFCL/live_simple_125-81-0 (concat_strings): strings=['John','Doe'], separator=' '. EXPECTED MODEL MISS: 'combine ... into a full name' requires separator=' ' (John Doe). The 27B tends to omit separator (=> 'JohnDoe'); that is a legit model failure of the measured competence, so the ' ' assertion stays strict.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.separator.AnyOf([" "])
Values:
args.separator = null |
Failed | IN4 830 OUT1 057 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 2)
3 passed |
100.0% |
IN16 109 OUT6 207 | |
| ✓ | SimpleCmdDirDesktopBFCL/live_simple_142-94-1 (cmd_controller.execute): command='dir Desktop'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 041 OUT860 |
| ✓ | SimpleRecordBankNineIntentsBFCL/live_simple_106-63-0 (record): classify 20 bank queries into 9 intents (2 buckets empty).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 963 OUT2 817 |
| ✓ | SimpleReqNodesFabEdBFCL/live_simple_129-83-1 (requests.get): nodes list url with ip 3.321.3232.2, fabricName='fab-ed'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 105 OUT2 530 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 3)
4 passed |
100.0% |
IN21 459 OUT4 882 | |
| ✓ | SimpleCmdDockerPsBFCL/live_simple_143-95-0 (cmd_controller.execute): command='docker ps'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 217 OUT899 |
| ✓ | SimpleRecordTelephoneHoursAccountsBFCL/live_simple_107-64-0 (record): classify 7 queries into telephone-banking / bank-hours / accounts.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 640 OUT2 621 |
| ✓ | SimpleTelemetryGlobalFabricStatusBFCL/live_simple_126-82-0 (telemetry.flowrules.interfaceInfo.get): fabric='Global-Fabric', node=1200, pod=3, ethernet, status.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 299 OUT610 |
| ✓ | SimpleUpdateProfileEmailAgeBFCL/live_simple_114-70-0 (update_user_profile): user_id=12345, email+age set, bio omitted, notify default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 303 OUT752 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 4)
3 passed1 failed |
75.0% |
IN20 858 OUT4 962 | |
| ✓ | SimpleCmdKillFirefoxBFCL/live_simple_144-95-1 (cmd_controller.execute): close firefox via taskkill.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 081 OUT887 |
| × | SimpleRecordMoneyTransactionsCreditBFCL/live_simple_108-65-0 (record): classify 10 queries into money-movement / transactions / credit-limit / link-accounts. EXPECTED MODEL MISS: the 27B classifies 'show me all outgoing wire transfers' as a transaction-view (get_transactions_start) rather than money_movement_start per GT, so the get_transactions count comes out as 5 instead of 4. GT bucketing kept strict; this is a legit model classification failure.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.money_movement_start.Count == 1
Values:
args.money_movement_start.Count = 0 |
Failed | IN5 549 OUT2 813 |
| ✓ | SimpleReverseSayHiBFCL/live_simple_117-73-0 (reverse_input): input_value='say hi'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 069 OUT742 |
| ✓ | SimpleTelemetryTestDeStatusBFCL/live_simple_127-82-1 (telemetry.flowrules.interfaceInfo.get): fabric='test-de', node=5, pod=1, ethernet, status.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 159 OUT520 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 5)
4 passed |
100.0% |
IN20 882 OUT6 952 | |
| ✓ | SimpleCmdDirCDriveBFCL/live_simple_145-95-2 (cmd_controller.execute): list C drive => 'dir C:\'. EXPECTED MODEL MISS: for the terse/typo'd 'LIST C DRIIVE' the 27B currently returns 'dir d:' — the wrong drive. The accepted-command list is broadened with the no-backslash 'dir c:' variant for fairness, but a wrong drive letter is a legit model failure.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 105 OUT2 020 |
| ✓ | SimplePrimeMinistersThreeBFCL/live_simple_119-75-0 (GetPrimeMinisters): countries=[Australia, Canada, India].1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 149 OUT718 |
| ✓ | SimpleRecordAutoWithdrawIncomeBFCL/live_simple_109-66-0 (record): classify 5 loan/withdraw queries; outofscope empty.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 357 OUT2 032 |
| ✓ | SimpleReqInterfaceStatsNetwork1BFCL/live_simple_130-84-0 (requests.get): interfaceInfo stats, ip 192.120.45.67, Network1/node 12/pod 10/ethernet/statistics.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 271 OUT2 182 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 6)
4 passed |
100.0% |
IN21 534 OUT3 581 | |
| ✓ | SimpleCmdStartHuggingfaceBFCL/live_simple_146-95-3 (cmd_controller.execute): open huggingface via start command.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 292 OUT349 |
| ✓ | SimpleGenerateSalaryChartBFCL/live_simple_121-77-0 (generate_chart): 5 job labels and salaries in order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 446 OUT861 |
| ✓ | SimpleRecordAtmLocationOnlyBFCL/live_simple_110-67-0 (record): single query 'Need ATM location' -> atm_finder_start only.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 291 OUT280 |
| ✓ | SimpleReqInterfaceSpeedNetwork1BFCL/live_simple_131-84-1 (requests.get): interfaceInfo Speed, ip 192.120.45.67, Network1/node 12/pod 10/ethernet/Speed.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 505 OUT2 091 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 7)
2 passed1 failed |
66.7% |
IN24 416 OUT6 351 | |
| ✓ | SimpleCmdKillTimerBFCL/live_simple_147-95-4 (cmd_controller.execute): remove timer.exe via taskkill.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 865 OUT407 |
| × | SimpleRecordBalanceAndAtmBFCL/live_simple_111-67-1 (record): 10 queries -> 8 balance + 2 atm; other buckets empty.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 4 |
Failed | IN14 679 OUT3 808 |
| ✓ | SimpleReqDeviceConnectorVersionsBFCL/live_simple_132-85-0 (requests.get): device connector versions JSON url, no params.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 872 OUT2 136 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 8)
2 passed1 failed |
66.7% |
IN12 351 OUT2 173 | |
| × | SimpleCmdWhatFileInCBFCL/live_simple_148-95-5 (cmd_controller.execute): 'what is the file in c' => 'dir C:\'. EXPECTED MODEL MISS: the 27B mis-selects the 'record' classifier (dropping the request into its 'outofscope' bucket) instead of cmd_controller_execute. record is a legitimate distractor here (not a looping universal collider), so this stays as a real tool-selection failure — no isolation needed.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 0 |
Failed | IN2 431 OUT218 |
| ✓ | SimpleRecordWeatherOutOfScopeBFCL/live_simple_112-68-0 (record): single query 'what is the weather like' -> outofscope only.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 958 OUT500 |
| ✓ | SimpleReqRecommendationsDacBFCL/live_simple_133-86-0 (requests.get): recommendations url, params advisoryId='dac'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 962 OUT1 455 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 9)
2 passed1 failed |
66.7% |
IN13 865 OUT3 447 | |
| ✓ | SimpleCalculateSum133And34BFCL/live_simple_115-71-0 (calculate_sum): number1=133, number2=34.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 607 OUT569 |
| ✓ | SimpleCmdEchoHiBFCL/live_simple_149-95-6 (cmd_controller.execute): say hi via echo command.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 581 OUT532 |
| × | SimpleReqInsightsGroupsBFCL/live_simple_134-87-0 (requests.get): insights groups url, params limit=20, sort='desc'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.params.sort?.ToLower()).AnyOf(["desc", "descending"]) |
Failed | IN4 677 OUT2 346 |
Tool Calling - Simple: Math, Physics & Geometry (set 1)
11 passed1 failed |
91.7% |
IN92 444 OUT14 691 | |
| ✓ | SimpleAlgebraQuadraticRootsBFCL/simple_python_3 (algebra.quadratic_roots): a=1, b=-3, c=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 395 OUT1 011 |
| ✓ | SimpleAreaUnderCurveXSquaredBFCL/simple_python_13 (calculate_area_under_curve): function y=x^2, interval [1,3], method omitted/trapezoidal.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 388 OUT1 338 |
| ✓ | SimpleCircleCircumferenceInchesBFCL/simple_python_7 (calculate_circumference): radius=4, unit inches/in.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 376 OUT1 408 |
| ✓ | SimpleDerivativeCubicPolyBFCL/simple_python_14 (calculate_derivative): function 3x^2+2x-1, x_value omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 383 OUT1 203 |
| ✓ | SimpleElectricFieldFromChargeBFCL/simple_python_39 (calculate_electric_field): charge=2, distance=3, permitivity omitted/8.854e-12.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 384 OUT1 307 |
| ✓ | SimpleFinalVelocityFromDistanceBFCL/simple_python_30 (kinematics.final_velocity_from_distance): acceleration=4, distance=300, initial_velocity omitted/0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 411 OUT1 217 |
| ✓ | SimpleFinalVelocityFromHeightBFCL/simple_python_25 (calculate_final_velocity): height=150, initial_velocity omitted/0, gravity omitted/9.81.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 392 OUT1 990 |
| ✓ | SimpleGeometryAreaCircleBFCL/simple_python_8 (geometry.area_circle): radius=10, units omitted or 'meters'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 365 OUT581 |
| × | SimpleGetPrimeFactors450BFCL/simple_python_17 (get_prime_factors): number=450, formatted omitted or true.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 190 OUT2 107 |
| ✓ | SimpleMagneticFieldLoopBFCL/simple_python_40 (calculate_magnetic_field): current=5, radius=4, permeability omitted/12.57e10.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 396 OUT1 213 |
| ✓ | SimpleMathGcd40And50BFCL/simple_python_19 (math.gcd): num1=40, num2=50.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 379 OUT524 |
| ✓ | SimpleTriangleAreaWithUnitBFCL/simple_python_0 (calculate_triangle_area): base=10, height=5, unit omitted or 'units'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 385 OUT792 |
Tool Calling - Simple: Math, Physics & Geometry (set 2)
12 passed |
100.0% |
IN88 315 OUT12 720 | |
| ✓ | SimpleAverageVelocityBFCL/simple_python_26 (calculate_velocity): distance=50, duration=2, unit omitted or 'km/h'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 359 OUT819 |
| ✓ | SimpleCalculusDerivativeAtValueBFCL/simple_python_16 (calculus.derivative): function 2x^2, value=1, function_variable omitted or 'x'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 350 OUT768 |
| ✓ | SimpleElectricFieldStrengthBFCL/simple_python_44 (calculate_electric_field_strength): charge=0.01, distance=4, medium omitted or 'vacuum'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 360 OUT1 477 |
| ✓ | SimpleFinalVelocityConstAccelBFCL/simple_python_31 (calculate_final_velocity): initial_velocity=0, acceleration=9.8, time=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 397 OUT1 001 |
| ✓ | SimpleGeometryCalculateAreaCircleBFCL/simple_python_9 (geometry.calculate_area_circle): radius=5, unit omitted or 'units'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 343 OUT1 004 |
| ✓ | SimpleGeometryCircumferenceBFCL/simple_python_12 (geometry.circumference): radius=3, units omitted or 'cm'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 327 OUT798 |
| ✓ | SimpleIntegrateCubeSimpsonBFCL/simple_python_15 (integrate): function x^3, start_x=-2, end_x=3, method simpson.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 392 OUT1 556 |
| ✓ | SimpleMagneticFieldStrengthWireBFCL/simple_python_43 (calculate_magnetic_field_strength): current=20, distance=10, permeability omitted/12.57e-7.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 367 OUT1 053 |
| ✓ | SimpleMathHcfBFCL/simple_python_20 (math.hcf): number1=36, number2=24.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 345 OUT547 |
| ✓ | SimpleNumberAnalysisPrimeFactorsBFCL/simple_python_18 (number_analysis.prime_factors): number=123456.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 341 OUT1 394 |
| ✓ | SimpleRightTriangleAreaCmBFCL/simple_python_10 (calculate_area): base=6, height=10, unit omitted or 'cm'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 376 OUT1 160 |
| ✓ | SimpleSolveQuadraticEquation265BFCL/simple_python_4 (solve_quadratic_equation): a=2, b=6, c=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 358 OUT1 143 |
Tool Calling - Simple: Math, Physics & Geometry (set 3)
9 passed |
100.0% |
IN60 212 OUT12 421 | |
| ✓ | SimpleBoilingMeltingPointsBFCL/simple_python_47 (get_boiling_melting_points): substance='water', sea_level=5000.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 684 OUT902 |
| ✓ | SimpleDisplacementBFCL/simple_python_28 (calculate_displacement): initial_velocity=10, time=5, acceleration=9.8.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 702 OUT1 121 |
| ✓ | SimpleElectrostaticPotentialBFCL/simple_python_38 (calculate_electrostatic_potential): charge1=1e-9, charge2=2e-9, distance=0.05, constant omitted/8.99e9.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 716 OUT2 443 |
| ✓ | SimpleFinalSpeedFromHeightBFCL/simple_python_32 (calculate_final_speed): initial_velocity=0, height=100, gravity omitted or 9.8.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 672 OUT1 339 |
| ✓ | SimpleNumberTheoryGcdBFCL/simple_python_21 (number_theory.gcd): number1=36, number2=48.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 685 OUT904 |
| ✓ | SimplePrimeFactorizeDictionaryBFCL/simple_python_23 (prime_factorize): number=60, return_type='dictionary'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 676 OUT783 |
| ✓ | SimpleRouteEstimateTimeStopsBFCL/simple_python_37 (route.estimate_time): SF -> LA, stops {Santa Barbara, Monterey} in either order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 696 OUT3 204 |
| ✓ | SimpleSolveQuadraticAllRootsBFCL/simple_python_5 (solve_quadratic): a=3, b=-11, c=-4, root_type='all' (user asked for all roots).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 705 OUT878 |
| ✓ | SimpleTriangleAreaNoUnitBFCL/simple_python_11 (calculate_triangle_area): base=10, height=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 676 OUT847 |
Tool Calling - Simple: Math, Physics & Geometry (set 4)
9 passed |
100.0% |
IN59 738 OUT13 717 | |
| ✓ | SimpleCalculateDensityBFCL/simple_python_48 (calculate_density): mass=45, volume=15, unit omitted or 'kg/m^3'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 640 OUT622 |
| ✓ | SimpleElectromagneticForceBFCL/simple_python_41 (electromagnetic_force): charge1=5, charge2=7, distance=3, medium_permittivity omitted/8.854e-12.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 642 OUT1 609 |
| ✓ | SimpleFinalSpeedFreeFallTimeBFCL/simple_python_29 (calculate_final_speed): initial_speed omitted/0, time=5, gravity omitted/-9.81.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 636 OUT1 358 |
| ✓ | SimpleMathFactorial5BFCL/simple_python_1 (math.factorial): number=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 600 OUT616 |
| ✓ | SimpleMathGcd12And15BFCL/simple_python_22 (math.gcd): num1=12, num2=15.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 633 OUT541 |
| ✓ | SimpleShortestDrivingDistanceBFCL/simple_python_36 (get_shortest_driving_distance): origin=New York City, destination=Washington D.C., unit omitted/km.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 625 OUT1 189 |
| ✓ | SimpleSolveQuadraticRoots253BFCL/simple_python_6 (solve_quadratic): a=2, b=5, c=3.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 638 OUT1 989 |
| ✓ | SimpleThermoEnergyVaporizationBFCL/simple_python_45 (thermo.calculate_energy): mass=100, phase_transition=vaporization, substance omitted/water.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 661 OUT3 321 |
| ✓ | SimpleTravelItineraryTokyoBFCL/simple_python_34 (travel_itinerary_generator): destination=Tokyo, days=7, daily_budget=100, exploration_type=nature.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 663 OUT2 472 |
Tool Calling - Simple: Math, Physics & Geometry (set 5)
8 passed |
100.0% |
IN51 550 OUT9 598 | |
| ✓ | SimpleAbsolutePressureBFCL/simple_python_49 (calc_absolute_pressure): atm_pressure=1, gauge_pressure=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 437 OUT1 330 |
| ✓ | SimpleFinalTemperatureMixBFCL/simple_python_46 (calculate_final_temperature): mass1=20, temperature1=30, mass2=15, temperature2=60, shc omitted/4.2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 475 OUT2 215 |
| ✓ | SimpleFinalVelocityAccelTimeBFCL/simple_python_27 (final_velocity): initial_velocity=10, acceleration=2, time=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 469 OUT1 008 |
| ✓ | SimpleGetDirectionsFastestBFCL/simple_python_33 (get_directions): start=Sydney, end=Melbourne, route_type omitted or 'fastest'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 422 OUT723 |
| ✓ | SimpleMathGcd12And18BFCL/simple_python_24 (math.gcd): num1=12, num2=18.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 423 OUT495 |
| ✓ | SimpleMathHypotenuseBFCL/simple_python_2 (math.hypot): x=4, y=5, z omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 433 OUT869 |
| ✓ | SimpleResonantFrequencyLCBFCL/simple_python_42 (calculate_resonant_frequency): inductance=0.05 (50mH), capacitance=0.0001 (100uF), round_off omitted/2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 460 OUT1 915 |
| ✓ | SimpleVeganRestaurantFindBFCL/simple_python_35 (vegan_restaurant.find_nearby): location='New York, NY', operating_hours=23 (11 PM).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 431 OUT1 043 |
Tool Calling - Simple: Science, Data & Biology (set 1)
12 passed1 failed |
92.3% |
IN107 008 OUT17 326 | |
| ✓ | SimpleArraySortBFCL/simple_python_87 (array_sort): list=[5,3,4,1,2], order=ascending.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 213 OUT1 699 |
| ✓ | SimpleAverageTemperatureBFCL/simple_python_78 (average_temperature): location=Austin, days=3, temp_unit=Celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 197 OUT693 |
| ✓ | SimpleBacteriaEvolutionRateBFCL/simple_python_74 (calculate_bacteria_evolution_rate): start=5000, dup_freq=1, duration=6, gen_time omitted or 20.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 230 OUT1 764 |
| ✓ | SimpleBmiKgCmBFCL/simple_python_84 (calculate_bmi): weight=85, height=180, unit omitted or 'metric'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 216 OUT1 651 |
| ✓ | SimpleCreateHistogramBFCL/simple_python_79 (create_histogram): data=[85,90,88,92,86,89,91], bins=5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 273 OUT1 254 |
| ✓ | SimpleCreatureFitnessBFCL/simple_python_72 (calculate_fitness): trait_values=[0.8,0.7], trait_contributions=[0.4,0.6].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 276 OUT1 275 |
| ✓ | SimpleDbFetchStudentsBFCL/simple_python_89 (db_fetch_records): database=StudentDB, table=students, conditions={department:Science, school:Bluebird High School}, fetch_limit omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 227 OUT1 279 |
| ✓ | SimpleEntropyChangeSubstanceBFCL/simple_python_50 (entropy_change.calculate): substance=ice, mass=1, init=0, final=100, pressure omitted or 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 265 OUT1 843 |
| ✓ | SimpleGenerateDnaSequenceBFCL/simple_python_71 (generate_DNA_sequence): length=100, preferences=[G,C] in either order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 216 OUT419 |
| ✓ | SimpleGpsCoordinateDistanceBFCL/simple_python_83 (calculate_distance): coord1=[33.4484,-112.074], coord2=[34.0522,-118.2437], unit=miles.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 305 OUT2 910 |
| ✓ | SimpleImdbMoviesByActorBFCL/simple_python_92 (imdb.find_movies_by_actor): actor='Leonardo DiCaprio', year=2010, category omitted or 'all'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 210 OUT840 |
| ✓ | SimplePopulationProjectionBFCL/simple_python_73 (population_projections): country=United States/USA, years=20, growth_rate omitted or 1.2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 193 OUT888 |
| × | SimpleRestaurantFindNearbyBFCL/simple_python_77 (restaurant.find_nearby): location='Los Angeles, CA', dietary_preference=[Vegan].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["los angeles, ca"]) |
Failed | IN8 187 OUT811 |
Tool Calling - Simple: Science, Data & Biology (set 2)
13 passed |
100.0% |
IN106 617 OUT16 397 | |
| ✓ | SimpleBmiMetersBFCL/simple_python_88 (calculate_BMI): weight_kg=70, height_m=1.75.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 202 OUT1 182 |
| ✓ | SimpleClockAngleBFCL/simple_python_98 (calculate_clock_angle): hours=6, minutes=30, round_to omitted or 2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 186 OUT1 360 |
| ✓ | SimpleDatabaseQueryConditionsBFCL/simple_python_96 (database.query): table=user, conditions=[{age,>,25},{job,=,engineer}].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 225 OUT1 102 |
| ✓ | SimpleElephantPopulationBFCL/simple_python_75 (elephant_population_estimate): current=35000, growth_rate=0.015, years=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 226 OUT1 682 |
| ✓ | SimpleEntropyChangeIsothermalBFCL/simple_python_51 (calculate_entropy_change): initial_temp=300, final_temp=400, heat_capacity=5, isothermal omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 234 OUT1 601 |
| ✓ | SimpleEvolutionPredictionDarwinBFCL/simple_python_76 (prediction.evolution): species=Homo Sapiens, years=50, model=Darwin.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 200 OUT998 |
| ✓ | SimpleFindRestaurantsThaiBFCL/simple_python_80 (find_restaurants): location=Manhattan variants, food_type=Thai, number=5, dietary_requirements=[vegan].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 208 OUT2 242 |
| ✓ | SimpleGeoDistanceCitiesBFCL/simple_python_85 (geo_distance.calculate): start='Boston, MA', end='Washington, D.C.', units omitted or 'miles'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 187 OUT1 436 |
| ✓ | SimpleMathFactorialBFCL/simple_python_97 (math.factorial): number=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 149 OUT722 |
| ✓ | SimplePlotSineWaveBFCL/simple_python_99 (plot_sine_wave): start=0.0, end=6.2832, frequency=5, amplitude omitted or 1, phase_shift omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 197 OUT824 |
| ✓ | SimpleTheaterMovieReleasesBFCL/simple_python_93 (get_theater_movie_releases): location=LA, timeframe=7, format=IMAX.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 191 OUT1 170 |
| ✓ | SimpleTriangleAreaMetersBFCL/simple_python_95 (calc_area_triangle): base=5, height=3.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 175 OUT853 |
| ✓ | SimpleUpdateUserInfoBFCL/simple_python_94 (update_user_info): user_id=43523, update_info={name:John Doe, email:johndoe@email.com}, database omitted or CustomerInfo. NOTE: single-call contract kept intentionally; the model very occasionally emits a spurious second call (args are correct ~3/3 sampled runs), a model-reliability signal rather than a test defect.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 237 OUT1 225 |
Tool Calling - Simple: Science, Data & Biology (set 3)
12 passed |
100.0% |
IN91 462 OUT14 353 | |
| ✓ | SimpleCalculateAverageBFCL/simple_python_82 (calculate_average): numbers=[12,15,18,20,21,26,30].1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 660 OUT1 248 |
| ✓ | SimpleCellDensityBFCL/simple_python_57 (calculate_cell_density): optical_density=0.6, dilution=5, calibration_factor omitted or 1e9.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 632 OUT1 019 |
| ✓ | SimpleCellInfoHumanBFCL/simple_python_55 (biology.get_cell_info): cell_type=human, detailed=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 596 OUT1 476 |
| ✓ | SimpleCellProteinsMembraneBFCL/simple_python_56 (cellbio.get_proteins): cell_compartment=plasma membrane, include_description omitted/true/false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 597 OUT1 135 |
| ✓ | SimpleCityDistanceTrainBFCL/simple_python_86 (city_distance.find_shortest): start=New York, end=Los Angeles, transportation=train, allow_transfer=true.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 640 OUT2 462 |
| ✓ | SimpleEmployeeFetchDataBFCL/simple_python_90 (employee.fetch_data): company='ABC Ltd.', employee_id=345, data_field=[Personal Info, Job History].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 641 OUT777 |
| ✓ | SimpleFetchDnaSequenceBFCL/simple_python_53 (fetch_DNA_sequence): DNA_id=DNA123, format omitted or 'fasta', upstream omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 599 OUT693 |
| ✓ | SimpleGetRestaurantSushiBFCL/simple_python_91 (get_restaurant): cuisine=sushi, location=Boston, condition=open on Sundays.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 614 OUT1 274 |
| ✓ | SimpleHeatCapacityAirBFCL/simple_python_52 (calc_heat_capacity): temp=298, volume=10, gas omitted or 'air'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 643 OUT953 |
| ✓ | SimpleMolecularWeightBFCL/simple_python_59 (calculate_molecular_weight): compound=C6H12O6, to_unit=grams/mole or g/mol.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 626 OUT896 |
| ✓ | SimpleOrganelleFunctionLookupBFCL/simple_python_58 (cell_biology.function_lookup): molecule=ATP synthase, organelle=mitochondria, specific_function=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 615 OUT1 306 |
| ✓ | SimpleProteinSequenceBFCL/simple_python_54 (get_protein_sequence): gene=BRCA1, species omitted or 'Homo sapiens'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 599 OUT1 114 |
Tool Calling - Simple: Science, Data & Biology (set 4)
12 passed |
100.0% |
IN94 128 OUT13 927 | |
| ✓ | SimpleAnalyzeDnaSequenceBFCL/simple_python_62 (analyze_dna_sequence): sequence + reference_sequence given, mutation_type omitted or 'substitution'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 900 OUT1 363 |
| ✓ | SimpleDiabetesPredictionBFCL/simple_python_61 (diabetes_prediction): weight=150, height=70 (5ft10in), activity_level=lightly active.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 861 OUT1 536 |
| ✓ | SimpleFastestRouteAvoidTollsBFCL/simple_python_81 (map_routing.fastest_route): start=San Francisco/SF, end=Los Angeles/LA, avoid_tolls=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 830 OUT700 |
| ✓ | SimpleForestGrowthForecastBFCL/simple_python_68 (forest_growth_forecast): location=Yellowstone National Park, years=5, include_human_impact=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 837 OUT916 |
| ✓ | SimpleGeneticSimilarityBFCL/simple_python_63 (genetics.calculate_similarity): species1=Human, species2=Chimp/Chimpanzee, format omitted or 'percentage'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 820 OUT1 402 |
| ✓ | SimpleGenotypeFrequencyBFCL/simple_python_64 (calculate_genotype_frequency): allele_frequency=0.3, genotype=AA.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 839 OUT588 |
| ✓ | SimpleIdentifyBirdBFCL/simple_python_67 (identify_bird): color=green, habitat=forest, size=small.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 813 OUT1 287 |
| ✓ | SimpleMutationTypeFindBFCL/simple_python_60 (mutation_type.find): snp_id=rs6034464, species omitted or 'Homo sapiens'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 843 OUT1 367 |
| ✓ | SimplePopulationDensityBFCL/simple_python_65 (calculate_density): country=Brazil, year='2022', population=213000000, land_area=8500000.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 886 OUT1 283 |
| ✓ | SimplePrecipitationStatsBFCL/simple_python_66 (ecology_data.precipitation_stats): location=Amazon rainforest, time_frame=six_months.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 828 OUT879 |
| ✓ | SimpleTurtlePopulationBFCL/simple_python_69 (ecology.get_turtle_population): location=Mississippi river, year=2020, species=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 836 OUT1 324 |
| ✓ | SimpleVehicleEmissionBFCL/simple_python_70 (calculate_vehicle_emission): vehicle_type=gas, miles_driven=1500, emission_factor omitted or 355.48.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 835 OUT1 282 |
Tool Calling - Simple: Sports & Games (set 1)
11 passed1 failed |
91.7% |
IN95 925 OUT15 624 | |
| ✓ | SimpleBattingAverageComputeBFCL/simple_python_302 (calculate_batting_average): hits=180, at_bats=600, decimal_places=3 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 667 OUT1 348 |
| ✓ | SimpleBoardGameInfoTicketBFCL/simple_python_328 (boardgame.get_info): Ticket to Ride, {rating, player count} either order, language omitted/English.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 628 OUT1 282 |
| ✓ | SimpleFindCardInDeckBFCL/simple_python_335 (find_card_in_deck): rank Queen, suit Hearts, deck omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 629 OUT774 |
| ✓ | SimpleGameListByYearBFCL/simple_python_342 (game_list.get_games): year 2019, multiplayer true, ESRB Everyone.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 650 OUT1 195 |
| ✓ | SimpleGameWinnerByDateBFCL/simple_python_307 (game_result.get_winner): Lakers/Clippers either order, date 2021-01-28, venue omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 656 OUT848 |
| ✓ | SimpleHighestGameScoreBFCL/simple_python_349 (game_score.highest): Overwatch, PC, region Global or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 633 OUT1 071 |
| × | SimpleMatchScheduleBFCL/simple_python_308 (sports.match_schedule): Man United, 5 matches, English Premier League or omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 651 OUT3 110 |
| ✓ | SimpleNoteDurationBFCL/simple_python_300 (music.calculate_note_duration): 440/880 Hz, tempo omitted or 120.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 648 OUT988 |
| ✓ | SimplePokerGameWinnerMultiBFCL/simple_python_337 (poker_game_winner): 4 players + per-player hands dict, type omitted/Texas Holdem.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 839 OUT2 062 |
| ✓ | SimpleSoccerPlayerStatsBFCL/simple_python_303 (soccer_stat.get_player_stats): Cristiano Ronaldo, 2019-2020, league omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 646 OUT790 |
| ✓ | SimpleSportRankingPlayerBFCL/simple_python_316 (get_sport_ranking): Tennis, Serena Williams, gender omitted/all/female.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 631 OUT1 258 |
| ✓ | SimpleTeamRankRegularBFCL/simple_python_317 (get_team_rank): LA Lakers, NBA, 2021, type regular.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 647 OUT898 |
Tool Calling - Simple: Sports & Games (set 2)
11 passed |
100.0% |
IN78 470 OUT13 268 | |
| ✓ | SimpleBoardGameInfoCatanBFCL/simple_python_330 (board_game_info): Catan, {average_review_rating, age_range}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 126 OUT976 |
| ✓ | SimpleCricketBattingScoreBFCL/simple_python_306 (average_batting_score): Virat Kohli, 10 matches, T20 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 125 OUT1 031 |
| ✓ | SimpleFifaTeamRankingBFCL/simple_python_318 (get_team_ranking): Germany, 2021, gender men or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 132 OUT800 |
| ✓ | SimpleMatchResultsTwoTeamsBFCL/simple_python_325 (sports.match_results): Chicago Bulls vs Los Angeles Lakers, season omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 110 OUT1 407 |
| ✓ | SimplePokerDetermineWinnerTwoBFCL/simple_python_340 (card_games.poker_determine_winner): John vs Mike with 5-card hands (suit symbols).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 258 OUT2 354 |
| ✓ | SimpleShuffleAndDrawBFCL/simple_python_336 (cards.shuffle_and_draw): num_cards 3.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 104 OUT966 |
| ✓ | SimpleSoccerPerformanceBFCL/simple_python_305 (sports_stats.get_performance): Messi, La Liga, 2020-2021, {goals scored, assists made}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 160 OUT1 192 |
| ✓ | SimpleTeamScheduleGamesBFCL/simple_python_327 (sports_team.get_schedule): Man United, 6 games, Premier League, location omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 128 OUT1 178 |
| ✓ | SimpleThirdChordBFCL/simple_python_301 (get_third_chord): key C, type major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 102 OUT1 823 |
| ✓ | SimpleTopPlayerBySportBFCL/simple_python_323 (sports_ranking.get_top_player): tennis, gender women.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 105 OUT777 |
| ✓ | SimpleVideoGamePlayerStatsBFCL/simple_python_343 (game_stats.fetch_player_statistics): Zelda, user Sam, platform Switch.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 120 OUT764 |
Tool Calling - Simple: Sports & Games (set 3)
9 passed |
100.0% |
IN58 073 OUT12 794 | |
| ✓ | SimpleCardDrawProbabilityBFCL/simple_python_338 (card_game_probability.calculate): total 52, desired 13 (hearts), cards_drawn 1 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 464 OUT1 044 |
| ✓ | SimpleCelebrityNetWorthBFCL/simple_python_313 (celebrity_net_worth.get): Messi, currency EUR/euro.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 448 OUT1 482 |
| ✓ | SimpleDefenseRankingBFCL/simple_python_315 (get_defense_ranking): season 2021, top 1 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 447 OUT1 000 |
| ✓ | SimpleFindAthleteBFCL/simple_python_311 (sports_db.find_athlete): Lebron James, sport Basketball, team omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 440 OUT1 860 |
| ✓ | SimpleGameItemStatsBFCL/simple_python_344 (get_game_item_stats): BotW, Guardian Sword+, stat power/power rating.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 484 OUT1 649 |
| ✓ | SimpleMonopolyOddsBFCL/simple_python_329 (monopoly_odds_calculator): number 7, dice_number 2, dice_faces 6 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 452 OUT1 060 |
| ✓ | SimpleNflPlayerRecordBFCL/simple_python_309 (nfl_data.player_record): Tom Brady, 2020, team omitted or Tampa Bay Buccaneers.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 454 OUT1 143 |
| ✓ | SimpleSportsRankingSeasonIntBFCL/simple_python_319 (sports_ranking): Man United, Premier League, season omitted or 2023.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 438 OUT1 262 |
| ✓ | SimpleTeamLatestScoreBFCL/simple_python_324 (team_score.get_latest): Los Angeles Lakers, include_opponent true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 446 OUT2 294 |
Tool Calling - Simple: Sports & Games (set 4)
6 passed1 failed |
85.7% |
IN45 001 OUT10 721 | |
| ✓ | SimpleCareerStatsBFCL/simple_python_310 (get_career_stats): LeBron James, team omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 954 OUT1 580 |
| ✓ | SimpleCelebrityAchievementsBFCL/simple_python_314 (sports_celebrity.get_major_achievements): Messi, sports Football/Soccer or omitted, team all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 972 OUT1 325 |
| ✓ | SimpleChessTopPlayersBFCL/simple_python_331 (board_game.chess.get_top_players): New York, min rating 2300, number_of_players 10 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 979 OUT960 |
| ✓ | SimpleDeckOddsSuitBFCL/simple_python_341 (deck_of_cards.odds): suit hearts, deck_type without_joker or normal.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 971 OUT1 158 |
| ✓ | SimpleGameValuationBFCL/simple_python_345 (game_valuation): Super Mario Bros., 1985, condition Like New/New.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 991 OUT2 173 |
| ✓ | SimpleTeamPositionDetailedBFCL/simple_python_320 (sports_ranking.get_team_position): Golden State Warriors, 2022-2023, detailed true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 007 OUT847 |
| × | SimpleTeamScoreWithStatsBFCL/simple_python_326 (get_team_score): Los Angeles Lakers, NBA, include_player_stats any.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 127 OUT2 678 |
Tool Calling - Simple: Sports & Games (set 5)
7 passed |
100.0% |
IN40 525 OUT10 732 | |
| ✓ | SimpleChessRatingBFCL/simple_python_332 (chess.rating): Magnus Carlsen, variant classical or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 772 OUT2 257 |
| ✓ | SimpleCollectablesInSeasonBFCL/simple_python_346 (get_collectables_in_season): Animal Crossing: New Horizons, Spring, item_type all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 795 OUT1 410 |
| ✓ | SimpleDetailedWeatherForecastBFCL/simple_python_333 (detailed_weather_forecast): London UK, 3 days, {high/low temp, humidity, precipitation}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 830 OUT1 495 |
| ✓ | SimpleFullHouseProbabilityBFCL/simple_python_339 (poker_probability.full_house): defaults deck_size 52, hand_size 5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 773 OUT1 184 |
| ✓ | SimplePlayerStatisticYearBFCL/simple_python_312 (player_statistic): Ronaldo, year 2021, team omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 779 OUT1 139 |
| ✓ | SimpleSoccerLastMatchBFCL/simple_python_347 (soccer.get_last_match): Liverpool F.C., include_stats true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 785 OUT2 450 |
| ✓ | SimpleSportsRankingSeasonStrBFCL/simple_python_321 (sports_ranking): Barcelona, La Liga, 2021.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 791 OUT797 |
Tool Calling - Simple: Sports & Games (set 6)
4 passed |
100.0% |
IN20 201 OUT5 235 | |
| ✓ | SimpleBlackjackWinnerBFCL/simple_python_334 (blackjack.check_winner): player [A,10], dealer [10,9], ace_value 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 080 OUT1 133 |
| ✓ | SimpleCreatePlayerProfileBFCL/simple_python_348 (create_player_profile): StarPlayer, class Mage, starting_level 5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 057 OUT703 |
| ✓ | SimpleCurrentTeamRankingBFCL/simple_python_322 (sports_ranking.get_current): Liverpool, Premier League, season omitted or 2023-2024.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 023 OUT1 174 |
| ✓ | SimplePlayerLastGameStatsBFCL/simple_python_304 (player_stats.getLastGame): LeBron James, Lakers, metrics {Points, Rebounds}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 041 OUT2 225 |
Tool Calling - Simple: Stats, Finance & ML (set 1)
8 passed |
100.0% |
IN51 703 OUT8 897 | |
| ✓ | SimpleCardProbabilityBFCL/simple_python_112 (calculate_probability): total_outcomes=52, favorable_outcomes=4 (king from deck), round_to omitted or 2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 411 OUT870 |
| ✓ | SimpleCompoundInterestNBFCL/simple_python_131 (calculate_compound_interest): principal=10000, rate=0.05, time=10, n=4.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 509 OUT1 158 |
| ✓ | SimpleInvestmentProfitBFCL/simple_python_134 (investment.predictProfit): investment_amount=5000, annual_return=0.07, years=5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 454 OUT1 526 |
| ✓ | SimpleLightTravelTimeBFCL/simple_python_100 (light_travel_time): distance_in_light_years=4, speed_of_light omitted or default.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 417 OUT731 |
| ✓ | SimpleQuarterlyDividendBFCL/simple_python_128 (finance.calculate_quarterly_dividend_per_share): total_payout=50000000, outstanding_shares=100000000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 465 OUT1 307 |
| ✓ | SimpleRandomForestTrainBFCL/simple_python_109 (random_forest.train): n_estimators=100, max_depth=5, data='my_data'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 443 OUT768 |
| ✓ | SimpleStockPriceByDateBFCL/simple_python_142 (get_stock_price): company_name Amazon/AMZN, date=2022-03-11, exchange NASDAQ or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 445 OUT757 |
| ✓ | SimpleTTestStatsBFCL/simple_python_118 (stats.t_test): array_1=[10,15,12,14,11], array_2=[18,16,17,20,22], alpha=0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 559 OUT1 780 |
Tool Calling - Simple: Stats, Finance & ML (set 2)
6 passed2 failed |
75.0% |
IN53 922 OUT15 046 | |
| ✓ | SimpleCalculateSpeedBFCL/simple_python_101 (calculate_speed): distance=450, time=20, to_unit=km/h.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 717 OUT1 250 |
| ✓ | SimpleCardProbabilitySingleBFCL/simple_python_116 (probabilities.calculate_single): total_outcomes=52, event_outcomes=4 (king), round omitted or 2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 705 OUT751 |
| × | SimpleCompoundInterestFreqBFCL/simple_python_136 (compound_interest): principal=10000, annual_rate=5.0, compounding_freq=monthly, time_in_years=5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.annual_rate == 5.0
Values:
args.annual_rate = 0.05 |
Failed | IN6 745 OUT2 611 |
| ✓ | SimpleDiscountedCashFlowBFCL/simple_python_129 (calculate_discounted_cash_flow): coupon_payment=100, period=5, discount_rate=0.04, face_value omitted or 1000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 737 OUT1 544 |
| ✓ | SimpleHousePriceBedroomsBFCL/simple_python_110 (predict_house_price): bedrooms=3, bathrooms=2, area=1800, location San Francisco.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 738 OUT3 203 |
| × | SimpleStockPriceDaysBFCL/simple_python_143 (get_stock_price): company=AAPL, days=5, exchange NASDAQ or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.company?.ToLower()).AnyOf(["aapl"]) |
Failed | IN6 697 OUT1 390 |
| ✓ | SimpleStockReturnProjectedBFCL/simple_python_137 (calculate_stock_return): investment_amount=5000, annual_growth_rate=0.06, holding_period=5, dividends omitted or false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 750 OUT3 084 |
| ✓ | SimpleTTestIndBFCL/simple_python_119 (hypothesis_testing.ttest_ind): sample1=[22,33,42,12,34], sample2=[23,45,44,14,38], significance_level=0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 833 OUT1 213 |
Tool Calling - Simple: Stats, Finance & ML (set 3)
8 passed |
100.0% |
IN52 210 OUT11 743 | |
| ✓ | SimpleCompoundedInterestAnnuallyBFCL/simple_python_145 (calculate_compounded_interest): principal=5000, interest_rate=0.05, period=10, compounding_frequency omitted or Annually.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 523 OUT1 527 |
| ✓ | SimpleDistanceEarthMoonBFCL/simple_python_102 (calculate_distance): body1=Earth, body2=Moon, unit mi/miles/mile.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 479 OUT748 |
| ✓ | SimpleHeartRatioProbabilityBFCL/simple_python_117 (probability_of_event): success_outcomes=13 (hearts), total_outcomes=52, format_as_ratio=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 511 OUT705 |
| ✓ | SimpleHousePriceRoomsBFCL/simple_python_125 (predict_house_price): area=2500, rooms=5, year=1990, location San Francisco/SF.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 538 OUT2 417 |
| ✓ | SimpleMutualFundReturnBFCL/simple_python_139 (estimate_mutual_fund_return): yearly_yield=5.0, investment_amount=2000, years=3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 539 OUT1 133 |
| ✓ | SimpleReturnOnEquityBFCL/simple_python_132 (calculate_return_on_equity): net_income=2000000, shareholder_equity=10000000, dividends_paid=200000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 581 OUT1 581 |
| ✓ | SimpleStockPriceCurrentMultiBFCL/simple_python_149 (get_stock_price): company_names=[Apple, Microsoft] (or tickers AAPL, MSFT).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 461 OUT1 256 |
| ✓ | SimpleTwoSampleTTestRunBFCL/simple_python_120 (run_two_sample_ttest): group1=[3,4,5,6,4], group2=[7,8,9,8,7], equal_variance=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 578 OUT2 376 |
Tool Calling - Simple: Stats, Finance & ML (set 4)
8 passed |
100.0% |
IN52 865 OUT10 353 | |
| ✓ | SimpleBinomialProbDistBFCL/simple_python_114 (prob_dist.binomial): trials=10, successes=5, p omitted or 0.5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 574 OUT1 572 |
| ✓ | SimpleFutureValueMonthlyBFCL/simple_python_133 (finance.predict_future_value): present_value=5000, annual_interest_rate=0.05, compounding_periods_per_year=12, time_years=3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 616 OUT1 496 |
| ✓ | SimpleGeometryAreaTriangleBFCL/simple_python_104 (geometry.area_triangle): base=6, height=10, unit omitted or 'square meters'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 557 OUT796 |
| ✓ | SimpleMathPowerBFCL/simple_python_105 (math.power): base=3, exponent=4, mod omitted or 1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 550 OUT581 |
| ✓ | SimpleNPVCashFlowsBFCL/simple_python_127 (calculate_NPV): cash_flows=[200,300,400,500], discount_rate=0.1, initial_investment=2000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 665 OUT2 682 |
| ✓ | SimpleReturnOnInvestmentBFCL/simple_python_135 (calculate_return_on_investment): purchase_price=20, sale_price=25, dividend=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 593 OUT1 027 |
| ✓ | SimpleStockPriceDataTypeBFCL/simple_python_146 (stock_price): company Amazon/AMZN, days=3, data_type omitted or Close.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 551 OUT690 |
| ✓ | SimpleTwoSampleTTestFloatBFCL/simple_python_123 (hypothesis_testing.two_sample_t_test): group1=[12.4,15.6,11.2,18.9], group2=[10.5,9.8,15.2,13.8], alpha omitted or 0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 759 OUT1 509 |
Tool Calling - Simple: Stats, Finance & ML (set 5)
8 passed |
100.0% |
IN53 401 OUT13 232 | |
| ✓ | SimpleAreaUnderCurvePolyBFCL/simple_python_103 (mathematics.calculate_area_under_curve): polynomial=[3.0,2.0,-4.0], limits=[-1.0,2.0].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 676 OUT1 861 |
| ✓ | SimpleBinomialCalcBFCL/simple_python_115 (calculate_binomial_probability): number_of_trials=8, number_of_successes=5, probability_of_success omitted or 0.5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 644 OUT1 028 |
| ✓ | SimpleCalculateBmiBFCL/simple_python_107 (calculate_bmi): weight=70, height=175, system omitted or metric.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 643 OUT788 |
| ✓ | SimpleFutureValueInvestmentBFCL/simple_python_148 (finance.calculate_future_value): initial_investment=20000, rate_of_return=0.08, years=5, contribution omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 683 OUT1 519 |
| ✓ | SimpleLinearRegressionBFCL/simple_python_108 (run_linear_regression): predictors=[Age,Income,Education], target=Purchase_Amount, standardize=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 669 OUT841 |
| ✓ | SimpleNPVInvestmentBFCL/simple_python_130 (finance_calculator.npv): cash_flows=[-50000,10000,15000,20000,25000,30000], discount_rate=0.08, years omitted or empty.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 767 OUT2 605 |
| ✓ | SimpleStockPricesDurationBFCL/simple_python_147 (get_stock_prices): companies=[Microsoft,Google], duration='2 weeks'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 623 OUT1 668 |
| ✓ | SimpleTTestDatasetsBFCL/simple_python_124 (t_test): dataset_A=[12,24,36], dataset_B=[15,30,45], alpha omitted or 0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 696 OUT2 922 |
Tool Calling - Simple: Stats, Finance & ML (set 6)
10 passed |
100.0% |
IN71 473 OUT10 413 | |
| ✓ | SimpleBinomialCoin100BFCL/simple_python_121 (calc_binomial_prob): num_trials=100, num_success=60, prob_success=0.5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 153 OUT1 165 |
| ✓ | SimpleCagrBFCL/simple_python_140 (calculate_cagr): initial_value=2000, final_value=3000, period_in_years=4.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 172 OUT984 |
| ✓ | SimpleChiSquaredTestBFCL/simple_python_122 (chi_squared_test): table=[[10,20],[30,40]], alpha omitted or 0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 156 OUT2 332 |
| ✓ | SimpleDiceRollBFCL/simple_python_113 (probability.dice_roll): desired_number=6, number_of_rolls=2, die_sides omitted or 6.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 130 OUT659 |
| ✓ | SimpleMarketPerformanceBFCL/simple_python_144 (market_performance.get_data): indexes=[S&P 500, Dow Jones], days=5, detailed omitted or true/false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 137 OUT1 253 |
| ✓ | SimpleMetalPriceBFCL/simple_python_141 (get_metal_price): metal=Gold, measure=ounce.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 096 OUT898 |
| ✓ | SimpleNormalVariateBFCL/simple_python_111 (random.normalvariate): mu=0, sigma=1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 117 OUT628 |
| ✓ | SimplePortfolioFutureValueBFCL/simple_python_138 (portfolio_future_value): stock=X, invested_amount=5000, expected_annual_return=0.05, years=7.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 175 OUT933 |
| ✓ | SimpleRandomForestClassifierBFCL/simple_python_106 (train_random_forest_classifier): dataset=your_dataset_name, max_depth=5, n_estimators=100.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 156 OUT564 |
| ✓ | SimpleRSquaredBFCL/simple_python_126 (linear_regression.get_r_squared): dataset_path=C:/data/cars.csv, independent_variables=[engine_size,fuel_economy], dependent_variable=car_price.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 181 OUT997 |
Tool Calling - Tool Selection: Area (Triangle/Circle)
1 passed |
100.0% |
IN4 535 OUT1 864 | |
| ✓ | SelectTriangleAreaHeronToolBFCL/multiple_1 (math.triangle_area_heron): triangle area from three side lengths (Heron).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 535 OUT1 864 |
Tool Calling - Tool Selection: Art & Music
9 passed |
100.0% |
IN105 603 OUT11 091 | |
| ✓ | SelectAnalyzeStructureToolBFCL/multiple_162 (analyze_structure): dynamic structural analysis of specific floors.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 764 OUT853 |
| ✓ | SelectConcertSearchToolBFCL/multiple_166 (concert.search): concert search by genre/location/date/price.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 730 OUT1 055 |
| ✓ | SelectGuitarPriceToolBFCL/multiple_165 (guitar_price.find): guitar price by model/condition/location (not instrument_price.get).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 733 OUT1 215 |
| ✓ | SelectInstrumentPriceToolBFCL/multiple_164 (instrument_price.get): instrument price by brand/model/finish (not guitar_price.find).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 738 OUT1 893 |
| ✓ | SelectMelodyGeneratorToolBFCL/multiple_167 (music_generator.generate_melody): melody generation with key/note/length/tempo. Tightened per methodology: prompt explicitly gives "120 beats per minute", so tempo is required exactly 120 (ground truth also accepts omission, which equals the schema default).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 767 OUT501 |
| ✓ | SelectMetropolitanArtworksToolBFCL/multiple_163 (metropolitan_museum.get_top_artworks): top artworks sorted by popularity.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 732 OUT1 995 |
| ✓ | SelectMusicalScaleToolBFCL/multiple_169 (musical_scale): musical scale for a key.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 697 OUT1 780 |
| ✓ | SelectSculptureInfoToolBFCL/multiple_160 (get_sculpture_info): sculpture info by artist with detail.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 717 OUT937 |
| ✓ | SelectSongLyricsToolBFCL/multiple_168 (get_song_lyrics): lyrics by song title/artist.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 725 OUT862 |
Tool Calling - Tool Selection: Calculators & Directories
8 passed |
100.0% |
IN76 781 OUT8 344 | |
| ✓ | SelectCalculateGcdToolBFCL/multiple_35 (calculate_gcd): greatest common divisor, not lcm.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 591 OUT726 |
| ✓ | SelectCurrencyGetRateToolBFCL/multiple_17 (currency_conversion.get_rate): rate at a specific date, not convert/latest-rate.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 610 OUT1 407 |
| ✓ | SelectGeodistanceToolBFCL/multiple_65 (geodistance.find): distance between cities, not timezone difference/flights.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 580 OUT896 |
| ✓ | SelectLibrarySearchBookToolBFCL/multiple_38 (library.search_book): search for a book, not reserve one.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 592 OUT896 |
| ✓ | SelectLinearRegressionToolBFCL/multiple_53 (linear_regression): linear regression, not random forest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 607 OUT1 143 |
| ✓ | SelectMagneticFieldCalculateToolBFCL/multiple_41 (magnetic_field.calculate): magnetic field via Ampere's law, not electric field/force.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 623 OUT1 092 |
| ✓ | SelectSolveQuadraticEquationToolBFCL/multiple_28 (solve.quadratic_equation): solve a quadratic, not rgb/string-reverse.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 599 OUT1 000 |
| ✓ | SelectStockForecastToolBFCL/multiple_55 (stock_forecast): stock price prediction, not weather forecast.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 579 OUT1 184 |
Tool Calling - Tool Selection: Finance & Legal
7 passed3 failed |
70.0% |
IN151 869 OUT13 601 | |
| ✓ | SelectCagrCalculationToolBFCL/multiple_130 (calculate_cagr): compound annual growth rate from initial/final/years, not other finance tools.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 172 OUT1 176 |
| ✓ | SelectCaseInfoToolBFCL/multiple_135 (get_case_info): retrieve case info (victim) by docket/court.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 148 OUT1 559 |
| × | SelectCrimeRateToolBFCL/multiple_136 (get_crime_rate): crime rate by city/state/type/year.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.type.ToLower() == "violent" |
Failed | IN13 140 OUT1 508 |
| ✓ | SelectCrimeRecordToolBFCL/multiple_134 (crime_record.get_record): detailed crime record by case number/county.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 140 OUT1 163 |
| ✓ | SelectFutureValueToolBFCL/multiple_132 (finance.calculate_future_value): future value from rate/initial/years, not compound_interest.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 175 OUT1 230 |
| × | SelectLawsuitDetailsToolBFCL/multiple_139 (lawsuit_details.find): patent lawsuit details by company/year/type.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 3 |
Failed | IN26 772 OUT1 832 |
| × | SelectLawsuitSearchToolBFCL/multiple_137 (lawsuit_search): lawsuit search by company/date/location/status. Tightened per methodology: prompt explicitly asks for "ongoing" lawsuits, so status is required exactly "ongoing" (ground truth also accepts omission, but an explicitly-stated value must be passed).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN19 913 OUT1 513 |
| ✓ | SelectLegalCaseFetchToolBFCL/multiple_138 (legal_case.fetch): fetch detailed legal case by id.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 112 OUT1 513 |
| ✓ | SelectMarketPerformanceToolBFCL/multiple_131 (market_performance.get_data): index performance over N days.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 137 OUT1 013 |
| ✓ | SelectMutualFundBalanceToolBFCL/multiple_133 (calculate_mutual_fund_balance): mutual fund balance from investment/yield/years.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 160 OUT1 094 |
Tool Calling - Tool Selection: Finance & Markets
8 passed |
100.0% |
IN82 450 OUT10 883 | |
| ✓ | SelectAvgClosingPriceToolBFCL/multiple_56 (avg_closing_price): average closing price, not volume/total revenue.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 275 OUT755 |
| ✓ | SelectCompoundInterestCalcToolBFCL/multiple_49 (calc_Compound_Interest): compound interest, not future value/simple interest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 324 OUT1 227 |
| ✓ | SelectCurrencyConversionToolBFCL/multiple_52 (currency_conversion): currency conversion, not unit conversion.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 285 OUT2 204 |
| ✓ | SelectDividendDataToolBFCL/multiple_54 (corporate_finance.dividend_data): dividend history, not generic stock market data.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 276 OUT1 555 |
| ✓ | SelectHousePriceForecastToolBFCL/multiple_50 (house_price_forecast): house price forecast, not weather/stock forecast.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 269 OUT895 |
| ✓ | SelectPropertyDepreciationToolBFCL/multiple_14 (finance.property_depreciation): depreciation, not loan/inflation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 324 OUT1 882 |
| ✓ | SelectRevenueForecastToolBFCL/multiple_13 (corporate_finance.revenue_forecast): revenue forecast, not product price.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 316 OUT1 197 |
| ✓ | SelectYoyGrowthRateToolBFCL/multiple_12 (corporate_finance.calculate_YOY_growth_rate): YOY growth, not ROE/ROA.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 381 OUT1 168 |
Tool Calling - Tool Selection: Food & Travel
7 passed |
100.0% |
IN64 556 OUT7 219 | |
| ✓ | SelectCurrencyConvertToolBFCL/multiple_95 (currency_exchange.convert): currency conversion, not unit_conversion. live_conversion is required true: prompt says "using current exchange rate" (ground truth = [true]).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 223 OUT1 476 |
| ✓ | SelectDrivingDistanceToolBFCL/multiple_104 (get_shortest_driving_distance): driving distance, not instrument price / biology / scientist lookup.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 212 OUT933 |
| ✓ | SelectHotelBookToolBFCL/multiple_93 (hotel.book): book a hotel room, not car.rental.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 241 OUT467 |
| ✓ | SelectRecipeFindToolBFCL/multiple_91 (recipe.find): recipe by main ingredient + limit, not restaurant.find.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 210 OUT740 |
| ✓ | SelectRecipeSearchToolBFCL/multiple_89 (recipe_search): find a recipe from ingredients, not restaurant_search/ingredient_replace.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 229 OUT805 |
| ✓ | SelectRestaurantForGroupToolBFCL/multiple_90 (restaurant.find_group): restaurant for a group, not events.find_event.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 220 OUT612 |
| ✓ | SelectWalmartVeganToolBFCL/multiple_92 (walmart.vegan_products): the named store is Walmart, not Safeway/Whole Foods.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 221 OUT2 186 |
Tool Calling - Tool Selection: Games, Food & Utilities
10 passed |
100.0% |
IN137 973 OUT11 053 | |
| ✓ | SelectBestGroceryStoreToolBFCL/multiple_188 (grocery_store.find_best): best grocery store by rating/products.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 824 OUT1 043 |
| ✓ | SelectClosestRestaurantToolBFCL/multiple_185 (restaurant_search.find_closest): closest restaurant by cuisine/amenities.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 789 OUT1 070 |
| ✓ | SelectFindRecipeToolBFCL/multiple_186 (find_recipe): recipe by restrictions/type/time.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 802 OUT1 242 |
| ✓ | SelectGamePlayerStatsToolBFCL/multiple_180 (game_stats.fetch_player_statistics): player stats by game/username/platform.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 794 OUT839 |
| ✓ | SelectMultiplayerGameFinderToolBFCL/multiple_182 (multiplayer_game_finder): multiplayer game by platform/rating.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 798 OUT1 260 |
| ✓ | SelectRecipeCaloriesToolBFCL/multiple_183 (recipe_info.get_calories): calories of a recipe from a website.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 788 OUT1 335 |
| ✓ | SelectRecipeSearchToolBFCL/multiple_184 (recipe_search): recipe by restriction/ingredients/servings.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 800 OUT1 552 |
| ✓ | SelectSoccerLastMatchToolBFCL/multiple_181 (soccer.get_last_match): last match with stats for a club.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 793 OUT1 255 |
| ✓ | SelectTimezoneConvertToolBFCL/multiple_189 (timezone.convert): convert a time between timezones.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 794 OUT926 |
| ✓ | SelectWholeFoodsPriceToolBFCL/multiple_187 (whole_foods.check_price): price of items at a Whole Foods location.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 791 OUT531 |
Tool Calling - Tool Selection: Geometry, Finance & Services
7 passed |
100.0% |
IN58 975 OUT7 735 | |
| ✓ | SelectAirQualityForecastToolBFCL/multiple_63 (air_quality_forecast): air quality, not weather/news.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 409 OUT1 371 |
| ✓ | SelectFinancialCompoundInterestToolBFCL/multiple_57 (financial.compound_interest): compound interest, not simple interest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 472 OUT1 336 |
| ✓ | SelectGeometryRectangleToolBFCL/multiple_31 (geometry_rectangle.calculate): rectangle area+perimeter, not square/circle.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 425 OUT575 |
| ✓ | SelectLawyerFinderToolBFCL/multiple_59 (lawyer_finder): find a lawyer by specialization, not rent a car.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 406 OUT1 038 |
| ✓ | SelectLibrarySearchBooksToolBFCL/multiple_68 (library.search_books): search a specific library, not Google Books/Open Library.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 409 OUT312 |
| ✓ | SelectMapsShortestPathToolBFCL/multiple_27 (maps.shortest_path): shortest path with transport mode, not route-time estimation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 433 OUT1 726 |
| ✓ | SelectTrafficEstimateToolBFCL/multiple_66 (traffic_estimate): traffic for a time period, not raw distance/weather.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 421 OUT1 377 |
Tool Calling - Tool Selection: History & Knowledge
9 passed1 failed |
90.0% |
IN142 973 OUT15 172 | |
| ✓ | SelectDiscovererToolBFCL/multiple_155 (get_discoverer): discoverer of a discovery, with detail.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 576 OUT1 030 |
| × | SelectEarliestReferenceToolBFCL/multiple_157 (get_earliest_reference): earliest reference of a person from a source.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN20 530 OUT3 484 |
| ✓ | SelectEventDateToolBFCL/multiple_153 (get_event_date): date of a historical event.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 571 OUT794 |
| ✓ | SelectHistoricalContribToolBFCL/multiple_156 (historical_contrib.get_contrib): scientist's contribution on a date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 608 OUT1 076 |
| ✓ | SelectHistoryKeyEventsToolBFCL/multiple_152 (history.get_key_events): key war events by country/years.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 621 OUT2 094 |
| ✓ | SelectMostFollowedToolBFCL/multiple_151 (social_media_analytics.most_followed): most-followed by topic/sub-topics.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 610 OUT1 615 |
| ✓ | SelectNeuronalActivityToolBFCL/multiple_150 (calculate_neuronal_activity): neuronal activity from synaptic rate/weight/decay.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 632 OUT885 |
| ✓ | SelectPaintNeededToolBFCL/multiple_159 (calculate_paint_needed): paint quantity for a wall.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 639 OUT672 |
| ✓ | SelectPapalBiographyToolBFCL/multiple_158 (religious_history.get_papal_biography): papal biography with contributions.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 589 OUT2 916 |
| ✓ | SelectUsPresidentToolBFCL/multiple_154 (US_president.in_year): president in a given year. Tightened per methodology: prompt explicitly asks for the "full name", so full_name is required true (ground truth also accepts omission, but an explicitly-requested option must be passed).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 597 OUT606 |
Tool Calling - Tool Selection: History, Demographics & Art
8 passed2 skipped |
100.0% |
IN85 985 OUT7 018 | |
| ✓ | SelectArtworkPriceToolBFCL/multiple_74 (art_auction.fetch_artwork_price): artwork price, not library book search. Tightened per methodology: prompt explicitly asks "on all auction platforms", so require platform == "all" (ground truth also accepts omission, but an explicitly-given value must pass).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 764 OUT969 |
| ✓ | SelectArtworkSearchToolBFCL/multiple_77 (artwork_search.find): artwork search, not tourist_attraction/park search.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 747 OUT928 |
| – | SelectCustomSculptureToolBFCL/multiple_76 (sculpture.create_custom): custom sculpture (material), not painting (color).0 AI queries
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Skipped: Ambiguous action-vs-question item (item flaw, not model behavior): the prompt mixes an order ('I want to order a custom bronze sculpture of a horse') with a question ('What material options are available?'), so a competent model may either call sculpture_create_custom or answer the question in text - not deterministic. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | SelectEuropeanMonarchsToolBFCL/multiple_70 (european_history.get_monarchs): monarch lookup, not events/culture.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 748 OUT1 305 |
| ✓ | SelectExhibitionInfoToolBFCL/multiple_79 (exhibition_info): museum exhibitions, not restaurant_info.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 754 OUT648 |
| – | SelectMuseumInfoToolBFCL/multiple_78 (museum_info): museum info with a feature list, not generic tourist_spot_info.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Ambiguous single-call item (item flaw, not model behavior): a dedicated exhibition_info tool coexists with museum_info, so the 'exhibitions' part can reasonably route either to museum_info's features or to a separate exhibition_info call; a competent model may make one or two calls - not deterministic. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | SelectPaintColorTrendsToolBFCL/multiple_75 (paint_color.trends): paint color trend, not weather/house-price trends.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 732 OUT871 |
| ✓ | SelectPopulationByCategoryToolBFCL/multiple_71 (get_population): population by category, not bureau statistics/demographics.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 756 OUT749 |
| ✓ | SelectReligionOriginToolBFCL/multiple_73 (religion.get_origin): founder/origin, not core beliefs.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 737 OUT747 |
| ✓ | SelectStatePopulationToolBFCL/multiple_72 (us_history.population_by_state_year): state population, not GDP.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 747 OUT801 |
Tool Calling - Tool Selection: Isolated Collisions: Exhibition, Hotel & Currency
3 passed |
100.0% |
IN21 028 OUT5 703 | |
| ✓ | SelectBookLuxuryHotelToolBFCL/multiple_191 (book_hotel): luxury room with view, stay_duration variant.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 061 OUT481 |
| ✓ | SelectCurrencyConversionToolBFCL/multiple_192 (currency_conversion.convert): convert an amount between currencies.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 977 OUT1 444 |
| ✓ | SelectExhibitionToolBFCL/multiple_161 (find_exhibition): top-rated sculpture exhibition by location/art_form.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 990 OUT3 778 |
Tool Calling - Tool Selection: Knowledge & Media
8 passed2 failed |
80.0% |
IN129 775 OUT15 140 | |
| ✓ | SelectBattleDetailsToolBFCL/multiple_18 (european_history.battle_details): battle details, not war/leader info.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 747 OUT678 |
| ✓ | SelectDatabaseModifyColumnsToolBFCL/multiple_10 (database.modify_columns): delete columns, not create a backup.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 797 OUT1 091 |
| ✓ | SelectFiveFactorModelToolBFCL/multiple_69 (five_factor_model.analyse): Big Five personality, not MBTI.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 818 OUT1 625 |
| × | SelectGameRewardsToolBFCL/multiple_26 (game_rewards.get): rewards, not scores/missions list.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 771 OUT3 747 |
| ✓ | SelectGenerateSoundWaveToolBFCL/multiple_21 (generate_sound_wave): generate a sound, not play a file.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 771 OUT973 |
| ✓ | SelectMostPointsSingleGameToolBFCL/multiple_22 (sports_data.basketball.most_points_single_game): single-game record, not season/career.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 747 OUT713 |
| ✓ | SelectPlayerStatsToolBFCL/multiple_23 (basketball.player_stats.get): player stats, not game/team stats.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 769 OUT1 105 |
| ✓ | SelectReligionSchismsToolBFCL/multiple_19 (religion_history.get_schisms): schisms, not councils/reformations.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 739 OUT1 914 |
| ✓ | SelectSculpturePriceToolBFCL/multiple_20 (sculpture_price.calculate): commission price, not sculptor info/availability.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 753 OUT1 369 |
| × | SelectVideoGameStorePriceToolBFCL/multiple_25 (video_games.store_price): store price, not store currency/on-sale.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 863 OUT1 925 |
Tool Calling - Tool Selection: Legal, Environment & Services
10 passed |
100.0% |
IN121 607 OUT10 191 | |
| ✓ | SelectAirQualityToolBFCL/multiple_144 (air_quality): AQI by location/date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 151 OUT1 153 |
| ✓ | SelectEmissionsToolBFCL/multiple_145 (calculate_emissions): annual CO2 from distance/fuel/efficiency.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 186 OUT1 091 |
| ✓ | SelectHumidityForecastToolBFCL/multiple_142 (weather.humidity_forecast): humidity forecast by location/days.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 138 OUT993 |
| ✓ | SelectLawsuitCheckCaseToolBFCL/multiple_141 (lawsuit.check_case): verify a case's closed status by id.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 157 OUT973 |
| ✓ | SelectLawsuitsSearchToolBFCL/multiple_140 (lawsuits_search): lawsuits by company/location/year.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 152 OUT1 180 |
| ✓ | SelectMapDirectionsToolBFCL/multiple_147 (map_service.get_directions): directions avoiding features.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN12 153 OUT1 300 |
| ✓ | SelectRestaurantFindNearbyToolBFCL/multiple_146 (restaurant.find_nearby): nearby restaurants by cuisine/distance.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 146 OUT830 |
| ✓ | SelectSentimentAnalysisToolBFCL/multiple_149 (sentiment_analysis): sentiment of a review text.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 158 OUT905 |
| ✓ | SelectSlopeGradientToolBFCL/multiple_143 (calculate_slope_gradient): slope gradient between two coordinates. Tightened per methodology: prompt explicitly asks for the gradient "in degree", so unit is required exactly "degree" (ground truth also accepts omission, which equals the schema default).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN12 239 OUT1 153 |
| ✓ | SelectStockInfoToolBFCL/multiple_148 (get_stock_info): detailed stock information by company.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 127 OUT613 |
Tool Calling - Tool Selection: Local Services & Weather
9 passed |
100.0% |
IN86 766 OUT11 365 | |
| ✓ | SelectHumidityTemperatureForecastToolBFCL/multiple_60 (humidity_temperature_forecast): humidity/temperature, not events/movies.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 632 OUT1 165 |
| ✓ | SelectLandscapeArchitectToolBFCL/multiple_61 (landscape_architect.find_specialty): landscape architect, not home renovation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 649 OUT1 469 |
| ✓ | SelectLawyerSearchToolBFCL/multiple_58 (lawyer.search): find a lawyer, not a doctor.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 618 OUT1 091 |
| ✓ | SelectLibraryFindNearbyToolBFCL/multiple_48 (library.find_nearby): find a library, not a store.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 643 OUT1 826 |
| ✓ | SelectNatureParkToolBFCL/multiple_62 (nature_park.find_nearby): find a nature park, not a restaurant.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 645 OUT1 572 |
| ✓ | SelectRideHailingToolBFCL/multiple_39 (ride_hailing.get_rides): find a ride, not a grocery delivery.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 641 OUT626 |
| ✓ | SelectRoutePlannerToolBFCL/multiple_24 (route_planner.calculate_route): plan a route, not chess-club lookup.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 657 OUT960 |
| ✓ | SelectTranslateToolBFCL/multiple_67 (translate): translate text, not sentiment/word-count.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 634 OUT1 464 |
| ✓ | SelectUvIndexToolBFCL/multiple_64 (uv_index.get_future): UV index, not rainfall/snowfall.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 647 OUT1 192 |
Tool Calling - Tool Selection: Mathematics
8 passed |
100.0% |
IN75 269 OUT8 553 | |
| ✓ | SelectCalculateIntegralToolBFCL/multiple_33 (calculate_integral): definite integral, not derivative.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 413 OUT890 |
| ✓ | SelectCosineSimilarityToolBFCL/multiple_47 (cosine_similarity.calculate): cosine similarity, not correlation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 432 OUT2 653 |
| ✓ | SelectDiceRollProbabilityToolBFCL/multiple_51 (dice_roll_probability): dice sum probability, not coin/card.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 407 OUT781 |
| ✓ | SelectFunctionsIntersectToolBFCL/multiple_29 (functions.intersect): intersection of two functions, not zeros of one.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 409 OUT1 182 |
| ✓ | SelectMathLcmToolBFCL/multiple_34 (math.lcm): least common multiple, not gcd/sqrt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 399 OUT627 |
| ✓ | SelectQuadraticRootsToolBFCL/multiple_11 (math_roots.quadratic): quadratic roots from coefficients, not cubic/polynomial.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 411 OUT1 202 |
| ✓ | SelectRectangleAreaToolBFCL/multiple_30 (rectangle.area): rectangle area, not circle/triangle area.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 397 OUT582 |
| ✓ | SelectSortListToolBFCL/multiple_46 (sort_list): sort a list, not filter/sum.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 401 OUT636 |
Tool Calling - Tool Selection: Mathematics & Geometry
7 passed |
100.0% |
IN70 323 OUT5 080 | |
| ✓ | SelectCircleAreaToolBFCL/multiple_97 (geometry.area_circle): circle area, not plot_sine_wave.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 026 OUT731 |
| ✓ | SelectCircumferenceToolBFCL/multiple_98 (geometry.circumference): circle circumference, not the reference/time/melody tools.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 017 OUT631 |
| ✓ | SelectDatabaseQueryToolBFCL/multiple_119 (database.query): structured DB query with conditions, not the score/ROI/probability distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN10 108 OUT658 |
| ✓ | SelectDerivativeToolBFCL/multiple_99 (calculus.derivative): derivative at a point, not get_personality_traits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 050 OUT766 |
| ✓ | SelectHcfToolBFCL/multiple_100 (math.hcf): highest common factor, not the scale-progression distractor.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 036 OUT625 |
| ✓ | SelectSolveQuadraticToolBFCL/multiple_96 (solve_quadratic_equation): solve quadratic, not stock/portfolio tools.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 049 OUT1 047 |
| ✓ | SelectTriangleAreaToolBFCL/multiple_121 (geometry.area_triangle): triangle area, not map directions / invention lookup.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 037 OUT622 |
Tool Calling - Tool Selection: Mixed Domains
7 passed1 skipped |
100.0% |
IN68 830 OUT9 902 | |
| ✓ | SelectAverageGradeToolBFCL/multiple_9 (calculate_average): selects average, not standard_deviation/highest_grade.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 853 OUT1 039 |
| ✓ | SelectCapacitanceToolBFCL/multiple_6 (capacitance_calculator.calculate): selects capacitance, not resistance/magnetic_field. Tightened per methodology: prompt explicitly states "dielectric constant K is 1.0", so K is required exactly 1.0 (ground truth also accepts omission, but an explicitly-given value must be passed).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 855 OUT2 100 |
| ✓ | SelectCapitalToolBFCL/multiple_2 (country_info.capital): selects capital lookup, not largest_city/population.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 754 OUT751 |
| ✓ | SelectDisplacementToolBFCL/multiple_4 (kinematics.calculate_displacement): selects displacement, not final_speed.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 847 OUT851 |
| ✓ | SelectEuclideanDistanceToolBFCL/multiple_3 (EuclideanDistance.calculate): selects distance, not angleToXAxis.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 799 OUT1 759 |
| ✓ | SelectRealEstateSearchToolBFCL/multiple_8 (realestate.find_properties): property search with budget, not property_valuation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 858 OUT1 931 |
| ✓ | SelectWeatherByCoordinatesDateToolBFCL/multiple_5 (weather.get_by_coordinates_date): historical-by-coordinates, not by-city/forecast.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 864 OUT1 471 |
| – | SelectWildlifeGrowthToolBFCL/multiple_7 (wildlife_population.assess_growth): population growth, not ecological_impact.0 AI queries
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Skipped: Ambiguous single-call item (item flaw, not model behavior): the prompt asks for BOTH population growth AND impact on woodland, and two distinct tools genuinely fit (wildlife_population_assess_growth + ecological_impact_analyze); a competent human would reasonably make two calls, so a single-call ground truth is unfair. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Tool Selection: Music, Sports & Games
8 passed1 failed |
88.9% |
IN95 639 OUT11 428 | |
| ✓ | SelectBoardGameRecommendToolBFCL/multiple_86 (BoardGameGeek.recommend): BoardGameGeek store, not AmazonGameStore.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 019 OUT1 447 |
| ✓ | SelectConcertTicketToolBFCL/multiple_81 (concert.book_ticket): concert ticket, not festival ticket.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 042 OUT1 162 |
| ✓ | SelectGameUpdateToolBFCL/multiple_87 (games.update.find): game update/patch, not reviews/price.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 030 OUT1 231 |
| ✓ | SelectMusicGenerateToolBFCL/multiple_82 (music.generate): musical piece from key/tempo, not raw audio.generate.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 008 OUT771 |
| × | SelectMusicShopToolBFCL/multiple_80 (music_shop.find_nearby): music shop with lessons, not gym.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN15 453 OUT1 510 |
| ✓ | SelectPlayerCountToolBFCL/multiple_88 (video_games.get_player_count): active player count, not sales.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 027 OUT1 720 |
| ✓ | SelectPlayerGoalsToolBFCL/multiple_83 (player_stats.get_all_time_goals): per-player goals, not team/league top scorer.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 008 OUT1 042 |
| ✓ | SelectSoccerScoresToolBFCL/multiple_85 (soccer_scores.get_scores): soccer scores, not basketball.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 024 OUT650 |
| ✓ | SelectTopGoalScorersToolBFCL/multiple_84 (getTopGoalScorers): top goal scorers, not top assists.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 028 OUT1 895 |
Tool Calling - Tool Selection: Physical Sciences
10 passed |
100.0% |
IN113 073 OUT12 964 | |
| ✓ | SelectBiologicalEnergyToolBFCL/multiple_43 (biological.calc_energy): energy from moles of substance, not biomass/work.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 291 OUT810 |
| ✓ | SelectBiotSavartLawToolBFCL/multiple_40 (electromagnetism.biot_savart_law): magnetic field via Biot-Savart, not Ampere's law.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 294 OUT1 111 |
| ✓ | SelectConeVolumeToolBFCL/multiple_32 (geometry.calculate_cone_volume): cone volume, not cone mass.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 274 OUT1 615 |
| ✓ | SelectFinalTemperatureToolBFCL/multiple_42 (calculate_final_temperature): mixing two gas samples, not mass calculation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 335 OUT1 037 |
| ✓ | SelectFinalVelocityToolBFCL/multiple_37 (kinematics.final_velocity): final velocity from initial velocity + time, not wave velocity/distance.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 284 OUT1 029 |
| ✓ | SelectGeologyEraToolBFCL/multiple_45 (geology.get_era): geological era date, not historical event date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 267 OUT1 222 |
| ✓ | SelectPopulationGeneticsNeToolBFCL/multiple_16 (population_genetics.calculate_ne): effective population size, not range-shift/carrying-capacity.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 334 OUT1 013 |
| ✓ | SelectSolarFarmPotentialToolBFCL/multiple_15 (solarFarm.potential): solar (not wind) energy potential from coordinates + panel area.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 390 OUT3 422 |
| ✓ | SelectSpeedFromRestToolBFCL/multiple_36 (kinematics.calculate_speed_from_rest): speed from rest over a distance, not acceleration.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 319 OUT878 |
| ✓ | SelectWeightInSpaceToolBFCL/multiple_44 (calculate.weight_in_space): weight on a planet, not unit/currency conversion.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 285 OUT827 |
Tool Calling - Tool Selection: Physics & Biology
13 passed |
100.0% |
IN195 030 OUT15 674 | |
| ✓ | SelectCellProteinsToolBFCL/multiple_109 (cellbio.get_proteins): proteins by cell compartment, not find_exhibition / mutation lookup.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 970 OUT2 636 |
| ✓ | SelectDensityToolBFCL/multiple_107 (calculate_density): mass/volume density, not mix_paint_color.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 997 OUT1 294 |
| ✓ | SelectDisplacementToolBFCL/multiple_102 (calculate_displacement): kinematic displacement, not poker/musical distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 015 OUT1 049 |
| ✓ | SelectElectricFieldToolBFCL/multiple_106 (calculate_electric_field_strength): electric field, not paint/cooking/group distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 993 OUT1 516 |
| ✓ | SelectEvolutionPredictionToolBFCL/multiple_114 (prediction.evolution): evolutionary-rate prediction, not modify_painting / calculate_probability.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 005 OUT920 |
| ✓ | SelectFinalSpeedToolBFCL/multiple_103 (calculate_final_speed): free-fall final speed, not chess/court/event distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 985 OUT815 |
| ✓ | SelectFitnessToolBFCL/multiple_113 (calculate_fitness): evolutionary fitness from trait values/contributions, not chess/walmart/lawyer distractors. Percentages map to decimal contributions [0.4, 0.6].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN15 065 OUT999 |
| ✓ | SelectForestGrowthToolBFCL/multiple_112 (forest_growth_forecast): forest growth forecast, not get_scientist_for_discovery. include_human_impact is required true: prompt says "including human impact" (ground truth = [true]).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 998 OUT1 185 |
| ✓ | SelectGenotypeFrequencyToolBFCL/multiple_111 (calculate_genotype_frequency): genotype frequency, not factorial / find_flute.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 000 OUT733 |
| ✓ | SelectHeatCapacityToolBFCL/multiple_108 (calc_heat_capacity): gas heat capacity, not discounted_cash_flow.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 016 OUT1 677 |
| ✓ | SelectLightTravelTimeToolBFCL/multiple_120 (light_travel_time): light travel time, not grocery/hotel/exchange distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN14 983 OUT631 |
| ✓ | SelectMagneticFieldToolBFCL/multiple_105 (calculate_magnetic_field): magnetic field of a loop, not concert/lawsuit distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 999 OUT1 285 |
| ✓ | SelectMutationTypeToolBFCL/multiple_110 (mutation_type.find): SNP mutation type, not get_collectables_in_season.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 004 OUT934 |
Tool Calling - Tool Selection: Sports & Card Games
10 passed |
100.0% |
IN134 189 OUT12 118 | |
| ✓ | SelectChessTopPlayersToolBFCL/multiple_177 (board_game.chess.get_top_players): top chess players by location/rating.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 423 OUT1 055 |
| ✓ | SelectDefenseRankingToolBFCL/multiple_173 (get_defense_ranking): NBA best-defence ranking by season.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 421 OUT1 666 |
| ✓ | SelectFindAthleteToolBFCL/multiple_172 (sports_db.find_athlete): athlete profile by name/sport.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 406 OUT1 598 |
| ✓ | SelectFindCardToolBFCL/multiple_178 (find_card_in_deck): find a card by rank/suit in default deck.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 417 OUT1 055 |
| ✓ | SelectGameWinnerToolBFCL/multiple_171 (game_result.get_winner): winner of a game between two teams on a date.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 444 OUT1 122 |
| ✓ | SelectPokerFullHouseToolBFCL/multiple_179 (poker_probability.full_house): probability of a full house (default deck/hand).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 407 OUT1 502 |
| ✓ | SelectSoccerPlayerStatsToolBFCL/multiple_170 (soccer_stat.get_player_stats): player stats by name/season.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 434 OUT797 |
| ✓ | SelectSportsRankingToolBFCL/multiple_174 (sports_ranking): team ranking in a league.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 404 OUT872 |
| ✓ | SelectTeamScheduleToolBFCL/multiple_176 (sports_team.get_schedule): team schedule for N games in a league.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 428 OUT1 556 |
| ✓ | SelectTopPlayerToolBFCL/multiple_175 (sports_ranking.get_top_player): top player by sport/gender.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 405 OUT895 |
Tool Calling - Tool Selection: Statistics, Finance & Health
5 passed1 failed |
83.3% |
IN56 326 OUT6 342 | |
| ✓ | SelectBmiMetersToolBFCL/multiple_117 (calculate_BMI): BMI in kg/m, not soccer / hotel_booking distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN9 367 OUT805 |
| ✓ | SelectGcdToolBFCL/multiple_101 (math.gcd): greatest common divisor, not get_top_cases.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN9 344 OUT395 |
| × | SelectHotelPricingToolBFCL/multiple_94 (hotel_room_pricing.get): hotel room pricing, not car/flight pricing.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.roomType.ToLower() == "suite with queen size bed" |
Failed | IN9 375 OUT1 145 |
| ✓ | SelectReturnOnEquityToolBFCL/multiple_128 (calculate_return_on_equity): ROE from income/equity/dividends, not library/lyrics/law distractors. dividends_paid is required 200000: prompt explicitly supplies "dividends paid of $200,000".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 445 OUT1 784 |
| ✓ | SelectSingleProbabilityToolBFCL/multiple_124 (probabilities.calculate_single): single-event probability, not math.power / fetch_DNA_sequence.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 353 OUT833 |
| ✓ | SelectTwoSampleTtestToolBFCL/multiple_125 (run_two_sample_ttest): two-group t-test, not restaurant/personality distractors. equal_variance is required true: prompt says "assuming equal variance" (ground truth = [true, ""]).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 442 OUT1 380 |
Tool Calling - Tool Selection: Statistics, Finance & Services
8 passed |
100.0% |
IN91 257 OUT8 501 | |
| ✓ | SelectBmiMetricToolBFCL/multiple_116 (calculate_bmi): BMI in kg/cm with unit option, not celebrity_net_worth.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 395 OUT787 |
| ✓ | SelectCalculateProbabilityToolBFCL/multiple_123 (calculate_probability): probability from favorable/total, not lawsuit/house-price distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 367 OUT671 |
| ✓ | SelectCompoundInterestToolBFCL/multiple_129 (compound_interest): compound interest, not lawsuits_search.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 422 OUT2 321 |
| ✓ | SelectFindRestaurantsToolBFCL/multiple_115 (find_restaurants): restaurant search with count + dietary, not instrument/stock/sports.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 402 OUT789 |
| ✓ | SelectLinearRegressionToolBFCL/multiple_122 (run_linear_regression): regression with predictors, not recipe/itinerary/cooking distractors. standardize is required true: prompt says "Also apply standardization" (ground truth = [true]).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 417 OUT794 |
| ✓ | SelectMoviesByActorToolBFCL/multiple_118 (imdb.find_movies_by_actor): movies by actor+year, not flight.book / lawsuits_search.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 389 OUT972 |
| ✓ | SelectQuarterlyDividendToolBFCL/multiple_127 (finance.calculate_quarterly_dividend_per_share): dividend per share, not lyrics/paint/movie distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN11 421 OUT1 290 |
| ✓ | SelectTTestToolBFCL/multiple_126 (t_test): two-dataset t-test, not event_finder / triangle-area distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 444 OUT877 |
Tool Calling - Tool Selection: Travel & Biology
8 passed |
100.0% |
IN97 927 OUT9 006 | |
| ✓ | SelectBookHotelToolBFCL/multiple_190 (book_hotel): book a room by hotel/location/room_type/start/nights.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN12 281 OUT1 573 |
| ✓ | SelectCellProteinsToolBFCL/multiple_196 (cellbio.get_proteins): proteins in a cell compartment.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 212 OUT869 |
| ✓ | SelectDistanceDurationToolBFCL/multiple_193 (maps.get_distance_duration): distance/duration between two places.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 231 OUT1 525 |
| ✓ | SelectForestGrowthToolBFCL/multiple_199 (forest_growth_forecast): forest growth forecast with human impact.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 234 OUT411 |
| ✓ | SelectGenotypeFrequencyToolBFCL/multiple_198 (calculate_genotype_frequency): genotype frequency from allele frequency.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 242 OUT1 042 |
| ✓ | SelectHeatCapacityToolBFCL/multiple_195 (calc_heat_capacity): heat capacity for a gas by temp/volume. Tightened per methodology: prompt explicitly states the gas is "air", so gas is required exactly "air" (ground truth also accepts omission, which equals the schema default).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 258 OUT785 |
| ✓ | SelectMuseumHoursToolBFCL/multiple_194 (get_museum_hours): museum opening hours on a day.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 223 OUT912 |
| ✓ | SelectMutationTypeToolBFCL/multiple_197 (mutation_type.find): gene mutation type by SNP id.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 246 OUT1 889 |
Tool Calling - Tool Selection: Triangle Properties
1 passed |
100.0% |
IN4 613 OUT2 021 | |
| ✓ | SelectTrianglePropertiesToolBFCL/multiple_0 (triangle_properties.get): retrieves triangle dimensions/properties from 3 sides.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 613 OUT2 021 |