Benchmark summary
A BFCL-based tool-calling benchmark ported to the SC# evaluation runtime. It tests whether the model selects the correct function, uses the right arguments, and avoids calling tools that do not match the request. Each test can see all tools declared in its script, so the model often has more tools to choose from than in the original case.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Tool Calling - Irrelevance: Architecture & Artwork
8 passed1 failed |
88.9% |
IN34 622 OUT4 235 | |
| ✓ | NoToolForApartmentRentCostBFCL/irrelevance_158: apartment rental cost asked; only a new-construction cost estimator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 461 OUT409 |
| ✓ | NoToolForBestJazzSingerBFCL/irrelevance_164: best Jazz singer asked; only a top-instrument-players search (singers are not instrument players) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 456 OUT527 |
| ✓ | NoToolForCathedralCeilingDesignBFCL/irrelevance_157: how to design a cathedral ceiling asked; only a lookup of facts about an existing building is available (retrieves data, does not advise on design) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 457 OUT516 |
| ✓ | NoToolForHolyBookDefinitionBFCL/irrelevance_144: definition of a holy book asked; only a search-within-a-named-holy-book tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 455 OUT143 |
| ✓ | NoToolForLectureTeacherBFCL/irrelevance_167: teacher of upcoming lectures asked; only an upcoming-concert details lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 458 OUT254 |
| ✓ | NoToolForMostFamousSculptureBFCL/irrelevance_153: most famous sculpture asked; only a sculpting-tools lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 463 OUT552 |
| ✓ | NoToolForPianoMaintenanceTipsBFCL/irrelevance_166: piano maintenance tips asked; only an instrument rental-prices lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 458 OUT467 |
| × | NoToolForScreamArtistBFCL/irrelevance_159: artist behind 'The Scream' asked. NOTE: the only tool (artwork_search) requires a museum_location that the user did not provide, so it cannot be correctly called; BFCL labels this irrelevant. Kept faithfully with ground truth = no tool call (flagged as borderline).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 951 OUT777 |
| ✓ | NoToolForSculptureRockTypeBFCL/irrelevance_152: which rock suits a garden sculpture asked; only a sculpture 3D-model creator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 463 OUT590 |
Tool Calling - Irrelevance: Buildings, Sculpture & Music
7 passed1 skipped |
100.0% |
IN22 989 OUT3 016 | |
| – | NoToolForCelloInstrumentTypeBFCL/irrelevance_165: instrument classification of a cello asked. NOTE: BFCL labels this irrelevant, but the only tool (get_instrument_info with detail=type) arguably answers it; flagged for review as a borderline BFCL item. Kept faithfully with ground truth = no tool call.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_instrument_info' with detail='type' answers 'what type of instrument is a cello' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForDeclarationSignersBFCL/irrelevance_137: who signed the declaration asked; only a historical-event DATE lookup is available (returns a date, not the signatories) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 282 OUT601 |
| ✓ | NoToolForEiffelTowerHeightBFCL/irrelevance_156: Eiffel Tower height asked; only an architecture-plan generator (creates a new design, does not retrieve facts) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 286 OUT365 |
| ✓ | NoToolForFamousDancersBFCL/irrelevance_162: famous dancers asked; only a museum-visual-artists lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 288 OUT538 |
| ✓ | NoToolForReformationInitiatorBFCL/irrelevance_145: who initiated the Reformation asked; only a religious-event YEAR lookup is available (returns a year, not a person) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 281 OUT160 |
| ✓ | NoToolForScientistDefinitionBFCL/irrelevance_143: definition of "scientist" asked; only a named-historical-figure info lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 278 OUT453 |
| ✓ | NoToolForSellingGuitarBFCL/irrelevance_163: how to sell a guitar asked; only an instrument-tuning tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 283 OUT465 |
| ✓ | NoToolForSimilarBlueColorBFCL/irrelevance_148: a color similar to blue asked; only a complementary-color tool (opposite, not similar) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 291 OUT434 |
Tool Calling - Irrelevance: Cell Biology & Genetics
8 passed2 skipped |
100.0% |
IN30 315 OUT5 020 | |
| ✓ | NoToolForClottingBloodCellsBFCL/irrelevance_32: blood cell type responsible for clotting asked; only a nucleus-count-based cell-type classifier is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 790 OUT396 |
| ✓ | NoToolForCommonAncestorBFCL/irrelevance_41: common ancestor of two animals asked (phylogenetics knowledge); only a fibonacci-sequence calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 789 OUT589 |
| ✓ | NoToolForEvolutionaryAlgorithmInGameAIBFCL/irrelevance_43: how to apply an Evolutionary Algorithm in game AI asked (advice); only a creature-evolution simulator is available (wrong context).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 790 OUT1 190 |
| ✓ | NoToolForGeneVariantFrequencyBFCL/irrelevance_36: gene-variant frequency in a population asked; only a dominant-trait-from-alleles tool is available (the true-match variant-frequency tool is deliberately isolated in A4).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 795 OUT603 |
| – | NoToolForIdentifyingGeneticSequenceBFCL/irrelevance_33: interpret a genetic code sequence asked; only a species-identification-from-sequence tool is available (returns a species, not what the sequence means).0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'identify_species' (identifies a species from a genetic sequence) plausibly answers the query about a DNA sequence, so a competent model may call it - BFCL-mislabeled/borderline-relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForIncreasingDeerPopulationBFCL/irrelevance_39: how-to advice on growing an animal population asked; only a population-growth CALCULATOR (needs numeric rates) is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 789 OUT602 |
| ✓ | NoToolForLionMatingProcessBFCL/irrelevance_35: mating process of a species asked; only a dominant/recessive genetic-traits lookup is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 786 OUT483 |
| ✓ | NoToolForMitochondriaCountBFCL/irrelevance_29: number/count of mitochondria asked; only a cell-part FUNCTION lookup is available (wrong query direction).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 790 OUT576 |
| ✓ | NoToolForPandaEvolutionaryHistoryBFCL/irrelevance_42: evolutionary history asked (a narrative fact); only a biodiversity-index calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 786 OUT581 |
| – | NoToolForWhaleGeneSequenceBFCL/irrelevance_44: the actual gene sequence of a species asked; only a gene-sequence GENERATOR (needs numeric mutation rate + duration) is available.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'gene_sequencer' (generate gene sequences for evolutionary changes) directly answers the query - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Irrelevance: Color, Geometry & Population
11 passed |
100.0% |
IN40 836 OUT4 163 | |
| ✓ | NoToolForColorInComputerVisionBFCL/irrelevance_131: color meaning in computer vision asked; only a color-in-personality-psychology analyzer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 712 OUT542 |
| ✓ | NoToolForDeclarationSigningDateBFCL/irrelevance_135: signing date asked; only date-arithmetic (add days) and age tools are available, neither retrieves an event date -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 710 OUT277 |
| ✓ | NoToolForElectricityDiscovererBFCL/irrelevance_140: who discovered electricity asked; only an electric-current calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 706 OUT478 |
| ✓ | NoToolForHexColorMixBFCL/irrelevance_146: mixing two hex colors asked; only a prophet-details lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 722 OUT637 |
| ✓ | NoToolForIndependenceSigningDateBFCL/irrelevance_138: signing date asked; only an age calculator (needs birthdate + current date) and a date-adder are available, neither retrieves an event date -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 710 OUT355 |
| ✓ | NoToolForMostVisitedMarketBFCL/irrelevance_161: most visited market asked; only a museum visitation-statistics lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 712 OUT425 |
| ✓ | NoToolForNbaFinalWinnerBFCL/irrelevance_133: NBA final winner asked; only a historical-battle details lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 713 OUT288 |
| ✓ | NoToolForPicassoPaintingYearBFCL/irrelevance_151: year of a Picasso painting asked; only a sculpture-dimensions lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 711 OUT312 |
| ✓ | NoToolForRelativityAuthorBFCL/irrelevance_142: who proposed special relativity asked; only a chemical-element properties lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 715 OUT178 |
| ✓ | NoToolForSculptureCreatorBFCL/irrelevance_155: sculpture creator asked; only a max-seating-capacity calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 711 OUT184 |
| ✓ | NoToolForWhichColorsToMixBFCL/irrelevance_150: which colors to mix for a shade asked; only an RGB-to-Pantone converter is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 714 OUT487 |
Tool Calling - Irrelevance: Cooking, Currency & Markets
7 passed1 failed |
87.5% |
IN29 204 OUT3 875 | |
| × | NoToolForBakerySourdoughBFCL/irrelevance_223: a BAKERY selling sourdough asked; only a grocery-shop product finder available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 506 OUT605 |
| ✓ | NoToolForChickenRecipeBFCL/irrelevance_211: the RECIPE to cook chicken asked; only a cook-TIME calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 241 OUT677 |
| ✓ | NoToolForChocolateCakeIngredientsBFCL/irrelevance_220: cake INGREDIENTS asked; only a cooking-TEMPERATURE tool available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 244 OUT408 |
| ✓ | NoToolForCurrencyToGbpBFCL/irrelevance_233: CURRENCY conversion asked; only a WEATHER tool available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 246 OUT321 |
| ✓ | NoToolForHowToBoilAnEggBFCL/irrelevance_212: best WAY to boil an egg asked; only a boiling-TIME calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 243 OUT617 |
| ✓ | NoToolForLasagnaRecipeBFCL/irrelevance_219: the RECIPE for lasagna asked; only a recipe-CALORIE calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 241 OUT445 |
| ✓ | NoToolForLegExercisesBFCL/irrelevance_221: leg EXERCISES asked; only a grocery-list generator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 241 OUT452 |
| ✓ | NoToolForPetStoreBFCL/irrelevance_224: a PET store asked; only a GROCERY-store locator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 242 OUT350 |
Tool Calling - Irrelevance: Dates & Elements
10 passed |
100.0% |
IN36 270 OUT5 169 | |
| ✓ | NoToolForHydrogenPropertiesBFCL/irrelevance_141: chemical properties of Hydrogen asked; only a scientist-contributions lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 626 OUT614 |
| ✓ | NoToolForLargestPlanetBFCL/irrelevance_139: largest planet asked; only a star-info lookup (needs a star name) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 626 OUT564 |
| ✓ | NoToolForMostImportantProphetBFCL/irrelevance_147: most important prophet asked; only a two-color mixer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 626 OUT1 037 |
| ✓ | NoToolForMuseumVisitFrequencyBFCL/irrelevance_160: museum member visit frequency asked; only a single most-frequent-visitor lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 631 OUT547 |
| ✓ | NoToolForPantoneOfSkyBlueBFCL/irrelevance_149: Pantone code of sky blue asked; only a paint-proportions calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 628 OUT413 |
| ✓ | NoToolForPersonalityTypeBFCL/irrelevance_130: user asks what personality type they are; only a big-five calculator (which needs the user's rated answers) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 624 OUT212 |
| ✓ | NoToolForStadiumSeatingCapacityBFCL/irrelevance_154: stadium seating capacity asked; only a sculpture-creator lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 627 OUT370 |
| ✓ | NoToolForVicePresidentBFCL/irrelevance_136: Vice President asked; only a US-president-by-year lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 626 OUT572 |
| ✓ | NoToolForWaterlooCasualtiesBFCL/irrelevance_132: casualty count of a battle asked; only a historical-event date lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 628 OUT509 |
| ✓ | NoToolForWorldCupWinnerBFCL/irrelevance_134: sports World Cup winner asked; only a historical-battle outcome predictor is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 628 OUT331 |
Tool Calling - Irrelevance: Electromagnetism, Thermo & Data
9 passed1 failed |
90.0% |
IN37 489 OUT4 878 | |
| ✓ | NoToolForElectronRestMassBFCL/irrelevance_23: mass of an electron asked (a knowledge fact); only magnetic-field / current tools are available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 397 OUT227 |
| ✓ | NoToolForFlightDurationBFCL/irrelevance_20: flight duration asked; only a currency converter is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 400 OUT310 |
| ✓ | NoToolForFreezingPointBFCL/irrelevance_25: freezing point asked; only boiling-point calculators are available (wrong phase transition).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 407 OUT575 |
| × | NoToolForGasAmountGeneratedBFCL/irrelevance_26: amount of gas generated asked; only a gas-PRESSURE calculator is available (wrong quantity).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN6 864 OUT1 362 |
| ✓ | NoToolForGiniCoefficientBFCL/irrelevance_48: Gini coefficient asked; only a GDP-fetch tool is available (different economic indicator).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 400 OUT293 |
| ✓ | NoToolForHeatingWaterBFCL/irrelevance_27: heat energy to warm water asked. NOTE: the original tool (calculate_heat) is a fair match for this query (only the well-known specific heat is missing), which would make "no call" ambiguous; per human-baseline fairness it is deliberately NOT shown here (kept as a distractor in A3), so no visible tool fits and "no call" is unambiguously correct.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 411 OUT529 |
| ✓ | NoToolForHexagonSidesBFCL/irrelevance_28: number of sides of a hexagon asked (basic geometry knowledge); only a boiling-point tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 398 OUT106 |
| ✓ | NoToolForSymbolicMagneticFieldBFCL/irrelevance_22: magnetic field asked with SYMBOLIC values (distance 'r', current 'I'); the two magnetic-field tools require concrete numeric distance + current, so there is nothing to pass -> no fit.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 411 OUT493 |
| ✓ | NoToolForTimeDifferenceBFCL/irrelevance_49: time-zone difference asked; only a geo-coordinate lookup tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 400 OUT716 |
| ✓ | NoToolForWorldSeriesWinnerBFCL/irrelevance_53: sports trivia asked; only a SQL database-query runner is available (no database/query provided).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 401 OUT267 |
Tool Calling - Irrelevance: Entertainment, Sports & Knowledge
9 passed |
100.0% |
IN31 410 OUT3 109 | |
| ✓ | NoToolForBasketballResultBFCL/irrelevance_84: basketball game result asked; only a stock-price retrieval tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 490 OUT332 |
| ✓ | NoToolForBestSoccerPlayerBFCL/irrelevance_79: best current soccer player asked; only a stock-market prediction tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 488 OUT442 |
| ✓ | NoToolForCaliforniaGovernorBFCL/irrelevance_87: current Governor of California asked; only a court-case filing tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 486 OUT182 |
| ✓ | NoToolForCrimeThrillerMovieListBFCL/irrelevance_88: best crime-thriller movies of 2020 asked; only a document forgery detector is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 495 OUT499 |
| ✓ | NoToolForFifaWorldCupWinnerBFCL/irrelevance_80: FIFA World Cup 2010 winner asked; only a stock-ticker lookup is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 491 OUT298 |
| ✓ | NoToolForFrenchTranslationBFCL/irrelevance_61: French translation of a word asked; only an SVM-model tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 490 OUT233 |
| ✓ | NoToolForHorrorMovieListBFCL/irrelevance_77: horror movie recommendations asked; only an investment future-value tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 489 OUT484 |
| ✓ | NoToolForPresidentialElectionWinnerBFCL/irrelevance_85: 2020 presidential election winner asked; only a criminal-case detail lookup is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 491 OUT266 |
| ✓ | NoToolForSportsNewsBFCL/irrelevance_76: latest sports news asked; only a portfolio investment-trend analyzer is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 490 OUT373 |
Tool Calling - Irrelevance: Finance & Economics
8 passed |
100.0% |
IN26 826 OUT3 094 | |
| ✓ | NoToolForAverageHousePriceBFCL/irrelevance_69: market-wide average house price asked; only a regression predictor (needs a trained model + feature vector, not provided) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 346 OUT313 |
| ✓ | NoToolForCompoundInterestBFCL/irrelevance_70: compound-interest computation asked; only a mortgage-payment calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 372 OUT657 |
| ✓ | NoToolForGoldPriceBFCL/irrelevance_78: today's gold spot price asked; only a bond-price calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 348 OUT346 |
| ✓ | NoToolForHorrorMovieCatalogBFCL/irrelevance_81: horror movie recommendations asked; only a historical stock-price fetcher is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 349 OUT501 |
| ✓ | NoToolForLossProjectionBFCL/irrelevance_73: a loss projection asked; only a revenue-prediction tool is available (revenue != loss).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 351 OUT351 |
| ✓ | NoToolForPoundToKilogramBFCL/irrelevance_75: a mass unit conversion asked; only a portfolio-returns calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 347 OUT216 |
| ✓ | NoToolForProfitMarginBFCL/irrelevance_71: profit margin asked; only an ROI calculator is available (ROI != margin) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 369 OUT411 |
| ✓ | NoToolForStockNewsBFCL/irrelevance_82: stock news asked; only a capital-gains calculator is available (gains != news) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 344 OUT299 |
Tool Calling - Irrelevance: Finance Near-Matches
9 passed |
100.0% |
IN30 921 OUT3 397 | |
| ✓ | NoToolForBitcoinPriceBFCL/irrelevance_56: current bitcoin price asked; only a generic database record-select tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 426 OUT224 |
| ✓ | NoToolForBookstoreLookupBFCL/irrelevance_55: which online bookstore sells a title asked; only an add-to-cart tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 431 OUT317 |
| ✓ | NoToolForBusinessRateOfReturnBFCL/irrelevance_74: rate of return asked; only a net-profit calculator is available (profit != rate of return).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 447 OUT671 |
| ✓ | NoToolForCoinFlipProbabilityBFCL/irrelevance_66: probability of a coin landing heads asked; only a standard-deviation tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 431 OUT241 |
| ✓ | NoToolForCurrentInterestRateBFCL/irrelevance_83: the current market interest rate asked; only a mortgage-payment calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 425 OUT175 |
| ✓ | NoToolForExternalRateOfReturnBFCL/irrelevance_72: EXTERNAL rate of return asked; only an internal-rate-of-return (IRR) tool is available (ERR != IRR, even though cash flows are given) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 455 OUT381 |
| ✓ | NoToolForFinalPriceWithoutBasePriceBFCL/irrelevance_60: final price after discount+tax asked, but the original price is never given, so the (domain-matching) calculateFinalPrice cannot be correctly called -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 445 OUT267 |
| ✓ | NoToolForFrontendDevelopmentBFCL/irrelevance_62: how to build an e-commerce frontend asked; only a recommender-model builder is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 431 OUT697 |
| ✓ | NoToolForHighestGrossingMovieBFCL/irrelevance_54: highest-grossing movie of all time asked; only a movie search (requires a specific title + year, not provided) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 430 OUT424 |
Tool Calling - Irrelevance: Food, Time & Distance
3 passed1 failed1 skipped |
75.0% |
IN13 568 OUT1 672 | |
| ✓ | NoToolForCakeBakingMachineBFCL/irrelevance_218: a baking MACHINE/appliance asked; only a recipe retriever available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 706 OUT241 |
| ✓ | NoToolForDrivingTravelTimeBFCL/irrelevance_215: driving TRAVEL TIME asked; only a restaurant-bill tip calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 710 OUT257 |
| × | NoToolForLondonTimezoneNameBFCL/irrelevance_226: the NAME of London's timezone asked; only a local-TIME-by-timezone tool available (it needs the timezone as input and returns a time, not the timezone name) -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN5 438 OUT767 |
| – | NoToolForPizzaInBostonBFCL/irrelevance_213: a place for pizza asked; a cuisine+location restaurant_finder is available. REVIEW FLAG (fairness): restaurant_finder(cuisine=pizza/Italian, location=Boston) plausibly answers this, so a competent human might legitimately call it. BFCL labels it irrelevant; kept as such and marked Hard because the intended "no call" is debatable. No OTHER query here is a restaurant search, so the tool cannot wrongly satisfy the file's other cases.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'restaurant_finder' (cuisine+location) answers 'good place for pizza in Boston' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForTriangleHypotenuseBFCL/irrelevance_238: the HYPOTENUSE of a triangle asked; only a geo lat/long distance calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 714 OUT407 |
Tool Calling - Irrelevance: Geo, Travel & Sentiment
7 passed1 failed1 skipped |
87.5% |
IN34 535 OUT4 029 | |
| ✓ | NoToolForAverageTemperatureBmiBFCL/irrelevance_110: an average December temperature asked; only a BMI calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 396 OUT356 |
| ✓ | NoToolForCaseLawyerBFCL/irrelevance_95: the lawyer of a case asked; only a case-outcome retriever is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 399 OUT381 |
| ✓ | NoToolForCurrentTimeSunriseBFCL/irrelevance_102: the current time asked; only a sunrise/sunset time calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 395 OUT311 |
| ✓ | NoToolForFamousQuoteBFCL/irrelevance_119: a famous quote asked; only a sentiment-analysis tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 393 OUT100 |
| ✓ | NoToolForFastestRouteBFCL/irrelevance_114: the fastest route asked; only a closest-airport finder is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 397 OUT447 |
| – | NoToolForRestaurantsInLondonBFCL/irrelevance_113: restaurants in London asked; only a generic points-of-interest locator is available -> no tool fits.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'find_pois' (locate points of interest by category) answers 'restaurants in London' (restaurants are POIs) - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForSportsBrainImpactBFCL/irrelevance_120: the neurological impact of sports asked; only a caffeine neurological-impact tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 396 OUT583 |
| ✓ | NoToolForTrafficConditionBFCL/irrelevance_112: the current traffic condition asked; only an address geocoder is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 395 OUT284 |
| × | NoToolForTravelTimeByCarBFCL/irrelevance_115: travel time by car asked; only a coordinate-distance calculator is available -> no tool fits.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 3 |
Failed | IN10 764 OUT1 567 |
Tool Calling - Irrelevance: Legal & Case Lookup
10 passed |
100.0% |
IN35 507 OUT3 503 | |
| ✓ | NoToolForAppleCaloriesBFCL/irrelevance_100: calories in an apple asked; only a litigation-cost calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 548 OUT320 |
| ✓ | NoToolForBrainWeightBFCL/irrelevance_122: average human brain weight asked; only a per-patient neural-activity tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 550 OUT310 |
| ✓ | NoToolForCaseJudgementBFCL/irrelevance_93: a case judgement asked; only a law-firm impactful-cases lister is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 548 OUT373 |
| ✓ | NoToolForChineseRestaurantBFCL/irrelevance_97: a Chinese-restaurant recommendation asked; only a lawsuit-filing tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 551 OUT291 |
| ✓ | NoToolForMovieRecommendationBFCL/irrelevance_116: a movie recommendation asked; only a word-count tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 549 OUT329 |
| ✓ | NoToolForNbaChampionBFCL/irrelevance_118: the 1996 NBA champion asked; only an NBA playoff-brackets display tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 551 OUT410 |
| ✓ | NoToolForPaintingEiffelSettlementBFCL/irrelevance_98: how long to paint the Eiffel Tower asked; only a lawsuit-settlement estimator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 552 OUT310 |
| ✓ | NoToolForSecurityGuardSearchBFCL/irrelevance_92: finding a security guard asked; only a lawyer-search tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 558 OUT380 |
| ✓ | NoToolForSphereVolumeBFCL/irrelevance_104: sphere volume asked; only a circular-park area tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 552 OUT320 |
| ✓ | NoToolForTrafficLawsBFCL/irrelevance_99: traffic laws asked; only a lawsuit-search tool is available (lawsuits, not laws) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 548 OUT460 |
Tool Calling - Irrelevance: Math & Science Distractors
8 passed |
100.0% |
IN25 236 OUT2 623 | |
| ✓ | NoToolForDefiniteIntegralBFCL/irrelevance_9: definite integral asked; only an income-tax calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 157 OUT361 |
| ✓ | NoToolForIntegralOfCubeBFCL/irrelevance_8: definite integral asked; only a string-to-int converter is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 150 OUT467 |
| ✓ | NoToolForLatestMoviesBFCL/irrelevance_15: latest movie releases asked; only a physics final-velocity tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 144 OUT285 |
| ✓ | NoToolForPerpendicularSlopeBFCL/irrelevance_3: slope of a perpendicular line asked; only a critical-points finder is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 162 OUT348 |
| ✓ | NoToolForPrimeFactorsBFCL/irrelevance_13: prime factorization asked; only a compound-interest calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 147 OUT323 |
| ✓ | NoToolForRectangleAreaBFCL/irrelevance_6: rectangle area asked; only a circle-drawing tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 158 OUT192 |
| ✓ | NoToolForTriangleAreaBFCL/irrelevance_0: triangle area asked; only a BMI tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 158 OUT187 |
| ✓ | NoToolForUpwardAccelerationBFCL/irrelevance_14: acceleration asked (it is just gravity); the tempting max-height and final-velocity tools compute neither the acceleration nor what is asked -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 160 OUT460 |
Tool Calling - Irrelevance: Math & Statistics
6 passed |
100.0% |
IN18 070 OUT2 789 | |
| ✓ | NoToolForBinomialMeanBFCL/irrelevance_67: the MEAN of a binomial experiment asked; only p-value tools are available (they return a p-value, not the mean) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 026 OUT561 |
| ✓ | NoToolForCivilLawComponentsBFCL/irrelevance_90: the components/categories of Civil Law asked; only a noise-complaint filing tool is available (filing a complaint does not explain law) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN2 998 OUT350 |
| ✓ | NoToolForCoinTossMaxHeadsBFCL/irrelevance_63: how many heads CAN be obtained from tossing 3 coins (a count/maximum, not a probability) asked; only a probability-of-an-event calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 002 OUT349 |
| ✓ | NoToolForNullHypothesisStdDevBFCL/irrelevance_68: the STANDARD DEVIATION of a null-hypothesis test asked; only p-value tools are available (they return a p-value, not a standard deviation) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 036 OUT636 |
| ✓ | NoToolForPowerOfTwoBFCL/irrelevance_59: raising a number to a power asked; only a linear-equation solver is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 001 OUT213 |
| ✓ | NoToolForQuadraticEquationBFCL/irrelevance_57: solving a quadratic equation asked; only a genetic-algorithm optimizer and a LINEAR equation solver are available (a quadratic is not linear) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 007 OUT680 |
Tool Calling - Irrelevance: Medical, Social & Personality
6 passed3 skipped |
100.0% |
IN20 452 OUT2 362 | |
| ✓ | NoToolForBigMacCaloriesBFCL/irrelevance_123: calories of a Big Mac asked; only a BMI calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 407 OUT295 |
| ✓ | NoToolForEconomicHappinessEffectBFCL/irrelevance_126: the effect of economic status on happiness asked; only a happiness-index lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 409 OUT548 |
| ✓ | NoToolForFriendMbtiBFCL/irrelevance_129: a friend's MBTI asked; only a personality-type evaluator (needs trait percentages) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 407 OUT240 |
| – | NoToolForLatestTechTrendBFCL/irrelevance_124: the latest technology trend asked; only a trending-topics-by-category tool is available -> no tool fits.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_social_trends' (trending topics by category) answers 'latest trend in technology' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForMajorAirportsBFCL/irrelevance_117: major US airports asked; only a coordinate-distance calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 410 OUT426 |
| – | NoToolForMotorNeuronDiseaseBFCL/irrelevance_121: motor neuron disease info asked; only a generic disease-info retriever is available -> no tool fits.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'medical_records_get_disease_info' directly answers 'information on motor neuron diseases' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForPopularBooksBFCL/irrelevance_125: popular books by an author asked; only a recent-tweets retriever is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 409 OUT543 |
| – | NoToolForTwitterMoodBFCL/irrelevance_127: overall twitter mood asked; only a twitter sentiment-analysis tool is available -> BFCL ground truth is no tool call.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'sentiment_analysis_twitter' directly answers 'general mood of twitter regarding the new iPhone' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForVegetableServingsBFCL/irrelevance_128: daily vegetable servings asked; only a personality-test scorer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 410 OUT310 |
Tool Calling - Irrelevance: Mixed Distractors
8 passed |
100.0% |
IN25 992 OUT2 793 | |
| ✓ | NoToolForAirQualityBFCL/irrelevance_40: current air quality asked; only a plant-biomass calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 250 OUT327 |
| ✓ | NoToolForFavoriteRestaurantBFCL/irrelevance_31: a favorite restaurant recommendation asked; only a cell-division simulator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 246 OUT279 |
| ✓ | NoToolForGlobalWarmingDocumentaryBFCL/irrelevance_38: a documentary (video) asked; only a scientific-research-paper retriever is available (wrong medium).1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 247 OUT402 |
| ✓ | NoToolForLasagnaIngredientsBFCL/irrelevance_47: recipe ingredients asked; only a flight-timetable tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 247 OUT351 |
| ✓ | NoToolForLionDominantTraitBFCL/irrelevance_34: dominant genetic trait of a species asked; only a gene-variant-FREQUENCY lookup is available (different genetic output).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 249 OUT454 |
| ✓ | NoToolForMultinucleatedCellNameBFCL/irrelevance_30: name of a cell type with multiple nuclei asked; only a blood-cell classifier (by shape/size) is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 253 OUT347 |
| ✓ | NoToolForPicnicSpotBFCL/irrelevance_37: a picnic spot asked; only a local-fauna information tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 246 OUT378 |
| ✓ | NoToolForTranslationBFCL/irrelevance_52: natural-language translation asked; only a file-compression tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 254 OUT255 |
Tool Calling - Irrelevance: Motion, Electricity & Misc
10 passed |
100.0% |
IN35 408 OUT4 661 | |
| ✓ | NoToolForCarProjectileRangeBFCL/irrelevance_16: projectile-style "how far will a car travel" asked with symbolic v/t/theta; the range tool needs concrete numeric velocity + angle -> nothing to pass -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 551 OUT442 |
| ✓ | NoToolForConeVolumeBFCL/irrelevance_19: cone volume asked; only an investment-return calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 545 OUT399 |
| ✓ | NoToolForCurrentClockTimeBFCL/irrelevance_17: current clock time asked; only a travel-time (distance/speed) calculator is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 534 OUT274 |
| ✓ | NoToolForElectronMassBFCL/irrelevance_24: mass of an electron asked (a knowledge fact); only an electric-current tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 535 OUT314 |
| ✓ | NoToolForFibonacciOf20BFCL/irrelevance_51: fibonacci number asked; only a cryptocurrency-price tool is available (the fibonacci calculator lives in a different file so it cannot satisfy this query).1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 536 OUT665 |
| ✓ | NoToolForForceAngleFromMomentumBFCL/irrelevance_18: angle of a force from momentum asked; only a vector-angle-from-XY-components tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 541 OUT756 |
| ✓ | NoToolForHorrorMovieListBFCL/irrelevance_50: movie recommendations asked; only a Celsius-to-Fahrenheit converter is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 541 OUT488 |
| ✓ | NoToolForMagneticFieldNearWireBFCL/irrelevance_21: magnetic field near a current-carrying wire asked; only an EM-wave-amplitude tool is available (magnetic-field calculators are deliberately kept in a different file).1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 550 OUT629 |
| ✓ | NoToolForSineOf45DegreesBFCL/irrelevance_45: sine of an angle asked; only a polygon-drawing tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 536 OUT327 |
| ✓ | NoToolForTeslaPriceBFCL/irrelevance_46: product price of a car asked; only a currency-exchange-rate tool is available.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 539 OUT367 |
Tool Calling - Irrelevance: Music Theory & Lookup
8 passed |
100.0% |
IN25 572 OUT3 741 | |
| ✓ | NoToolForAncientGreekLawmakersBFCL/irrelevance_171: who established laws in Ancient Greece asked (history); only a musical-piece composer search is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 194 OUT597 |
| ✓ | NoToolForBasketballGameAtVenueBFCL/irrelevance_169: who is playing a basketball game at a venue asked; only concert-locating tools exist.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 197 OUT337 |
| ✓ | NoToolForBluesScalePatternBFCL/irrelevance_176: the interval pattern of a blues scale asked; only composer lookups are available (get_blues_scale lives in E2) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 199 OUT629 |
| ✓ | NoToolForDonQuixoteAuthorBFCL/irrelevance_172: author of the novel Don Quixote asked; only music-composition lookups exist -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 192 OUT297 |
| ✓ | NoToolForFamousUsComposersBFCL/irrelevance_170: most famous US composers asked (general knowledge, no specific piece); only a chord-progression creator + piece-specific composer lookups exist -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 195 OUT682 |
| ✓ | NoToolForGrammyBestAlbumBFCL/irrelevance_177: Grammy Award winner asked; only a song chord-progression lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 200 OUT479 |
| ✓ | NoToolForPrimaryTriadsOfKeyBFCL/irrelevance_173: primary triads of a key asked; only a genre-common-chords finder is available (primary_triads lives in E2) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 198 OUT396 |
| ✓ | NoToolForUniversityClassAvailabilityBFCL/irrelevance_168: availability of a university class asked; only concert-availability / concert-search tools are available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 197 OUT324 |
Tool Calling - Irrelevance: Probability & Legal
5 passed |
100.0% |
IN13 997 OUT2 416 | |
| ✓ | NoToolForFaceCardProbabilityBFCL/irrelevance_64: probability of drawing a face card from a deck asked; only a COIN-toss heads probability tool is available (coins, not cards) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 796 OUT423 |
| ✓ | NoToolForHomeElectricityNeedBFCL/irrelevance_58: home electricity demand (kWh) asked; only a solar-panel COUNT calculator is available (it sizes panels, not electricity need, and needs sunlight hours not given) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN2 799 OUT540 |
| ✓ | NoToolForNoiseComplaintProcessBFCL/irrelevance_91: how/whether to report a noise complaint asked; only a law-category listing tool is available (it lists legal categories, it does not file or advise on complaints) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 796 OUT355 |
| ✓ | NoToolForRedMarbleCountBFCL/irrelevance_65: number of red marbles asked; only a determine-population tool is available (it returns a total population from probability + sample size, not a sub-category count) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN2 811 OUT567 |
| ✓ | NoToolForTenantRightsBFCL/irrelevance_89: explaining tenant rights asked; only a contract-generation tool is available (generating a contract does not explain rights) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN2 795 OUT531 |
Tool Calling - Irrelevance: Recipes & Cooking
8 passed1 skipped |
100.0% |
IN27 642 OUT3 150 | |
| ✓ | NoToolForBestGameCharacterBFCL/irrelevance_206: best game character vs a dragon asked; only a best-WEAPON finder available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 459 OUT540 |
| ✓ | NoToolForBestMovie2020BFCL/irrelevance_235: best MOVIE of 2020 asked; only a currency-exchange calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 456 OUT321 |
| ✓ | NoToolForBuildingAPcBFCL/irrelevance_209: how to build a PC asked; only a cooking-recipe retriever available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 451 OUT679 |
| ✓ | NoToolForFamousPlaceInParisBFCL/irrelevance_210: most famous PLACE in Paris asked; only a restaurant-by-dish search available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 453 OUT368 |
| ✓ | NoToolForGameShortestPathBFCL/irrelevance_208: shortest in-game path asked; only a taxi-fare calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 461 OUT358 |
| ✓ | NoToolForInchToMeterBFCL/irrelevance_234: LENGTH conversion (inches to metres) asked; only a CURRENCY converter available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 452 OUT294 |
| ✓ | NoToolForLowestGameScoreBFCL/irrelevance_207: LOWEST game score asked; only a HIGHEST-score retriever available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 456 OUT336 |
| ✓ | NoToolForNewYorkPopulationBFCL/irrelevance_225: current POPULATION of New York asked; only a time-zone converter available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 454 OUT254 |
| – | NoToolForQuickestFlightLondonTokyoBFCL/irrelevance_236: quickest way to fly London->Tokyo asked; a flight-duration tool is available. REVIEW FLAG (fairness): get_flight_duration plausibly answers this ("quickest ... by plane" ~= quickest flight duration), so a competent human might legitimately call it. Kept as BFCL labels it (irrelevant), but this item is borderline. Marked Hard because the intended "no call" is debatable.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_flight_duration' (with flight_type non-stop) plausibly answers 'quickest way ... by plane' - BFCL-mislabeled/borderline-relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Irrelevance: Restaurants & Travel
8 passed |
100.0% |
IN26 515 OUT2 566 | |
| ✓ | NoToolForBeijingSunsetBFCL/irrelevance_227: SUNSET time asked; only a SUNRISE calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 313 OUT200 |
| ✓ | NoToolForClosestItalianRestaurantBFCL/irrelevance_216: FIND the closest Italian restaurant asked; only a currency converter available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 312 OUT251 |
| ✓ | NoToolForFindSushiRestaurantBFCL/irrelevance_214: FIND a sushi restaurant asked; only a tip calculator available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 314 OUT292 |
| ✓ | NoToolForNearestPharmacyBFCL/irrelevance_237: FIND the nearest pharmacy asked; only a route-to-a-known-location planner available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 314 OUT275 |
| ✓ | NoToolForPetFriendlyTrainStationBFCL/irrelevance_230: a pet-friendly TRAIN STATION asked; only hotel search tools available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 313 OUT311 |
| ✓ | NoToolForSushiRestaurantsTokyoBFCL/irrelevance_229: FIND sushi RESTAURANTS asked; only hotel-booking tools available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 314 OUT450 |
| ✓ | NoToolForThaiRestaurantChicagoBFCL/irrelevance_231: FIND a Thai RESTAURANT asked; only a hotel-availability checker available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 315 OUT244 |
| ✓ | NoToolForYogaRetreatBFCL/irrelevance_232: a yoga RETREAT asked; only a HOTEL search tool available. REVIEW FLAG (fairness): a retreat is not a hotel, but hotel_search_find_hotels accepts arbitrary amenities (spa, vegetarian) and a location, so it is somewhat tempting. BFCL labels this irrelevant (retreat != hotel); kept as such and marked Hard because the boundary is borderline.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 320 OUT543 |
Tool Calling - Irrelevance: Sports & Games (set 1)
10 passed |
100.0% |
IN35 688 OUT4 391 | |
| ✓ | NoToolForAssistAverageBFCL/irrelevance_185: assist average asked; only an average-SCORING tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 570 OUT380 |
| ✓ | NoToolForBestChessMoveBFCL/irrelevance_194: best next move asked; only a board-game-STATISTICS tool is available (the move-suggesting board_analyser lives in E4) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 572 OUT368 |
| ✓ | NoToolForChessArrangementCountBFCL/irrelevance_196: total theoretical arrangements asked; only a final-score calculator is available (possible_moves lives in E4) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 572 OUT626 |
| ✓ | NoToolForFootballTeamRankingBFCL/irrelevance_186: ranking of a team asked; only a season-MVP tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 568 OUT492 |
| ✓ | NoToolForIceCreamStoreBFCL/irrelevance_201: find a physical store asked; only a video-game guide is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 565 OUT295 |
| ✓ | NoToolForMostHeartsInPokerBFCL/irrelevance_199: who holds most of a suit asked; only a poker-dealing tool is available (get_highest_card_holder lives in E4) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 572 OUT445 |
| ✓ | NoToolForPlayerBiographyBFCL/irrelevance_189: a biographical "who is X" asked; only a top-ranked-athlete lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 565 OUT396 |
| ✓ | NoToolForTeamRosterBFCL/irrelevance_192: full roster asked; only a top-scorer tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 567 OUT356 |
| ✓ | NoToolForUnoRulesBFCL/irrelevance_198: the rules of a game asked; only a deck-shuffling tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 569 OUT635 |
| ✓ | NoToolForWorldCupWinnerBFCL/irrelevance_191: who won a tournament asked (no team named); only a per-team match-stats tool exists.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 568 OUT398 |
Tool Calling - Irrelevance: Sports & Games (set 2)
9 passed |
100.0% |
IN31 195 OUT3 411 | |
| ✓ | NoToolForBlackjackAceRuleBFCL/irrelevance_200: a rules question asked; only a poker highest-card-holder tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 467 OUT501 |
| ✓ | NoToolForBookAuthorBFCL/irrelevance_205: author of a book asked; only a video-game score calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 471 OUT133 |
| ✓ | NoToolForChessTournamentWinnerBFCL/irrelevance_195: who won a tournament asked; only a board-position move-suggester is available (the game-history fetch_game_stats lives in E3) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 468 OUT479 |
| ✓ | NoToolForCreateJigsawBFCL/irrelevance_204: CREATE a puzzle asked; only a puzzle-SOLVING tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 461 OUT452 |
| ✓ | NoToolForHaloPlayerRankBFCL/irrelevance_203: a player's RANK asked; only a player-SCORE lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 468 OUT362 |
| ✓ | NoToolForLakersTopScorerBFCL/irrelevance_193: top scorer of a team asked; only a team-details tool (roster/results/upcoming, not scorers) is available (sports_team.get_top_scorer lives in E3) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 466 OUT500 |
| ✓ | NoToolForMonopolyWinnerBFCL/irrelevance_197: who won a casual game asked; only a possible-moves calculator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 466 OUT286 |
| ✓ | NoToolForTopTennisPlayerBFCL/irrelevance_190: top ranked PLAYER asked; only a TEAM standing/ranking tool is available (the athlete-ranking get_top_ranked lives in E3) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 465 OUT367 |
| ✓ | NoToolForWorldSeriesWinnerBFCL/irrelevance_202: who won a real game asked; only a score calculator (from raw points) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 463 OUT331 |
Tool Calling - Irrelevance: Sports Statistics
9 passed |
100.0% |
IN31 560 OUT4 220 | |
| ✓ | NoToolForBasketballSeasonMvpBFCL/irrelevance_187: who won the MVP asked; only a team-ranking tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 509 OUT561 |
| ✓ | NoToolForClippersRosterBFCL/irrelevance_179: roster of a team asked; only a two-team game-results tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 502 OUT368 |
| ✓ | NoToolForCommonChordsInPopBFCL/irrelevance_174: most common chords in a pop song asked; only a key-signature primary-triads tool is available (find_common_chords lives in E1) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 507 OUT423 |
| ✓ | NoToolForCricketMatchParticipantsBFCL/irrelevance_180: who is playing in cricket matches today asked; only a cricket-schedule tool (dates/matches, not participants) is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 506 OUT681 |
| ✓ | NoToolForLaLigaPlayersBFCL/irrelevance_181: who played in a whole league asked; only a two-named-team last-match-result tool exists.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 502 OUT482 |
| ✓ | NoToolForMoonlightSonataComposerBFCL/irrelevance_175: composer of Moonlight Sonata asked; only a blues-scale generator is available here (all composer-lookup tools are isolated in E1) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 505 OUT226 |
| ✓ | NoToolForNbaMvpWinnerBFCL/irrelevance_184: who won an MVP award asked (no player named); only a per-player stats tool that requires a player name is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 511 OUT681 |
| ✓ | NoToolForTopAssistPlayerBFCL/irrelevance_178: top ASSIST player asked; only a top-GOAL-scorer tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 506 OUT385 |
| ✓ | NoToolForWimbledonChampionBFCL/irrelevance_183: winner of a tournament asked (no person named); only a celebrity-info tool that requires a name is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 512 OUT413 |
Tool Calling - Irrelevance: Time, Weather & Misc
3 passed1 skipped |
100.0% |
IN7 736 OUT1 435 | |
| ✓ | NoToolForCityDistanceKmBFCL/irrelevance_239: DISTANCE (km) between two cities asked; only a get_date tool that returns the TIME difference between two locations is available (its 'unit' arg mentions distance, but it computes time) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 582 OUT930 |
| – | NoToolForCurrentTimeSydneyBFCL/irrelevance_228: current time in Sydney asked; a local-time-by-city tool is available. REVIEW FLAG (fairness): get_local_time(location='Sydney') directly answers this, so a competent human would legitimately call it. BFCL nonetheless files it under irrelevance; kept as such to preserve the source label, but this item is effectively RELEVANT and should be reviewed. Marked Hard because the "no call" expectation is contradicted by the tool itself.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Not a valid irrelevance item (item flaw): the case's own tool 'get_local_time' directly answers 'current time in Sydney' - BFCL-mislabeled relevant. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | NoToolForTomatoCaloriesBFCL/irrelevance_222: CALORIES in a tomato asked; only a grocery-item-details lookup available. REVIEW FLAG (fairness): item_details at 'detailed' level could arguably surface nutrition info, and it requires a store_location the user never gave; BFCL labels it irrelevant (it is a grocery-store item lookup, not a nutrition-facts service). Kept as such; marked Medium as borderline.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN2 578 OUT374 |
| ✓ | NoToolForWriteABookBFCL/irrelevance_217: "can you write a book" asked; only a recipe-CREATION tool available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN2 576 OUT131 |
Tool Calling - Irrelevance: Weather, Environment & Services
9 passed1 failed |
90.0% |
IN40 303 OUT4 980 | |
| ✓ | NoToolForBestCo2ReductionBFCL/irrelevance_108: the best way to reduce CO2 asked; only a numeric emission-reduction estimator is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 660 OUT389 |
| ✓ | NoToolForBestMonthToVisitBFCL/irrelevance_101: best month to visit Hawaii asked; only an average-monthly-temperature tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 658 OUT568 |
| × | NoToolForCactusNutrientsBFCL/irrelevance_109: a plant's nutrient needs asked; only a weekly watering-needs calculator is available -> no tool fits.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 0
Values:
response.ToolCalls.Count = 1 |
Failed | IN7 351 OUT1 006 |
| ✓ | NoToolForCurrentTimeWeatherBFCL/irrelevance_103: the current time asked; only a current-weather forecast tool is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 659 OUT392 |
| ✓ | NoToolForGardenSpeciesCompositionBFCL/irrelevance_107: garden species composition asked; only a soil sand/silt/clay composition analyzer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 662 OUT426 |
| ✓ | NoToolForGmoYogaMatBFCL/irrelevance_111: a GMO yoga mat asked; only an eco-friendly product locator is available (GMO is not eco-friendly) -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN3 661 OUT276 |
| ✓ | NoToolForImpactfulCasesBFCL/irrelevance_94: a firm's impactful cases asked; only a single-case detail lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 669 OUT492 |
| ✓ | NoToolForPaintingEiffelCaseResultBFCL/irrelevance_96: how long to paint the Eiffel Tower asked; only a law-case result lookup is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 661 OUT381 |
| ✓ | NoToolForPollinationFlowersBFCL/irrelevance_105: top pollination flower species asked; only a route elevation-profile plotter is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN3 663 OUT532 |
| ✓ | NoToolForTomatoFertilizerBFCL/irrelevance_106: best fertilizer for tomatoes asked; only a soil-type analyzer is available -> no tool fits.1 AI query
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 659 OUT518 |
Tool Calling - Parallel Calls: Art, Music & Local Services
10 passed1 skipped |
100.0% |
IN75 666 OUT7 716 | |
| – | CalculateDrivingDistancesShortestRouteBFCL/parallel_38: two calc_distance calls — New York→Washington DC and Los Angeles→San Francisco. Tightened per methodology: the prompt explicitly enables "shortest route", so shortest_route == true is required (schema default is false).0 AI queries
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Skipped: Ambiguous scope (item flaw, not model behavior): the trailing 'with optional parameter shortest route enabled' can reasonably apply to only the second leg (its nearest clause) or to both; the BFCL ground truth requires it on both, but a competent human would not answer deterministically. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | CheckInstrumentAvailabilityInTwoCitiesBFCL/parallel_42: two check_instrument_availability calls — Yamaha P125 in Berlin and Madrid.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 486 OUT623 |
| ✓ | FindMusicStoresInTwoCitiesBFCL/parallel_41: two find_music_instrument_store calls — San Francisco and Chicago; instruments {Yamaha acoustic guitar, Kawai piano} in any order and either casing.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 511 OUT831 |
| ✓ | FindRockAndJazzConcertsInTwoCitiesBFCL/parallel_43: four concert_finder calls — {San Francisco, New York} x {rock, jazz} over the next month. time_period may be omitted (defaults to 30 days ≈ "next month"). Isolated from concert_find_nearby (parallel_44) into a separate script: both locate concerts by location+genre and would otherwise create an ambiguous tool choice.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 806 OUT1 181 |
| ✓ | GetExhibitionDetailsForTwoMuseumsBFCL/parallel_40: two museum_exhibition_detail calls — 'Wonder of Nature' at the Louvre and 'Age of Reptiles' at the British Museum. visitor_type {child, adult} (unordered). Tightened per methodology: the prompt explicitly asks for cost "for children and adult", so both visitor types are required (schema default is ['adult'] only).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 548 OUT688 |
| ✓ | GetMostPlayedPopAndRockSongsBFCL/parallel_45: two musicCharts_getMostPlayed calls — most played Pop and Rock song in Australia. Optional duration omitted (ground truth accepts the schema default of 0).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 436 OUT464 |
| ✓ | GetMuseumInfoForTwoCitiesBFCL/parallel_39: two museum_info_get_info calls — Washington D.C. and Paris; details {Opening hours, Adult tickets, Child tickets} in any order.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 500 OUT759 |
| ✓ | GetRgbAndHexForThreeColorsBFCL/parallel_37: three color_converter_get_color_info calls — navy, purple, maroon; each with {RGB, HEX} (unordered).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 622 OUT816 |
| ✓ | GetTeamRankingsInTwoLeaguesBFCL/parallel_47: two get_team_ranking calls — Barcelona in UEFA Champions League and Manchester United in La Liga.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 471 OUT733 |
| ✓ | MixPaintColorsForTwoRequestsBFCL/parallel_36: two paint_color_mixture calls — watercolor magenta and acrylic navy.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 520 OUT783 |
| ✓ | WinningPercentageForTwoTeamsTwoSeasonsBFCL/parallel_46: four calculate_winning_percentage calls — {Lakers, Bulls} x {2018, 2020}.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 766 OUT838 |
Tool Calling - Parallel Calls: Data, Geo & Life Sciences (set 1)
13 passed |
100.0% |
IN111 741 OUT13 608 | |
| ✓ | AverageFourNumberSetsBFCL/parallel_132: four calculate_average calls — one per list of numbers (element order preserved).1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 764 OUT1 116 |
| ✓ | CalculateBmiFourPeopleMetricBFCL/parallel_134: four calculate_bmi calls (weight kg, height cm). Tightened: the prompt states "All measurements are in the metric system", so unit == "metric" is required (also the default).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 790 OUT1 321 |
| ✓ | CalculateCityDistancesInKilometersBFCL/parallel_135: three geo_distance_calculate calls — NY->LA->Miami->NY, in kilometers. Tightened: the tool defaults to miles but the prompt asks "in kilometers", so units == "kilometers" is REQUIRED (omission would return miles, contradicting the request — the ground truth's "omit" option is unsafe here).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 523 OUT886 |
| ✓ | CalculateGpsDistancesAlongRouteBFCL/parallel_133: three calculate_distance calls over GPS coordinate pairs (Paris->Rome->Athens-> Giza), each in kilometers. Uses the coordinate-based tool (coord1/coord2), not the city-name distance tools that are also visible.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 790 OUT2 558 |
| ✓ | CompareAverageTemperatureNyAndLaBFCL/parallel_128: two average_temperature calls — New York 7 days in Fahrenheit, Los Angeles 7 days in Celsius. Tightened: the prompt explicitly requests Fahrenheit for NY, so temp_unit == "Fahrenheit" is required (also the default); Celsius is non-default and required for LA.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 358 OUT582 |
| ✓ | CreateTwoHistogramsWithFiveBinsBFCL/parallel_129: two create_histogram calls — one per data set, 5 bins each (data order preserved).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 822 OUT795 |
| ✓ | EstimateElephantPopulationThreeScenariosBFCL/parallel_125: three elephant_population_estimate calls — current 500 at 2%, 1.5%, 2.5% over 10y.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 699 OUT1 353 |
| ✓ | FindItalianRestaurantsInNyAndLaBFCL/parallel_130: two find_restaurants calls — New York and Los Angeles, Italian, 4 results each, dietary_requirements [vegan, gluten-free]. Distinct from restaurant_find_nearby (this tool carries food_type + number, which the prompt supplies, so it is the correct choice here).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 422 OUT800 |
| ✓ | FindRestaurantsByDietInThreeCitiesBFCL/parallel_127: three restaurant_find_nearby calls — New York, Los Angeles, Chicago, each with dietary_preference [Vegan, Gluten-free, Dairy-free]. List order follows the prompt (matches ground truth).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 545 OUT844 |
| ✓ | PredictEvolutionDarwinAndLamarckBFCL/parallel_126: two prediction_evolution calls — African Elephant over 5000y, Darwin then Lamarck. Tightened per methodology: the prompt explicitly says "using the Darwin model", so model == "Darwin" is required for the first call (ground truth also accepts omission via the 'Darwin' default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 381 OUT849 |
| ✓ | ProjectPopulationGrowthJapanAndIndiaBFCL/parallel_124: four population_projections calls — Japan/10y and India/20y, each at the current (default) growth rate and at an explicit rate. "current growth rate" maps to the default (growth_rate omitted, accepts null or 0.01); the explicit 1.5%/2.1% rates are required.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 706 OUT1 161 |
| ✓ | RouteThreeLegsAvoidingTollsBFCL/parallel_131: three map_routing_fastest_route calls — SF->Palo Alto->San Jose->home, all avoiding tolls. Tightened: the prompt explicitly says "avoiding toll roads", so avoid_tolls == true is required for every leg (non-default value stated in the prompt).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 551 OUT468 |
| ✓ | ShortestCityDistanceBusWithAndWithoutTransferBFCL/parallel_136: two city_distance_find_shortest calls — NY->LA by bus, first without transfer then allowing transfer. Tightened: the prompt explicitly says "using a bus", so transportation == "bus" is required for both (also the default). allow_transfer: false/omitted for the first, true for the second.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 390 OUT875 |
Tool Calling - Parallel Calls: Data, Geo & Life Sciences (set 2)
13 passed1 failed |
92.9% |
IN125 779 OUT12 436 | |
| ✓ | CalculateAreaOfThreeTrianglesBFCL/parallel_143: three calc_area_triangle calls — (10,5), (15,7), (20,10).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 958 OUT1 009 |
| ✓ | CalculateBinomialProbabilityBasketballBFCL/parallel_161: three calculate_binomial_probability calls — basketball shooter at 60%: (10,7), (15,10), (20,15). probability_of_success == 0.6 is required (prompt states 60%). ISOLATED here (away from prob_dist.binomial in ToolCalling-Parallel-D3): both tools compute binomial probability, and this prompt does not name a function, so having both visible would make the choice genuinely ambiguous. Kept apart so exactly one correct tool is available.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 037 OUT1 403 |
| ✓ | CalculateBmiInMetersForThreePeopleBFCL/parallel_138: three calculate_BMI calls (weight kg, height meters) — John, Sarah, Mike.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 019 OUT1 125 |
| ✓ | CalculateCelestialDistancesInMilesBFCL/parallel_149: two calculate_distance calls between celestial bodies (Mars-Venus, Mars-Jupiter), in miles. Uses the body-name distance tool (body1/body2). Tightened: prompt asks "in miles", so unit == "miles" is required (tool default is kilometers, so omission would be wrong).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 764 OUT762 |
| ✓ | CalculateClockAnglesAtThreeTimesBFCL/parallel_145: three calculate_clock_angle calls — 3:15, 8:20, 11:50. Tightened: the prompt asks for results "rounded to 2 decimal places", so round_to == 2 is required (also the default).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 939 OUT1 218 |
| ✓ | CalculateFactorialOfFourNumbersBFCL/parallel_144: four math_factorial calls — 5, 3, 4, 2.1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 968 OUT525 |
| ✓ | CalculateLightTravelTimeThreeLegsBFCL/parallel_147: three light_travel_time calls — 4.22, 6.1, 5.88 light years. Tightened: the prompt explicitly states the speed of light (299792458 m/s), so speed_of_light == 299792458 is required (also default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 025 OUT1 187 |
| × | CalculateSpeedForCarBicycleTrainBFCL/parallel_148: three calculate_speed calls — car 500m/25s in km/h, bicycle 1000m/200s in m/s, train 10000m/600s in km/h. Each output unit is explicitly named in the prompt, so to_unit is required.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match1 == 1
Values:
match1 = 0 |
Failed | IN9 029 OUT685 |
| ✓ | FetchAllDataFieldsForTwoEmployeesBFCL/parallel_139: two employee_fetch_data calls — Tech Solutions, employees 12345 and 67890, each fetching all four data fields (order follows the prompt / ground truth).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 872 OUT516 |
| ✓ | FindDiCaprioDramaAndComedyMoviesBFCL/parallel_140: two imdb_find_movies_by_actor calls — Leonardo DiCaprio, Drama in 2010 and Comedy in 2012. Tightened: the prompt explicitly names each category, so category is required for both.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 760 OUT701 |
| ✓ | GetImaxAndTwoDTheaterReleasesBFCL/parallel_141: two get_theater_movie_releases calls — New York IMAX over 7 days, Los Angeles 2D over 14 days. Tightened: the prompt explicitly names each format, so format is required for both.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 789 OUT570 |
| ✓ | PlotTwoSineWavesBFCL/parallel_146: two plot_sine_wave calls with all five parameters supplied by the prompt.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 895 OUT508 |
| ✓ | SortFourListsAscendingAndDescendingBFCL/parallel_137: eight array_sort calls — four lists, each sorted ascending AND descending (list element order preserved as given).1 AI query
8 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 844 OUT1 702 |
| ✓ | UpdateTwoCustomersNameAndEmailBFCL/parallel_142: two update_user_info calls — users 12345 and 67890, both set name "John" and email "example@.com". update_info is a nested object. Tightened: the prompt names the 'CustomerInfo' database, so database == "CustomerInfo" is required (also the default).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 880 OUT525 |
Tool Calling - Parallel Calls: Data, Geo & Life Sciences (set 3)
11 passed |
100.0% |
IN87 935 OUT9 276 | |
| ✓ | AreaUnderCurveForTwoPolynomialsBFCL/parallel_150: two mathematics_calculate_area_under_curve calls — polynomial [3,-2,1] over [-1,2] and [1,0,-1] over [0,3] (coefficient/limit order preserved).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 830 OUT937 |
| ✓ | BinomialProbabilitiesThreeGamesBFCL/parallel_160: three prob_dist_binomial calls — (20,5), (50,15), (100,30), each with p == 0.3 (prompt states a 0.3 win probability). The prompt explicitly names 'prob_dist.binomial'; the colliding calculate_binomial_probability tool is kept in a separate script (ToolCalling-Parallel-D2).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 001 OUT1 158 |
| ✓ | CalculateBmiSystemFourPeopleBFCL/parallel_154: four calculate_bmi calls (weight kg, height cm). Tightened: the prompt says "use the metric system for all calculations", so system == "metric" is required (also the default).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 128 OUT995 |
| ✓ | DiceRollProbabilitiesThreeScenariosBFCL/parallel_159: three probability_dice_roll calls — roll 4 three times and roll 2 twice on a six-sided die (die_sides default 6, accepts omission), then roll 7 twice on an eight-sided die (die_sides == 8 required, non-default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 958 OUT833 |
| ✓ | GenerateNormalVariatesTwoConfigsBFCL/parallel_158: four random_normalvariate calls — two at (mu 5, sigma 2) and two at (mu 10, sigma 3). Duplicate calls: each configuration must appear exactly twice (multiplicity 2).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 064 OUT947 |
| ✓ | PredictPricesForFourHousesBFCL/parallel_157: four predict_house_price calls — New York (3/2/1500), Los Angeles (4/3/2000), Chicago (2/1/1200), Miami (3/2/1800).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 252 OUT1 007 |
| ✓ | RaiseTwoNumbersToPowersBFCL/parallel_152: two math_power calls — 3^5 and 2^3, no modulus (mod omitted / null).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 684 OUT652 |
| ✓ | RunLinearRegressionWithAndWithoutStandardizationBFCL/parallel_155: two run_linear_regression calls — same predictors [Age, Income, Education] and target "Spending Score", first without standardization then with. standardize is explicitly requested (false then true), so it is required for both.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 876 OUT565 |
| ✓ | TrainRandomForestFourConfigsBFCL/parallel_156: four random_forest_train calls on "data_random_forest" — (100,10), (200,20), (300,30), (400,40). The prompt explicitly names this function, disambiguating it from train_random_forest_classifier visible in this script.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 310 OUT1 013 |
| ✓ | TrainRfClassifierTwoConfigsBFCL/parallel_153: two train_random_forest_classifier calls — dataset1 (depth 10, 100 trees) and dataset2 (depth 20, 200 trees). The prompt explicitly names this function, disambiguating it from the other random-forest training tool (random_forest_train) visible in this script.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 874 OUT517 |
| ✓ | TriangleAreasInDifferentUnitsBFCL/parallel_151: three geometry_area_triangle calls in different units — (15,20) square meters, (25,30) square feet, (35,40) square inches. First unit is the default (accepts omission / square meters); feet and inches are non-default and required.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 958 OUT652 |
Tool Calling - Parallel Calls: Finance, Stats & Legal
13 passed |
100.0% |
IN110 715 OUT11 472 | |
| ✓ | BuildTwoDecisionTreeClassifiersBFCL/parallel_12: two model_DecisionTreeClassifier calls — gini/depth 5 and entropy/depth 10, both random_state 1.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 440 OUT398 |
| ✓ | CalculateFutureValueForThreeTermsBFCL/parallel_18: three financials_calculate_future_value calls — years 1, 5, 10 (same 1000 @ 0.05).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 602 OUT822 |
| ✓ | CalculatePresentValueForThreeTermsBFCL/parallel_14: three calculate_present_value calls — years 10, 20, 30 (same payment 1000, rate 0.05).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 599 OUT1 050 |
| ✓ | CapitalGainsTaxForTwoStatesBFCL/parallel_15: two calculate_capital_gains_tax calls — CA (15000/25000) and FL (20000/50000).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 492 OUT1 075 |
| ✓ | ComputePythagorasForTwoTrianglesBFCL/parallel_10: two math_pythagoras calls — sides (3,4) and (5,12).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 383 OUT535 |
| ✓ | ConfidenceIntervalForTwoSampleSizesBFCL/parallel_13: two confidence_interval_calculate calls — sample sizes 50 and 150. Tightened per methodology: the prompt explicitly requests "95% confidence interval", so confidence_level == 0.95 is required (schema default is 0.9, but the prompt states the value).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 495 OUT1 945 |
| ✓ | CourtCaseStatusAndTrialDatesBFCL/parallel_22: four court_info_get_case_status calls — cases XY1234 (LA) and GH5678 (Orange County), each queried for status (details may be omitted via the 'status' default) and for trial_date.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 754 OUT1 308 |
| ✓ | GetStockPriceAndVolumeForThreeCompaniesBFCL/parallel_17: three get_stock_data calls — AAPL, GOOG/GOOGL, MSFT; each with data_points {price, volume} (unordered).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 534 OUT664 |
| ✓ | LoanEligibilityForTwoInstitutionsBFCL/parallel_20: two loan_eligibility_check calls — HSBC (500k/100k) and Wells Fargo (700k/120k).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 507 OUT609 |
| ✓ | MortgagePaymentForThreeLoanTermsBFCL/parallel_19: three calculate_mortgage_payment calls — loan terms 15, 20, 30 (400000 @ 0.04).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 627 OUT1 379 |
| ✓ | PredictHousePriceInTwoCitiesBFCL/parallel_11: two ml_predict_house_price calls — New York 3000 sq ft, Los Angeles 4000 sq ft.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 414 OUT489 |
| ✓ | ReturnOnInvestmentGainAndLossBFCL/parallel_16: two calculate_return_on_investment calls — gain 500 on 2000, loss -1000 on 5000.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 445 OUT777 |
| ✓ | SearchMoneyLaunderingConvictionsBFCL/parallel_21: two law_crimes_search calls — money laundering in San Francisco 2019 and Texas 2018.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 423 OUT421 |
Tool Calling - Parallel Calls: Legal, History & Environment
12 passed1 failed1 skipped |
92.3% |
IN117 728 OUT10 923 | |
| ✓ | BookTwoConnectingFlightsBFCL/parallel_30: two book_flight calls — San Francisco→Tokyo (2022-05-03) and Tokyo→Sydney (2022-05-18).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 057 OUT510 |
| ✓ | CalculateAlimonyForTwoDurationsBFCL/parallel_23: two alimony_calculator_ca_calculate calls — durations 10 and 20 years (10000/3000 income).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 120 OUT731 |
| × | CalculateWasteForFamilyAndBachelorBFCL/parallel_29: two waste_calculation_calculate calls — LA family (2 adults, 2 children) and New York bachelor (1 single). The population argument is a nested object.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match1 == 1
Values:
match1 = 0 |
Failed | IN9 070 OUT786 |
| ✓ | FetchTwoHistoricalFactsBFCL/parallel_31: two history_fact_fetch calls — Treaty of Paris and Magna Carta. Optional depth/year omitted (ground truth accepts the schema defaults).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 976 OUT1 091 |
| ✓ | FindClassicalConcertsWithParkingBFCL/parallel_44: two concert_find_nearby calls — classical concerts near Berlin and Paris next Friday. Tightened per methodology: the prompt explicitly asks for concerts "with available parking", so amenities == ['Parking'] is required (this also matches the schema default, but the prompt states it). Isolated from concert_finder (parallel_43) into a separate script: both locate concerts by location+genre and would otherwise create an ambiguous tool choice.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 034 OUT585 |
| – | FindCourtCasesByNumberAndTypeBFCL/parallel_26: four court_case_find calls — the cross product of case numbers {67813, 71249} and types {Civil, Criminal}; case_number is a single-element array; Civil may be omitted via its default. FLAGGED as borderline (see report): the prompt "case numbers '67813','71249' for type 'Civil' and 'Criminal'" is ambiguous about the pairing; BFCL ground truth expects all four combinations, kept faithful.0 AI queries
0 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL Skipped: Ambiguous call decomposition (item flaw, not model behavior): 'case numbers 67813, 71249 for type Civil and Criminal' has no single deterministic reading - two calls (both numbers per type, case_number as a list) is as valid as the BFCL 2x2 = four single-number calls; a competent human would not answer deterministically. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | FindNatureReservesNearTwoCitiesBFCL/parallel_27: two nature_reserve_find_nearby calls — Berkeley (10km, picnic tables + restrooms) and Tokyo (5km, playgrounds + biking trails). Amenity list order is unordered.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 058 OUT419 |
| ✓ | GetCaseDetailsWithHistoryAndLitigantsBFCL/parallel_24: two law_case_get_details calls — cases 28473 and 64725. Tightened per methodology: the prompt explicitly asks for "their history and details of litigants", so include_history == true and include_litigants == true are required (schema defaults are false).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 040 OUT809 |
| ✓ | GetEventsForTwoPresidenciesBFCL/parallel_32: two us_history_events_by_presidency calls — Abraham Lincoln and George Washington. Optional start_year/end_year omitted (ground truth accepts the schema defaults).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 971 OUT755 |
| ✓ | GetPresidentsAndVicePresidentsByYearBFCL/parallel_33: four get_president_and_vp calls — president for 1980 and 2016, vice president for 1975 and 2011.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 326 OUT1 119 |
| ✓ | GetReligionInfoForTwoEmpiresBFCL/parallel_35: two ancient_empires_get_religion_info calls — Persian Empire and Mauryan Empire. Tightened per methodology: the prompt explicitly asks for "religious history and influences", so include_influences == true is required (schema default is false).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 011 OUT644 |
| ✓ | GetTemperatureForTwoCitiesBFCL/parallel_28: two get_current_and_future_temperature calls — Seattle and Los Angeles, next 3 hours.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 983 OUT515 |
| ✓ | LookupLawsuitsForTwoYearsBFCL/parallel_25: two lawsuit_lookup calls — Dara Inc for years 2019 and 2018.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 028 OUT751 |
| ✓ | TrackChristianityInTwoRegionsBFCL/parallel_34: two religion_history_track calls — Christianity in Egypt and Turkey, 100–1500 A.D.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 054 OUT2 208 |
Tool Calling - Parallel Calls: Markets & Environment
12 passed1 failed |
92.3% |
IN119 347 OUT12 478 | |
| ✓ | CalculateCagrForTwoInvestmentsBFCL/parallel_176: two calculate_cagr calls — (5000,7000,5) and (8000,12000,3).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 665 OUT716 |
| × | CalculateEmissionSavingsForSolarAndWindBFCL/parallel_186: two calculate_emission_savings calls — solar(12 months) and wind(8 months), Midwest region. Prompt states the Midwest region explicitly -> region required (accept "Midwest"/"Midwest region").1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 4
expectedCount = 2 |
Failed | IN13 232 OUT1 154 |
| ✓ | CompareFutureValueWithAndWithoutContributionsBFCL/parallel_182: two finance_calculate_future_value calls — (5000,0.07,10, no contribution) and (3000,0.06,10, contribution 200). First scenario states "no additional contributions" -> null or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 725 OUT1 126 |
| ✓ | ComputeEuclideanNormForThreePointsBFCL/parallel_183: three math_hypot calls — (5,7), (10,15), (20,25). z omitted or 0 (2D points).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 761 OUT1 176 |
| ✓ | EstimatePopulationsForTwoSpeciesAcrossTwoYearsBFCL/parallel_185: four estimate_population calls — Bengal Tigers/India and African Elephants/Kenya for 2021 and for 2023. NOTE: upstream ground truth's third entry lists year "" (omitted -> default 2020), which is an upstream error: the prompt explicitly asks for 2023 for "both these species". We tighten both 2023 estimates to year == 2023 (fair deterministic answer).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 904 OUT948 |
| ✓ | FindRootsForTwoQuadraticEquationsBFCL/parallel_184: two algebra_quadratic_roots calls — (3,7,2) and (5,-4,1).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 678 OUT1 017 |
| ✓ | GetClosingPricesForTwoCompaniesOnTwoDatesBFCL/parallel_178: four get_stock_price (by-date variant) calls — the 2x2 cross product of {Microsoft, Apple} x {2022-01-01, 2022-02-01} on NASDAQ. Upstream ground truth's first entry loosely lists company_name as [Microsoft, Apple]; the deterministic human answer is the four distinct (company, date) combinations, which we require. exchange NASDAQ (default) may be omitted.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 894 OUT1 048 |
| ✓ | GetDetailedAirQualityForThreeDaysBFCL/parallel_187: three get_air_quality calls — New York City, detail true, for 2023-05-05, 05-04, 05-03. Prompt asks for PM2.5/PM10/ozone/sources -> detail == true; "today/yesterday/day before" with today = May 5, 2023.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 764 OUT873 |
| ✓ | GetFourPriceTypesForTwoCompaniesBFCL/parallel_180: eight stock_price calls — {Microsoft, Apple} x {Open, Close, High, Low}, days 30. Prompt explicitly lists all four data types, so data_type must be set explicitly (required) to distinguish the eight calls; omission (default 'Close') cannot represent the four requested types.1 AI query
8 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL |
Passed | IN9 438 OUT1 158 |
| ✓ | GetPricePerOunceForFourMetalsBFCL/parallel_177: four get_metal_price calls — gold, silver, platinum, palladium, measure 'ounce'. Prompt asks "price per ounce" -> measure == "ounce" required.1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 785 OUT716 |
| ✓ | GetStockPricesForFourCompaniesAndDurationsBFCL/parallel_181: four get_stock_prices calls — Apple(1 week), Microsoft(2 weeks), Amazon(3 weeks), Tesla(1 month). Each 'companies' argument is a single-element list.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 834 OUT719 |
| ✓ | GetTrafficInfoForThreeLegsWithModesBFCL/parallel_188: three get_traffic_info calls — NY->LA (driving), LA->SF (bicycling), SF->NY (transit). First leg explicitly says "driving" -> mode == "driving" required (tightened; also the default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 759 OUT854 |
| ✓ | ProjectFutureValueForThreeStocksBFCL/parallel_175: three portfolio_future_value calls — AAPL(5000,0.07,5), MSFT(8000,0.06,7), AMZN(10000,0.08,10).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 908 OUT973 |
Tool Calling - Parallel Calls: Mathematics & Physics (set 1)
12 passed1 failed |
92.3% |
IN103 958 OUT13 720 | |
| ✓ | AreaOfTwoRightTrianglesInMetersBFCL/parallel_68: two calculate_area calls — right triangles base/height 12x15 and 18x24 in meters. unit required (tool default is 'cm'; the prompt states meters/square meters).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 990 OUT1 033 |
| ✓ | CheckPlayerStatusAcrossThreeSeasonsBFCL/parallel_49: three player_status_check calls — player 3142 in RocketLeague for seasons 2017/2018/2019. Seasons are explicitly given, so each is required (default 2023 not applicable).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 071 OUT533 |
| ✓ | CircumferenceOfFourCirclesInCmBFCL/parallel_65: four calculate_circumference calls — radii 5/10/15/20. Tightened per methodology: the prompt gives every radius in cm (e.g. "5cm") and the tool default unit is 'm', so a correct call must pass unit == 'cm'/'centimeter' (ground truth requires it on the first call and merely allows omission on the rest; we require it consistently since it is stated).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 213 OUT1 436 |
| ✓ | ConvertEuroAndPoundsToDollarsBFCL/parallel_59: two get_conversion_cost calls — 15000 Euro->dollars and 200 pounds->dollars.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 903 OUT1 033 |
| × | FinalVelocityForFourDropsBFCL/parallel_81: four calculate_final_velocity calls — heights 10/20/15/25, initial_velocity 0 (stated), gravity 9.81 (default, may be omitted).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 318 OUT1 460 |
| ✓ | FindItalianAndJapaneseRestaurantsBFCL/parallel_52: two restaurant_finder calls — Italian in New York (preferences ['Vegetarian']) and Japanese in Los Angeles (delivery; ['Delivery'] or omitted since default is ['Delivery']).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 884 OUT464 |
| ✓ | FindRootsOfTwoQuadraticsAlgebraBFCL/parallel_62: two algebra_quadratic_roots calls — 3x^2+4x+2 and 5x^2-7x+3.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 050 OUT1 197 |
| ✓ | FindSoupAndVeganSaladRecipesBFCL/parallel_51: two recipe_search_find calls — Chicken Noodle Soup (diet omitted/default 'Keto') and a Vegan Salad (diet == 'Vegan', dish name accepts several phrasings).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 831 OUT626 |
| ✓ | GcdOfJohnAndMaryNumbersBFCL/parallel_80: two math_gcd calls — gcd(36,48) and gcd(60,96). Same tool as parallel_76.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 946 OUT1 026 |
| ✓ | GcdOfTwoPairsBFCL/parallel_76: two math_gcd calls — gcd(45,60) and gcd(81,27).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 878 OUT733 |
| ✓ | GetPokemonMovesAndCheckSolarBeamBFCL/parallel_48: two PokemonGO_get_moves calls — Pikachu's move list (move omitted/default 'Run') and a check whether Bulbasaur can learn 'Solar Beam'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 868 OUT1 108 |
| ✓ | PrimeFactorsFormattedAndArrayBFCL/parallel_74: four get_prime_factors calls — numbers 4567 and 7890, each formatted true and false (user wants "a formatted string as well as an array").1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 160 OUT2 745 |
| ✓ | SaveGameProgressEasyAndHardBFCL/parallel_50: two game_save_progress calls — stage 7 easy and stage 3 hard; level omitted/default 'user'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 846 OUT326 |
Tool Calling - Parallel Calls: Mathematics & Physics (set 2)
11 passed2 failed |
84.6% |
IN117 304 OUT13 039 | |
| ✓ | AreaOfThreeCirclesInMetersBFCL/parallel_66: three geometry_area_circle calls — radii 5/10/15 in meters (default is meters, so units may be omitted).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 335 OUT673 |
| ✓ | BookTwoHotelRoomsBFCL/parallel_57: two hotel_booking_book calls — Sheraton (NY, 2 adults/1 child) and Marriott (LA, 1 adult/2 children).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 383 OUT733 |
| ✓ | CheckAvailabilityForTwoHotelsBFCL/parallel_56: two hotel_booking_check_availability calls — Queens Hotel (Berlin) and Royal Hotel (Paris), each with check-in/out dates (several date formats accepted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 315 OUT631 |
| ✓ | CircumferenceOfFourCirclesInMetersBFCL/parallel_70: four geometry_circumference calls — radii 5/10/15/20. Tightened per methodology: the prompt asks for output in meters and gives radii in m, but the tool default unit is 'cm', so a correct call must pass units == 'm'/'meters' (ground truth requires it on the first call and only allows omission on the rest; we require it consistently since it is explicitly stated).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 515 OUT987 |
| ✓ | DerivativeOfPolynomialAndItsDerivativeBFCL/parallel_71: two calculate_derivative calls — 3x^3-2x^2+5x-7 at x=4, then its derivative 9x^2-4x+5 at x=2.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 230 OUT1 270 |
| ✓ | FinalVelocityForThreeDragRaceRoundsBFCL/parallel_83: three final_velocity calls — (v0=0,a=5,t=10), (v0=10,a=7,t=8), (v0=20,a=4,t=12).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 512 OUT656 |
| ✓ | FindGroceryStoresByCriteriaBFCL/parallel_55: two grocery_store_find_by_criteria calls — NYC ['24 hours'] and San Diego ['Home Delivery'].1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 148 OUT404 |
| ✓ | GetTwoCookingRecipesBFCL/parallel_53: two get_cooking_recipe calls — Lasagne Bolognese for 4 and Caesar Salad for 2.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 166 OUT1 018 |
| ✓ | GetTwoExchangeRatesBFCL/parallel_58: two get_exchange_rate calls — USD->AUD and USD->CAD.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 080 OUT313 |
| × | HighestCommonFactorOfFourPairsBFCL/parallel_77: four math_hcf calls — (45,60), (90,120), (36,48), (72,96). FLAGGED (see final report): the prompt is phrased as chaining ("use that result to find the HCF with...") but the ground truth treats it as four INDEPENDENT parallel calls on the literal pairs; a human might interpret it as sequential composition. Kept BFCL-faithful (four independent literal-pair calls) since parallel semantics preclude feeding one call's result into another.1 AI query
6 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 6
expectedCount = 4 |
Failed | IN13 470 OUT3 123 |
| ✓ | OrderSameItemsFromTwoWholeFoodsLocationsBFCL/parallel_54: two whole_foods_order calls — same two items (order-independent) at downtown and uptown, size large.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 196 OUT421 |
| ✓ | PrimeFactorizeAsDictionaryBFCL/parallel_79: two prime_factorize calls — 4567 and 7890, return_type 'dictionary' (explicitly requested).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 172 OUT1 106 |
| × | SolveTwoQuadraticEquationsBFCL/parallel_63: two solve_quadratic_equation calls — (5,6,1) and (3,2,1). Prompt names this function explicitly.1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 4
expectedCount = 2 |
Failed | IN12 782 OUT1 704 |
Tool Calling - Parallel Calls: Mathematics & Physics (set 3)
12 passed |
100.0% |
IN98 176 OUT11 182 | |
| ✓ | AreaOfTwoCirclesInMetersBFCL/parallel_67: two geometry_calculate_area_circle calls — radii 5 and 10 in meters. unit required (tool default is 'cm'; the prompt states meters).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 019 OUT818 |
| ✓ | AreaOfTwoTrianglesBFCL/parallel_69: two calculate_triangle_area calls — base/height 10x5 and 8x6.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 079 OUT670 |
| ✓ | DerivativeOfTwoFunctionsAtValuesBFCL/parallel_73: two calculus_derivative calls — 3x^2+2x-1 at 5 (variable x, default) and 4y^3-3y^2+2y-1 at 3 (variable y).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 124 OUT1 107 |
| ✓ | DisplacementForThreeCarsBFCL/parallel_84: three calculate_displacement calls — (15,7,accel 3.5), (20,10,accel 2.0), (25,8,no acceleration). The third car travels "without any acceleration" so acceleration == 0 (also the default; omission accepted).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 265 OUT1 735 |
| ✓ | EuclideanNormForThreePointsBFCL/parallel_61: three math_hypot calls — (3,4) and (6,8) with z omitted/0, and (9,12,15).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 254 OUT1 276 |
| ✓ | FactorialOfThreeNumbersBFCL/parallel_60: three math_factorial calls — 5, 7, 9.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 081 OUT417 |
| ✓ | FinalSpeedForTwoFreeFallObjectsBFCL/parallel_85: two calculate_final_speed calls — initial_speed 0 for 10 s and initial_speed 5 for 7 s; gravity -9.81 (default, may be omitted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 147 OUT1 125 |
| ✓ | GcdViaNumberTheoryBFCL/parallel_78: two number_theory_gcd calls — (45,60) and (81,63). Prompt names this function explicitly.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 022 OUT412 |
| ✓ | IntegrateTwoFunctionsWithTwoMethodsBFCL/parallel_72: four integrate calls — x**3 on [2,5] and 2x**2+3x-1 on [-1,3], each with trapezoid (default, may be omitted) and simpson (explicit).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 468 OUT979 |
| ✓ | PrimeFactorsOfThreeNumbersBFCL/parallel_75: three number_analysis_prime_factors calls — 45, 100, 150.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 087 OUT654 |
| ✓ | SolveFourQuadraticsWithRootTypeBFCL/parallel_64: four solve_quadratic calls — (2,5,3) all roots, (1,-3,2) real, (4,-7,3) all roots, (1,2,1) real. 'all' is the default so it may be omitted; 'real' is explicitly requested and required.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 538 OUT1 360 |
| ✓ | VelocityForTwoCyclingDaysBFCL/parallel_82: two calculate_velocity calls — 120 km in 5 h and 150 km in 6 h; unit km/h (default, may be omitted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 092 OUT629 |
Tool Calling - Parallel Calls: Mixed Domains
9 passed1 failed |
90.0% |
IN73 349 OUT7 039 | |
| ✓ | CalculateBmiForTwoPeopleBFCL/parallel_4: two calculate_bmi calls — 6 ft / 80 kg and 5.6 ft / 60 kg.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 278 OUT914 |
| ✓ | CalculateEmForceForTwoTimeIntervalsBFCL/parallel_1: two calculate_em_force calls — same field/area, d_time 4 then 10.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 299 OUT993 |
| ✓ | CalculateFactorialForThreeNumbersBFCL/parallel_7: three math_factorial calls — 5, 10, 15.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 338 OUT798 |
| ✓ | CalculateResistanceForCopperAndAluminumBFCL/parallel_2: two calculate_resistance calls — copper then aluminum, same length/area. Tightened per methodology: the prompt explicitly names "resistivity of copper", so the first call must pass resistivity == "copper" (ground truth also accepts omission via the 'copper' default, but an explicitly-stated value must be supplied). Aluminum is non-default → required.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 286 OUT812 |
| ✓ | CalculateSalesTaxForThreeCitiesBFCL/parallel_6: three calculate_sales_tax calls — Chicago/IL $30.45, Sacramento/CA $52.33, Portland/OR $11.23.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 470 OUT823 |
| ✓ | FetchPopulationForCitiesStateAndCountryBFCL/parallel_8: four database_us_census_get_population calls — NYC (city), LA (city), Alaska (state), USA (country); year defaults to 2000 (may be omitted). Requires area-type inference.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 554 OUT872 |
| × | FindMovieShowingsForTwoMoviesBFCL/parallel_9: two find_movie_showing calls near San Diego — Tenet at 5 pm and No Time To Die at 7:30 pm.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == expectedCount
Values:
response.ToolCalls.Count = 1
expectedCount = 2 |
Failed | IN7 144 OUT407 |
| ✓ | GetSequenceAnd3DForThreeProteinsBFCL/parallel_3: three protein_info_get_sequence_and_3D calls — HbA1c, normal + rat hemoglobin. Tightened per methodology: the prompt explicitly asks for "their 3D models", so model_3d == true is required (ground truth also accepts omission via the default of true).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 417 OUT522 |
| ✓ | ListShowRatingsOnNetflixAndHuluBFCL/parallel_5: two streaming_services calls — Netflix['Friends'] and Hulu['The Office','Stranger Things'] (Hulu show order unordered), both sorted by rating. Must group shows per service.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 301 OUT572 |
| ✓ | PlayTwoArtistsForGivenDurationsBFCL/parallel_0: two spotify_play calls — Taylor Swift for 20 min and Maroon 5 for 15 min.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 262 OUT326 |
Tool Calling - Parallel Calls: Physical & Life Sciences (set 1)
12 passed |
100.0% |
IN102 013 OUT11 526 | |
| ✓ | ElectricFieldForTwoChargesBFCL/parallel_94: two calculate_electric_field calls — charge 5/distance 2 and charge 3/distance 4. "Conducted in a vacuum" = the default permitivity → may be omitted or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 410 OUT880 |
| ✓ | ElectromagneticForceInTwoMediaBFCL/parallel_96: two electromagnetic_force calls — same charges 5,10 and distance 2, differing by medium_permittivity. Vacuum call (8.854e-12) equals the default → may be omitted or 8.854e-12; the second medium (5e-12) is non-default → required.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 566 OUT1 395 |
| ✓ | EstimateTravelTimeForThreeRoutesBFCL/parallel_93: three route_estimate_time calls — (1) NYC->Miami via Philadelphia/Washington D.C./Atlanta, (2) NYC->Miami skipping Atlanta (via Philadelphia/Washington D.C.), (3) Philadelphia-> Miami via Washington D.C. Stop orderings from ground truth accepted. FLAG parallel_93: BORDERLINE. Ground truth allows many stop-list variants, some including the start city itself as a stop; a competent human's stop set is not fully deterministic. Kept BFCL-faithful by accepting the natural stop orderings (start city excluded from stops).1 AI query
3 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL |
Passed | IN8 558 OUT932 |
| ✓ | FinalSpeedForTwoDroppedObjectsBFCL/parallel_88: two calculate_final_speed calls — v0 0/height 10 and v0 5/height 20. gravity stated as 9.8 (the default) so it may be omitted or 9.8 per ground truth.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 416 OUT938 |
| ✓ | FinalVelocityForTwoCarsBFCL/parallel_87: two calculate_final_velocity calls — Car A (v0 0, a 6, t 10), Car B (v0 20, a 4, t 15).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 433 OUT821 |
| ✓ | FinalVelocityFromDistanceForTwoObjectsBFCL/parallel_86: two kinematics_final_velocity_from_distance calls — accel 5/dist 100 and accel 10/dist 200. "Both objects start from rest" → initial_velocity may be omitted or 0 (default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 405 OUT918 |
| ✓ | FindVeganRestaurantsInThreeCitiesBFCL/parallel_91: three vegan_restaurant_find_nearby calls — LA, SF, Seattle, each with operating_hours 22 (prompt: "open until at least 22:00", a non-default value → required).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 462 OUT560 |
| ✓ | GenerateFourTravelItinerariesBFCL/parallel_90: four travel_itinerary_generator calls — Tokyo/7/200/urban, Paris/10/150/history, Sydney/5/100/nature, Rome/12/180/culture. Tokyo's "urban" equals the default → omit or "urban".1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 787 OUT659 |
| ✓ | GetDirectionsForRoundTripBFCL/parallel_89: three get_directions calls — SF->Palo Alto (fastest), Palo Alto->Golden Gate Bridge (scenic), Golden Gate Bridge->SF (fastest). route_type explicitly stated each time.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 512 OUT794 |
| ✓ | MagneticFieldForTwoLoopsBFCL/parallel_95: two calculate_magnetic_field calls — current 10/radius 0.5 and current 15/radius 1.0. "Same as in free space" = default permeability → may be omitted or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 514 OUT1 327 |
| ✓ | ResonantFrequencyForTwoCircuitsBFCL/parallel_97: two calculate_resonant_frequency calls — L 0.005/C 1e-07/round 3 and L 0.007/C 2e-07/round 4. Rounding is stated explicitly → required.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 480 OUT1 366 |
| ✓ | ShortestDrivingDistanceRoundTripBFCL/parallel_92: three get_shortest_driving_distance calls — NYC->LA, LA->Miami, Miami->NYC, all in miles (prompt explicitly "in miles" → non-default unit required).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 470 OUT936 |
Tool Calling - Parallel Calls: Physical & Life Sciences (set 2)
13 passed |
100.0% |
IN111 766 OUT15 186 | |
| ✓ | AbsolutePressureAtTwoAltitudesBFCL/parallel_102: two calc_absolute_pressure calls — gauge 2.5 with atm 1 (default; may be omitted) and gauge 2.5 with atm 0.85 (non-default → required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 519 OUT1 383 |
| ✓ | BoilingMeltingPointsAcrossSeaLevelsBFCL/parallel_100: four get_boiling_melting_points calls — water@0, iron@1000, water@1000, iron@0. FLAG parallel_100: BORDERLINE. The prompt ("water and iron at sea levels of 0 and 1000 meters respectively") reads most naturally as TWO calls (water@0, iron@1000), but the BFCL ground truth is the full 2x2 cross product (FOUR calls). Kept BFCL-faithful to the ground truth; a competent human could reasonably answer with two calls, so this item's expected call count is not fully deterministic.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 405 OUT1 127 |
| ✓ | CellInfoDetailedAndBriefBFCL/parallel_108: two biology_get_cell_info calls — neuron with detailed=true (prompt: "detailed description"), muscle with detailed=false ("less detailed", the default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 408 OUT1 531 |
| ✓ | DensityForTwoSubstancesBFCL/parallel_101: two calculate_density calls (mass/volume schema) — 10 kg / 2 m³ and 15 kg / 3 m³. unit "kg/m³" equals the default → may be omitted or "kg/m³".1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 536 OUT665 |
| ✓ | ElectricFieldStrengthAtSeveralDistancesBFCL/parallel_98: four calculate_electric_field_strength calls — charge 2, distances 0.5/1/2 in vacuum (default medium), plus distance 1 in air. The distance-1 vacuum and distance-1 air calls are distinguished by medium. Vacuum equals the default → may be omitted or "vacuum"; "air" is required.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 837 OUT1 339 |
| ✓ | EntropyChangeForTwoConditionsBFCL/parallel_103: two entropy_change_calculate calls — substance A, mass 2, temps 25->75 and 10->50. pressure 1 equals the default → may be omitted or 1.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 642 OUT1 172 |
| ✓ | FetchDnaSequenceThreeWaysBFCL/parallel_106: three fetch_DNA_sequence calls for id XYZ123 — default fasta/0, then genbank format, then 500 bp upstream. Defaults (fasta, upstream 0) may be omitted.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 613 OUT581 |
| ✓ | FunctionLookupForTwoMoleculesBFCL/parallel_110: two cell_biology_function_lookup calls — ATP in mitochondria and DNA in nucleus, both with specific_function=true (prompt asks for a specific function within the organelle).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 469 OUT1 581 |
| ✓ | HeatCapacityForThreeConditionsBFCL/parallel_105: three calc_heat_capacity calls — 300K/2.5m³, 350K/2.5m³, 300K/1.5m³, all for air. "air" equals the default gas → may be omitted or "air".1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 717 OUT878 |
| ✓ | MolecularWeightForTwoCompoundsBFCL/parallel_111: two calculate_molecular_weight calls — Glucose (C6H12O6) and Sucrose (C12H22O11), both in grams/mole (prompt explicitly requests that unit; ground truth also accepts "g/mol").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 470 OUT741 |
| ✓ | PhaseChangeEnergyForWaterBFCL/parallel_99: four thermo_calculate_energy calls — 500 g water, phase melting/freezing/ vaporization/condensation. Substance "water" equals the default → may be omitted or "water".1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 879 OUT2 063 |
| ✓ | ProteinSequencesForTwoGenesTwoSpeciesBFCL/parallel_107: four get_protein_sequence calls — BRCA1 and BRCA2 in Homo sapiens (default species, may be omitted) and in Pan troglodytes (non-default → required).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 708 OUT886 |
| ✓ | ProteinsInThreeCompartmentsBFCL/parallel_109: three cellbio_get_proteins calls — nucleus/mitochondria/cytoplasm, each with include_description=true (prompt asks for "a brief description of each protein").1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 563 OUT1 239 |
Tool Calling - Parallel Calls: Physical & Life Sciences (set 3)
13 passed |
100.0% |
IN113 530 OUT12 336 | |
| ✓ | AnalyzeDnaForThreeMutationTypesBFCL/parallel_114: six analyze_dna_sequence calls — two sequences ("AGCTTAGCTA" and "AGCTTAGGCTA") against reference "AGCTTAGCTA", each checked for insertion (default, may be omitted), deletion, and substitution mutations.1 AI query
6 tool callscategory:ToolCallinglevel:VeryHardsource:BFCL |
Passed | IN9 201 OUT1 705 |
| ✓ | DiabetesPredictionForFourPeopleBFCL/parallel_113: four diabetes_prediction calls for four people with distinct weight/height/activity.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 962 OUT842 |
| ✓ | EntropyChangeIsothermalAndNotBFCL/parallel_104: two calculate_entropy_change calls — same 300->350 K and heat_capacity 4.18, once isothermal (the default true, may be omitted) and once not isothermal (false, required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 624 OUT1 261 |
| ✓ | ForestGrowthForTwoRegionsBFCL/parallel_120: two forest_growth_forecast calls — Amazon Rainforest over 10 years and Boreal Forests of Canada over 20 years, excluding human impact (false = the default, may be omitted).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 515 OUT630 |
| ✓ | GenerateFourDnaSequencesBFCL/parallel_123: four generate_DNA_sequence calls — length 500 with a single-nucleotide preference of A, T, C, G respectively.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 822 OUT955 |
| ✓ | GeneticSimilarityInTwoFormatsBFCL/parallel_115: four genetics_calculate_similarity calls — human vs chimpanzee and human vs gorilla, each in percentage (default, may be omitted) and fraction. species1 accepts "human"/"Human".1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 810 OUT900 |
| ✓ | GenotypeFrequenciesForThreeGenotypesBFCL/parallel_116: three calculate_genotype_frequency calls — allele frequency 0.7, genotypes AA/Aa/aa.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 679 OUT799 |
| ✓ | IdentifyFourBirdsBFCL/parallel_119: four identify_bird calls — small blue/forest (default size, may be omitted), large black/lake, medium brown/desert, large green/tropical rainforest.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 918 OUT960 |
| ✓ | MutationTypeForTwoSnpsBFCL/parallel_112: two mutation_type_find calls — rs123456 in Homo sapiens (default species, may be omitted) and rs7891011 in Canis lupus familiaris (non-default → required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 537 OUT772 |
| ✓ | PopulationDensityForChinaTwoYearsBFCL/parallel_117: two calculate_density calls (country/population schema) — China in 2000 (pop 1.267 billion) and 2010 (pop 1.341 billion), land area 9.597 million km² both times.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 600 OUT970 |
| ✓ | PrecipitationStatsForThreeTimeFramesBFCL/parallel_118: three ecology_data_precipitation_stats calls — Amazon rainforest for six_months, year, five_years.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 610 OUT737 |
| ✓ | TurtlePopulationForTwoYearsBFCL/parallel_121: two ecology_get_turtle_population calls — Galapagos Islands in 2015 and 2020, both with species info requested (species=true; years stated explicitly → required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 530 OUT863 |
| ✓ | VehicleEmissionsForThreeVehiclesBFCL/parallel_122: three calculate_vehicle_emission calls — 15000 miles each: gas with default factor 1.4 (may be omitted), diesel with factor 2.7, EV with factor 0.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 722 OUT942 |
Tool Calling - Parallel Calls: Services & Lookups
11 passed1 failed |
91.7% |
IN98 414 OUT8 700 | |
| ✓ | CalculateShortestAndScenicDistancesForTwoLegsBFCL/parallel_190: four calculate_shortest_distance calls — {NYC->LA, LA->Miami} x {Shortest, Scenic}.1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 368 OUT881 |
| ✓ | FetchNewsForTwoTopicsInEuropeBFCL/parallel_192: two get_news calls — Climate Change and Artificial Intelligence, 5 articles each, region Europe. Region "Europe" explicitly requested -> required (accept "Europe"/"EU").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 049 OUT412 |
| ✓ | FindLibrariesWithFacilitiesInThreeCitiesBFCL/parallel_191: three public_library_find_nearby calls — NY[Reading Room,Fiction], LA[Wi-Fi,Children Section], Chicago[Cafe,Reading Room]. Facilities compared as an unordered pair.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 243 OUT635 |
| × | FindParksWithAmenitiesInThreeCitiesBFCL/parallel_189: three parks_find_nearby calls — NY[Tennis Court,Picnic Area], LA[Playground,Running Track], Chicago[Tennis Court,Playground]. Amenities compared as an unordered pair.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: match0 == 1
Values:
match0 = 0 |
Failed | IN8 248 OUT954 |
| ✓ | FindUpcomingEventsInThreeCitiesBFCL/parallel_194: three event_finder_find_upcoming calls — jazz/LA(14d), rock/Chicago(10d), classical/Boston(7d). All day windows explicitly stated -> days_ahead required (7 for Boston, which also equals the default).1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 261 OUT660 |
| ✓ | GetCurrentWeatherForFourCitiesBFCL/parallel_199: four get_current_weather calls — New York, Los Angeles, London, Tokyo, with both temperature and humidity included. Prompt explicitly asks to include both -> flags == true (also the default).1 AI query
4 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 343 OUT1 186 |
| ✓ | GetDetailedBriefsForTwoMoviesBFCL/parallel_195: two movie_details_brief calls — Inception and The Dark Knight, extra_info true. Prompt asks for Director/Cast/Awards -> extra_info == true required.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 027 OUT542 |
| ✓ | GetLawsuitDetailsForTwoCasesBFCL/parallel_196: two get_lawsuit_details calls — case 12345 (NY Supreme Court, with verdict) and case 67890 (LA Superior Court, no verdict). "would like the verdict" -> true; "do not need" -> null/false.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 168 OUT614 |
| ✓ | GetLawsuitInfoForTwoCasesByYearBFCL/parallel_197: two lawsuit_info calls — 12345ABC (2018, New York) and 67890XYZ (2019, California). Years are explicitly stated -> required. First location "New York" may be omitted (matches the default).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 167 OUT911 |
| ✓ | GetStockPricesForTwoCompaniesOverDaysBFCL/parallel_179: two get_stock_price (by-days variant) calls — Apple on NASDAQ for 10 days, Microsoft on NYSE for 15 days. Both exchanges are explicitly named -> required (NASDAQ / NYSE). The tool's 'company' field is a ticker symbol; company names are accepted as synonyms.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 050 OUT617 |
| ✓ | SearchLawsuitsForTwoEntitiesInCaliforniaBFCL/parallel_198: two lawsuit_search calls — Google/Santa Clara and Facebook/San Mateo, state California. Prompt states "both in the state of California" -> state required (accept "California"/"CA").1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 087 OUT735 |
| ✓ | SendTwoEmailsWithCcAndBccBFCL/parallel_193: two send_email calls — Project Update to john.doe (cc manager, bcc hr) and Meeting Reminder to jane.doe (cc assistant, bcc hr). Bodies matched exactly.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 403 OUT553 |
Tool Calling - Parallel Calls: Statistics & Finance
13 passed |
100.0% |
IN117 595 OUT13 542 | |
| ✓ | CalculateBinomialProbabilitiesForBasketballShotsBFCL/parallel_163: three calc_binomial_prob calls — (10,6,0.6), (10,6,0.5), (15,6,0.5).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 157 OUT797 |
| ✓ | CalculateCompoundInterestForThreePeriodsBFCL/parallel_170: three calculate_compound_interest calls — principal 5000, rate 0.025, time 2/3/5. Prompt states "compounded quarterly, 4 times in a year" -> n == 4 required (tightened; also ground truth).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 110 OUT1 424 |
| ✓ | CalculateDiscountedCashFlowForTwoBondsBFCL/parallel_169: two calculate_discounted_cash_flow calls — (50,5,0.05) and (60,7,0.04). Prompt explicitly states "face value of the bond is $1000" for both -> face_value == 1000 required (ground truth also accepts omission via the default, but an explicitly-stated value must be supplied).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 994 OUT1 267 |
| ✓ | CalculateEventProbabilitiesForCardDeckBFCL/parallel_162: three probability_of_event calls — Ace 4/52 as decimal, heart 13/52 as decimal, red card 26/52 as ratio. format_as_ratio: "as a decimal" -> false or omitted; "as a ratio" -> true (required).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 124 OUT1 240 |
| ✓ | CalculateQuarterlyDividendForThreeScenariosBFCL/parallel_168: three finance_calculate_quarterly_dividend_per_share calls — (5M,2M), (6M,2M), (6M,2.5M).1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 180 OUT1 173 |
| ✓ | CalculateReturnOnEquityForTwoCompaniesBFCL/parallel_171: two calculate_return_on_equity calls — (1M,5M,200k) and (2M,10M, no dividends). Second company "did not pay any dividends" -> dividends_paid null or 0.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 955 OUT875 |
| ✓ | CalculateRoiForThreeStocksBFCL/parallel_174: three calculate_return_on_investment calls — (150,180,20), (200,210,30), (250,300,40). Dividends are explicitly stated per stock -> required.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 096 OUT729 |
| ✓ | ComputeRSquaredForTwoRegressionModelsBFCL/parallel_167: two linear_regression_get_r_squared calls on the same CSV — swap education/credit_score as dependent variable. Independent-variable order follows the prompt/ground truth.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 930 OUT772 |
| ✓ | PredictFutureValueForTwoInvestmentsBFCL/parallel_172: two finance_predict_future_value calls — (5000,0.05,10) and (7000,0.04,15). Prompt states "compound interest annually" -> compounding_periods_per_year == 1 required (tightened; also the default).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 940 OUT1 093 |
| ✓ | PredictHousePricesForThreeHousesBFCL/parallel_166: three predict_house_price calls — NY 2500/3/2000, LA 3000/3/2005, Chicago 2000/2/1995. The second house is described as "similar" with only area/year/location changed, so rooms == 3 per ground truth.1 AI query
3 tool callscategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN9 147 OUT1 156 |
| ✓ | PredictProfitForTwoFundsBFCL/parallel_173: two investment_predictProfit calls — (5000,0.07,5) and (8000,0.05,7).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 916 OUT966 |
| ✓ | RunChiSquaredTestForTwoContingencyTablesBFCL/parallel_164: two chi_squared_test calls — tables [45,55,35,65] and [30,70,50,50], alpha 0.05. Prompt states "significance level of 0.05 for both" -> alpha == 0.05 required (tightened). Table order is fixed.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 900 OUT931 |
| ✓ | RunTTestAtTwoSignificanceLevelsBFCL/parallel_165: two t_test calls — same datasets, alpha 0.05 then 0.01 (both stated -> required).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 146 OUT1 119 |
Tool Calling - Simple: Arts, Venues & Music (set 1)
8 passed |
100.0% |
IN50 785 OUT4 844 | |
| ✓ | SimpleConcertDetailsBeyonceBFCL/simple_python_286 (concert.get_details): artist=Beyonce, location=San Diego, date=April 2022.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 358 OUT560 |
| ✓ | SimpleConcertInfoPopNyBFCL/simple_python_284 (concert_info.get): location=New York, date=next month, genre=Pop.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 333 OUT479 |
| ✓ | SimpleGenerateMelodyC4BFCL/simple_python_291 (music_generator.generate_melody): key=C, start_note=C4, length=16, tempo=120 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 381 OUT674 |
| ✓ | SimpleKeySignatureCsharpBFCL/simple_python_298 (music_theory.key_signature): key='C#', scale_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 318 OUT512 |
| ✓ | SimpleMuseumHoursLouvreBFCL/simple_python_273 (museum.get_hours): name='Louvre Museum', location=Paris, day omitted or Monday.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 325 OUT607 |
| ✓ | SimplePaintNeededWall30x12BFCL/simple_python_259 (calculate_paint_needed): coverage_rate=400, length=30, height=12.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 381 OUT574 |
| ✓ | SimpleReligionHistoryChristianity16cBFCL/simple_python_252 (get_religion_history): religion=Christianity, century=16, sort_by=importance, count=10.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 358 OUT868 |
| ✓ | SimpleSculptureInfoPlensaBFCL/simple_python_263 (get_sculpture_info): artist_name='James Plensa', detail=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 331 OUT570 |
Tool Calling - Simple: Arts, Venues & Music (set 2)
8 passed |
100.0% |
IN50 535 OUT4 585 | |
| ✓ | SimpleComposeMelodyCFGBFCL/simple_python_292 (compose_melody): progression=[C,F,G], measures=4, instrument=Piano or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 319 OUT348 |
| ✓ | SimpleConcertFindDetailsWeekndBFCL/simple_python_290 (concert.find_details): artist=The Weeknd, month=December, year=2022 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 297 OUT408 |
| ✓ | SimpleFindConcertRockChicagoBFCL/simple_python_285 (find_concert): location=Chicago, price=100, genre=Rock.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 307 OUT434 |
| ✓ | SimpleMuseumInfoMetBFCL/simple_python_274 (museum_info): museum_name=Metropolitan Museum of Art, info_type=opening_hours or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 285 OUT865 |
| ✓ | SimpleMusicalScaleCsharpBFCL/simple_python_299 (musical_scale): key='C#'/'C sharp', scale_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 286 OUT510 |
| ✓ | SimplePaintRequirementWallExclusionBFCL/simple_python_260 (paint_requirement.calculate): area {width=20,height=12}, paint_coverage=350, exclusion {type=window, area=15}.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 415 OUT646 |
| ✓ | SimpleReligionHistoryChristianity300To400BFCL/simple_python_254 (get_religion_history): religion=Christianity, start_year=300, end_year=400, event_type=all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 322 OUT785 |
| ✓ | SimpleSculptureDetailsDavidBFCL/simple_python_264 (sculpture.get_details): artist=Michelangelo, title=David, detail=size.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 304 OUT589 |
Tool Calling - Simple: Arts, Venues & Music (set 3)
7 passed2 failed |
77.8% |
IN67 253 OUT5 926 | |
| ✓ | SimpleChordProgressionKeyBFCL/simple_python_297 (music.theory.chordProgression): progression=[I,V,vi,IV], bools any.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 693 OUT724 |
| × | SimpleCompoundInterest10kBFCL/simple_python_269 (calculate_compound_interest): principle=10000, interest_rate=0.05, time=10, compounds_per_year=1 or omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 277 OUT1 181 |
| ✓ | SimpleConcertSearchClassicalLaBFCL/simple_python_287 (concert.search): genre=classical, location=Los Angeles/LA, date='this weekend', price_range=cheap.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 692 OUT282 |
| ✓ | SimpleCreateMixCMajorBFCL/simple_python_293 (music_composer.create_mix): scale='C Major', note_duration=quarter, track_length=180.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 715 OUT345 |
| ✓ | SimpleGenerateCircleImageRedBFCL/simple_python_256 (generate_circle_image): radius=50, color=Red, background=white or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 684 OUT231 |
| ✓ | SimpleMuseumWorkingHoursLouvreBFCL/simple_python_276 (museum_working_hours.get): museum=Louvre, location=Paris, day=Monday or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 670 OUT450 |
| ✓ | SimpleRetrieveReligionInfoBuddhismBFCL/simple_python_253 (retrieve_religion_info): religion_name=Buddhism, detail_level=full.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 665 OUT1 230 |
| ✓ | SimpleSculptureSearchChicago19cBFCL/simple_python_265 (sculpture_search): location=Chicago, time_frame='19th century', material=all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 682 OUT308 |
| × | SimpleTimeDifferenceSfSydneyBFCL/simple_python_250 (get_time_difference): place1=San Francisco/SF, place2=Sydney.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 175 OUT1 175 |
Tool Calling - Simple: Arts, Venues & Music (set 4)
7 passed2 failed |
77.8% |
IN63 599 OUT6 108 | |
| × | SimpleBuildingDimensionsEmpireStateBFCL/simple_python_270 (building.get_dimensions): building_name=Empire State Building, unit=feet.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 650 OUT1 392 |
| ✓ | SimpleConcertFindNearbySeattleBFCL/simple_python_289 (concert.find_nearby): location=Seattle, genre=jazz.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 311 OUT357 |
| ✓ | SimpleCreateChordProgressionCBFCL/simple_python_294 (music_generation.create_chord_progression): key=C, chords=4, progression_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 324 OUT300 |
| ✓ | SimpleEarliestReferenceJesusBFCL/simple_python_251 (get_earliest_reference): name='Jesus Christ', source='historical records'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 318 OUT405 |
| ✓ | SimpleFindInstrumentAcousticBFCL/simple_python_280 (find_instrument): budget=1000, type=acoustic, make omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 317 OUT580 |
| ✓ | SimpleGuitarPriceLesPaulBFCL/simple_python_283 (guitar_price.find): model='Gibson Les Paul', condition=Excellent, location=Chicago.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 331 OUT805 |
| ✓ | SimpleIdentifyColorSeaGreenBFCL/simple_python_257 (identify_color_rgb): color_name='Sea Green', standard=basic or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 315 OUT338 |
| × | SimpleMuseumInfoBritishDateBFCL/simple_python_277 (museum_info): museum='The British Museum', date=2023-06-20, information=[opening_hours,ticket_price] (either order).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 708 OUT1 534 |
| ✓ | SimpleSculptureValueThinkerBFCL/simple_python_266 (get_sculpture_value): sculpture='The Thinker', artist=Rodin.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 325 OUT397 |
Tool Calling - Simple: Arts, Venues & Music (set 5)
9 passed |
100.0% |
IN57 650 OUT4 221 | |
| ✓ | SimpleCircleDimensionsRadius5BFCL/simple_python_272 (calculate_circle_dimensions): radius=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 384 OUT355 |
| ✓ | SimpleDrawRectangleRedBFCL/simple_python_261 (draw_rectangle): width=20, height=10, color=red.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 412 OUT254 |
| ✓ | SimpleFindFluteYamahaBFCL/simple_python_282 (find_flute): brand=Yamaha, specs=[open hole, C foot, silver headjoint].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 416 OUT307 |
| ✓ | SimpleGenerateScaleProgressionCBFCL/simple_python_296 (music_generator.generate_scale_progression): key=C, tempo=80, duration=4, scale_type=major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 419 OUT362 |
| ✓ | SimpleInstrumentDetailsPianoYamahaBFCL/simple_python_278 (get_instrument_details): instrument=piano, manufacturer=Yamaha, features=[price,rating] (either order).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 400 OUT771 |
| ✓ | SimpleMetTopArtworksBFCL/simple_python_275 (metropolitan_museum.get_top_artworks): number=5, sort_by=popularity or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 410 OUT415 |
| ✓ | SimpleMixPaintYellowBlueBFCL/simple_python_258 (mix_paint_color): color1=yellow, color2=blue, lightness=60.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 406 OUT268 |
| ✓ | SimplePapalBiographyInnocentIIIBFCL/simple_python_255 (religious_history.get_papal_biography): papal_name=Innocent III, include_contributions=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 395 OUT974 |
| ✓ | SimpleSculptureLocatorMichelangeloBFCL/simple_python_268 (sculpture_locator.find_by_artist): artist=Michelangelo, material=Marble, location=Rome.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 408 OUT515 |
Tool Calling - Simple: Arts, Venues & Music (set 6)
5 passed2 failed |
71.4% |
IN45 048 OUT3 968 | |
| ✓ | SimpleAnalyzeStructureB1004BFCL/simple_python_271 (analyze_structure): building_id=B1004, floors=[2,3,4], mode=dynamic.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 006 OUT427 |
| ✓ | SimpleConcertBookingEminemBFCL/simple_python_288 (concert_booking.book_ticket): artist=Eminem, city=New York City, num_tickets=2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 966 OUT314 |
| × | SimpleFindExhibitionNyBFCL/simple_python_267 (find_exhibition): location='New York City, NY', art_form=sculpture, month omitted, user_ratings=high.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 145 OUT1 576 |
| ✓ | SimpleInstrumentInfoViolin1721BFCL/simple_python_281 (get_instrument_info): name=Violin, maker=Stradivarius, year=1721.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 000 OUT380 |
| × | SimpleInstrumentPriceStratocasterBFCL/simple_python_279 (instrument_price.get): brand=Fender, model='American Professional II Stratocaster', finish=Rosewood.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.finish?.ToLower()).AnyOf(["rosewood"]) |
Failed | IN5 981 OUT445 |
| ✓ | SimpleModifyPaintingOilBFCL/simple_python_262 (modify_painting): size='12x18', medium=oil, dominant_color=red.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 983 OUT283 |
| ✓ | SimpleSongLyricsBohemianBFCL/simple_python_295 (get_song_lyrics): song_title='Bohemian Rhapsody', artist_name=Queen, lang=English or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 967 OUT543 |
Tool Calling - Simple: Food, Travel & Shopping (set 1)
7 passed1 failed |
87.5% |
IN55 178 OUT4 098 | |
| ✓ | SimpleCookingConvertPoundsToOuncesBFCL/simple_python_365 (cooking_conversion.convert): 2 pounds->ounces of butter.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 466 OUT408 |
| × | SimpleCurrencyExchangeUsdToCadBFCL/simple_python_388 (currency_exchange.convert): USD->CAD, amount=500.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 878 OUT1 148 |
| ✓ | SimpleGroceryStoreOrganicHoustonBFCL/simple_python_369 (grocery_store.find_nearby): Houston, categories {Organic,Fruits,Vegetables} any order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 461 OUT596 |
| ✓ | SimpleHotelBookPetFriendlyManhattanBFCL/simple_python_380 (hotel_booking): Manhattan, single, 3 nights, 2023-03-10, pet_friendly.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 529 OUT243 |
| ✓ | SimpleMapsDistanceEiffelLouvreBFCL/simple_python_394 (maps.get_distance_duration): Eiffel Tower -> Louvre Museum, traffic omitted/false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 467 OUT588 |
| ✓ | SimpleRecipeFindByDietMealBFCL/simple_python_353 (find_recipes): diet=gluten-free, meal_type=dinner, ingredients omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 447 OUT451 |
| ✓ | SimpleRestaurantFinderItalianNYCBFCL/simple_python_361 (restaurant_finder): city=NYC, cuisine=Italian, diet=Gluten-free.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 462 OUT245 |
| ✓ | SimpleTimeZoneLondon24hBFCL/simple_python_376 (time_zone_converter): city=London, country=UK, display_format=24h.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 468 OUT419 |
Tool Calling - Simple: Food, Travel & Shopping (set 2)
9 passed |
100.0% |
IN63 426 OUT4 482 | |
| ✓ | SimpleBestSushiPlacesTokyoBFCL/simple_python_362 (get_best_sushi_places): city=Tokyo, top=5, review_rate=4.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 054 OUT399 |
| ✓ | SimpleCurrencyConverterUsdToGbpBFCL/simple_python_389 (currency_converter): USD->GBP, amount=200.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 049 OUT497 |
| ✓ | SimpleCurrentTimeSydneyCityBFCL/simple_python_377 (get_current_time): city=Sydney, country=Australia, format omitted/HH:MM:SS.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 020 OUT397 |
| ✓ | SimpleDistanceCalcNyToBostonTerrainBFCL/simple_python_397 (distance_calculator.calculate): New York -> Boston, consider_terrain=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 036 OUT421 |
| ✓ | SimpleHiltonAvailabilityParisBFCL/simple_python_381 (hilton_hotel.check_availability): Paris, 2023-04-04..08, 2 adults, chain Hilton/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 100 OUT416 |
| ✓ | SimpleRecipeCaloriesBeefLasagnaBFCL/simple_python_355 (recipe_info.get_calories): website=Foodnetwork.com, recipe=Beef Lasagna, meal_time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 035 OUT649 |
| ✓ | SimpleRecipeUnitTbspToTspBFCL/simple_python_366 (recipe.unit_conversion): 2 tablespoon->teaspoon, precision omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 041 OUT385 |
| ✓ | SimpleSafewayOrderPaloAltoBFCL/simple_python_370 (safeway.order): Palo Alto, [olive oil, rice], [3,1]. EXPECTED MODEL MISS (kept, do not loosen): "a five pound bag of rice" = quantity 1 (5 is the weight, not the count); models tend to send quantity 5. This is a genuine model error the test correctly catches.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 058 OUT686 |
| ✓ | SimpleVeganSoupRecipeUnder30BFCL/simple_python_354 (get_vegan_recipe): dish_type=soup, cooking_time=30, ingredient_preference omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 033 OUT632 |
Tool Calling - Simple: Food, Travel & Shopping (set 3)
7 passed1 failed |
87.5% |
IN55 377 OUT3 790 | |
| ✓ | SimpleBookHotelHiltonChicagoBFCL/simple_python_382 (book_hotel): Hilton Hotel, Chicago, single, 2022-12-10, 2 nights.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 544 OUT281 |
| ✓ | SimpleCookingTimeRoastChickenBFCL/simple_python_368 (calculate_cooking_time): weight_kg=1.5, method omitted/roast, temp omitted/180.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 484 OUT485 |
| × | SimpleCurrencyConvertEurToCadBFCL/simple_python_390 (currency_conversion.convert): 150 EUR->CAD.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 905 OUT1 153 |
| ✓ | SimpleCurrentTimeSydneyLocationBFCL/simple_python_379 (get_current_time): location=Sydney, country=Australia, timezone omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 472 OUT504 |
| ✓ | SimpleParkingLotCentralParkBFCL/simple_python_395 (parking_lot.find_nearest): Central Park NY, radius=2, type omitted/public.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 488 OUT392 |
| ✓ | SimpleRecipeFinderVeganServes2BFCL/simple_python_356 (recipe_finder.find): servings=2, diet=vegan, prep_time=30.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 508 OUT313 |
| ✓ | SimpleRestaurantClosestSushiBostonBFCL/simple_python_363 (restaurant_search.find_closest): Boston, Sushi, amenities=[Patio].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 487 OUT434 |
| ✓ | SimpleWholeFoodsPriceLABFCL/simple_python_371 (whole_foods.check_price): Los Angeles, [tomatoes, lettuce].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 489 OUT228 |
Tool Calling - Simple: Food, Travel & Shopping (set 4)
7 passed |
100.0% |
IN41 990 OUT3 026 | |
| ✓ | SimpleBookRoomThePlazaBFCL/simple_python_383 (book_room): The Plaza, single, 2 nights.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 000 OUT194 |
| ✓ | SimpleConvertCurrencyJpyToUsdBFCL/simple_python_393 (convert_currency): 20000 JPY->USD.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 006 OUT666 |
| ✓ | SimpleFindRestaurantItalianBrooklynBFCL/simple_python_364 (find_restaurant): Brooklyn, Italian, Gluten-free.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 995 OUT256 |
| ✓ | SimpleGetRecipeVeganChocCakeBFCL/simple_python_357 (get_recipe): dish_name=(vegan) chocolate cake, diet_preference=vegan.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 987 OUT546 |
| ✓ | SimpleHospitalLocateDenverBFCL/simple_python_396 (hospital.locate): Denver CO, radius=5, department=Pediatrics.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 002 OUT375 |
| ✓ | SimpleTimezoneConvertNyToLondonBFCL/simple_python_378 (timezone.convert): 3pm, New York -> London.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 998 OUT572 |
| ✓ | SimpleWholeFoodsTopBrandsBananasBFCL/simple_python_372 (whole_foods.find_top_brands): product=bananas, number omitted/5, organic=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 002 OUT417 |
Tool Calling - Simple: Food, Travel & Shopping (set 5)
4 passed2 failed |
66.7% |
IN41 718 OUT3 832 | |
| × | SimpleExchangeRateFeeGbpJpyBFCL/simple_python_391 (get_exchange_rate_with_fee): GBP->JPY, fee=0.02.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 028 OUT666 |
| ✓ | SimpleHotelBookingBookParisBFCL/simple_python_384 (hotel_booking.book): Paris, 07-10-2022..07-20-2022, 2 adults, 1 child, Standard/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 959 OUT427 |
| ✓ | SimpleMuseumHoursMetSaturdayBFCL/simple_python_398 (get_museum_hours): museum=Metropolitan Museum of Art/The Met, day=Saturday.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 873 OUT480 |
| ✓ | SimpleRecipeSearchGfCookieBFCL/simple_python_358 (recipe_search): diet=[gluten free], time_limit=30, dish=cookie.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 884 OUT532 |
| ✓ | SimpleRestaurantSearchRatedNYCBFCL/simple_python_399 (restaurant_search): NYC, Italian, rating=4, accepts_credit_cards=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 908 OUT689 |
| × | SimpleWalmartPurchaseSanJoseBFCL/simple_python_373 (walmart.purchase): San Jose, [apples, rice, (bottled) water], pack_size=[1,1,12].1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 066 OUT1 038 |
Tool Calling - Simple: Food, Travel & Shopping (set 6)
5 passed |
100.0% |
IN27 812 OUT2 249 | |
| ✓ | SimpleGameHighestScoringPlayerBFCL/simple_python_350 (get_highest_scoring_player): game=Valorant, season=2022.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 554 OUT422 |
| ✓ | SimpleHotelBookRoomKingLABFCL/simple_python_385 (hotel_bookings.book_room): LA, king size, 15-10-2023, 2 nights, rooms omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 601 OUT424 |
| ✓ | SimpleLatestExchangeGbpJpyBFCL/simple_python_392 (latest_exchange_rate): GBP->JPY, amount omitted/1.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 540 OUT498 |
| ✓ | SimpleNutritionalInfoAvocadoBFCL/simple_python_374 (grocery_info.nutritional_info): Walmart, avocado, [Protein,Calories,Carbohydrates].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 561 OUT546 |
| ✓ | SimpleRecipeSearchVegPastaBFCL/simple_python_359 (recipe_search): dietary_restriction=Vegetarian, ingredients=[pasta,cheese], servings=2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 556 OUT359 |
Tool Calling - Simple: Food, Travel & Shopping (set 7)
2 passed2 failed |
50.0% |
IN20 846 OUT1 545 | |
| × | SimpleBookHotelLuxuryParadiseBFCL/simple_python_386 (book_hotel): Hotel Paradise, Las Vegas, luxury, 05-12-2022, 3 days, view=city.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.room_type?.ToLower()).AnyOf(["luxury"]) |
Failed | IN5 265 OUT330 |
| ✓ | SimpleFindRecipeCaloriesCarbonaraBFCL/simple_python_360 (find_recipe): recipeName=pasta carbonara, maxCalories=500.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 193 OUT223 |
| ✓ | SimpleGameMultiplayerFinderBFCL/simple_python_351 (multiplayer_game_finder): platform=Windows 10, rating=4.5, genre omitted/Action.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 196 OUT364 |
| × | SimpleWalmartPriceCheckPumpkinsBFCL/simple_python_375 (walmart.check_price): [pumpkins,eggs], quantities [3,24] or [3,2], store omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.items[1].ToLower()).AnyOf(["eggs", "dozen eggs"]) |
Failed | IN5 192 OUT628 |
Tool Calling - Simple: Food, Travel & Shopping (set 8)
3 passed |
100.0% |
IN14 598 OUT1 261 | |
| ✓ | SimpleFindRecipeVeganBrowniesBFCL/simple_python_367 (find_recipe): dietary_restrictions=vegan, recipe_type=dessert (brownies), time=30.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 848 OUT304 |
| ✓ | SimpleGameAverageUserScoreBFCL/simple_python_352 (gamespot.getAverageUserScore): game=Zelda BotW, platform=Nintendo Switch/all platforms.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 858 OUT606 |
| ✓ | SimpleHotelBookingPlazaRangeBFCL/simple_python_387 (hotel_booking): Plaza Hotel, New York NY, 2022-06-01..2022-06-04, rooms omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 892 OUT351 |
Tool Calling - Simple: Knowledge, Places & History (set 1)
8 passed2 failed |
80.0% |
IN76 523 OUT5 730 | |
| × | SimpleAirQualityChicagoBFCL/simple_python_203 (get_air_quality): location=Chicago, detail=true.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 517 OUT1 118 |
| ✓ | SimpleEvolutionScientistBFCL/simple_python_242 (get_scientist_for_discovery): discovery=Theory of Evolution.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 914 OUT330 |
| ✓ | SimpleGermanWarEventsBFCL/simple_python_231 (history.get_key_events): country=Germany/DE, 1871-1945, event_type=[War].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 957 OUT888 |
| ✓ | SimpleKingOfFranceBFCL/simple_python_230 (historic_leader_search): location=France, date=1510, title omitted or King.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 928 OUT487 |
| ✓ | SimplePandaPopulationBFCL/simple_python_201 (estimate_population): species=panda(s), country=China/CN, year omitted or 2024.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 915 OUT433 |
| × | SimplePersonalityTraitsTypeBFCL/simple_python_227 (get_personality_traits by type): type=ENFJ, traits=[strengths, weaknesses].1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 565 OUT1 016 |
| ✓ | SimplePresidentCivilWarBFCL/simple_python_238 (us_history.get_president): event=American Civil War, year in 1861-1865.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 925 OUT436 |
| ✓ | SimpleRestaurantNearbyBFCL/simple_python_204 (restaurant.find_nearby): location=Seattle[, WA], cuisine=Chinese, max_distance=10.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 934 OUT253 |
| ✓ | SimpleTordesillasDateBFCL/simple_python_233 (european_history.get_event_date): event=Treaty of Tordesillas, format=YYYY.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 939 OUT314 |
| ✓ | SimpleTrafficBostonNewYorkBFCL/simple_python_205 (get_traffic_info): start=Boston, end=New York/NYC, mode omitted or driving.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 929 OUT455 |
Tool Calling - Simple: Knowledge, Places & History (set 2)
9 passed1 failed |
90.0% |
IN69 330 OUT5 359 | |
| ✓ | SimpleBitcoinNewsBFCL/simple_python_210 (get_news): topic=Bitcoin, quantity=5, region omitted or US.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 890 OUT418 |
| × | SimpleEuropeanWars19cBFCL/simple_python_234 (history_eu.fetch_events): century=19, region=any of N/S/E/W, category=Wars. EXPECTED MODEL MISS (kept, do not loosen): region is an enum (Northern/Southern/Eastern/Western); the prompt says only "European", so any of the four is accepted, but models often send the invalid value "Europe". Failing to respect the enum is a genuine model error the test correctly catches.1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 4 |
Failed | IN7 291 OUT1 191 |
| ✓ | SimpleLibraryNearbyBFCL/simple_python_209 (public_library.find_nearby): location=Boston MA/Massachusetts, facilities=[Fiction, Wi-Fi] (any order).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 913 OUT431 |
| ✓ | SimpleMonarchOfEnglandBFCL/simple_python_232 (monarch.getMonarchOfYear): location=England/ENG, year=1800, fullName=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 907 OUT403 |
| ✓ | SimpleNeutronDiscovererBFCL/simple_python_243 (get_discoverer): discovery=neutron, detail=true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 886 OUT541 |
| ✓ | SimpleParksNearbyBFCL/simple_python_206 (parks.find_nearby): location=London[, UK], amenities=[Tennis Court].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 894 OUT268 |
| ✓ | SimplePersonalityTraitsHobbyBFCL/simple_python_228 (get_personality_traits by hobby): hobby=jogging, trait_count=3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 889 OUT693 |
| ✓ | SimplePopulationGrowthBFCL/simple_python_221 (population_growth_estimate): location=London, years=5, rate omitted or 1.2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 892 OUT496 |
| ✓ | SimplePresidentDuringEventBFCL/simple_python_241 (US_President_During_Event): event=Civil War, country omitted or USA.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 885 OUT384 |
| ✓ | SimpleTreatyLisbonDateBFCL/simple_python_235 (get_event_date): event=Treaty of Lisbon (variants), location omitted or Lisbon[, Portugal].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 883 OUT534 |
Tool Calling - Simple: Knowledge, Places & History (set 3)
10 passed |
100.0% |
IN72 337 OUT4 958 | |
| ✓ | SimpleAppleStockInfoBFCL/simple_python_212 (get_stock_info): company=Apple Inc., detail_level=detailed, market omitted or NASDAQ.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 219 OUT555 |
| ✓ | SimpleCivilWarStartDateBFCL/simple_python_236 (us_history.get_event_info): event=American Civil War/Civil War, specific_info=Start Date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 220 OUT557 |
| ✓ | SimpleFlightBookBFCL/simple_python_213 (flight.book): departure=San Francisco/SF, destination=London, date=2022-04-27 (variants), time=afternoon, direct=true.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 274 OUT425 |
| ✓ | SimpleGravitationPublicationYearBFCL/simple_python_244 (publication_year.find): author=Isaac Newton, work=Law of Universal Gravitation (variants), location omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 230 OUT290 |
| ✓ | SimpleMovieBriefBFCL/simple_python_215 (movie_details.brief): title=Interstellar, extra_info omitted or false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 209 OUT502 |
| ✓ | SimplePresidentIn1861BFCL/simple_python_239 (US_president.in_year): year=1861, full_name omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 237 OUT485 |
| ✓ | SimpleRadiumDiscovererBFCL/simple_python_245 (discoverer.get): element_name=radium (quote variants), year omitted or 0, first omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 199 OUT548 |
| ✓ | SimpleSendEmailBFCL/simple_python_211 (send_email): to=john.doe@example.com, subject=Meeting, body (opt trailing period), cc/bcc empty.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 276 OUT210 |
| ✓ | SimpleShortestDistanceBFCL/simple_python_207 (calculate_shortest_distance): start=New York (variants), end=Miami (variants), route_preference=Shortest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 243 OUT803 |
| ✓ | SimpleUpcomingConcertsBFCL/simple_python_214 (event_finder.find_upcoming): location=New York (variants), genre=Rock, days_ahead=30.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 230 OUT583 |
Tool Calling - Simple: Knowledge, Places & History (set 4)
10 passed |
100.0% |
IN71 798 OUT5 408 | |
| ✓ | SimpleBmiCalcBFCL/simple_python_222 (calculate_bmi): weight=70, height=180, unit omitted or metric.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 179 OUT658 |
| ✓ | SimpleCarEmissionsBFCL/simple_python_200 (calculate_emissions): distance=12000, fuel_type=gas, fuel_efficiency omitted or 25.0, efficiency_reduction omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 212 OUT867 |
| ✓ | SimpleFmriAnalyzeBFCL/simple_python_217 (fMRI.analyze): data_source path, sequence_type=multi-band, smooth=6, voxel_size=2.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 222 OUT464 |
| ✓ | SimpleGetDirectionsBFCL/simple_python_208 (map_service.get_directions): start=New York/NYC, end=Los Angeles/LA, avoid=[highways, tolls] (any order).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 174 OUT534 |
| ✓ | SimpleGravityDiscoveryBFCL/simple_python_246 (science_history.get_discovery_details): discovery=Gravity, method_used omitted or default.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 139 OUT557 |
| ✓ | SimpleGroupDynamicsBFCL/simple_python_223 (group_dynamics.pattern): total=50, extroverts=15, introverts=35.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 199 OUT358 |
| ✓ | SimpleMostFollowedBFCL/simple_python_224 (social_media_analytics.most_followed): topic=psychology, sub_topics=[behaviour, group dynamics], region omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 170 OUT597 |
| ✓ | SimpleNeuronCoordinatesBFCL/simple_python_219 (get_neuron_coordinates): neuron_type=GABA, brain_region=All/all part of the brain/entire brain.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 174 OUT628 |
| ✓ | SimplePresidentIn1940BFCL/simple_python_240 (history_api.get_president_by_year): year=1940, full_term_only omitted/true/false.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 151 OUT409 |
| ✓ | SimpleSentimentAnalysisBFCL/simple_python_216 (sentiment_analysis): text=the review, language=english/en.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 178 OUT336 |
Tool Calling - Simple: Knowledge, Places & History (set 5)
10 passed |
100.0% |
IN73 074 OUT6 709 | |
| ✓ | SimpleBigFiveScoresBFCL/simple_python_229 (get_bigfive_scores): characteristics=[efficient, organized, easy going, compassionate], scale omitted or medium.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 304 OUT677 |
| ✓ | SimpleChristianityHistoryBFCL/simple_python_249 (religion.history_info): religion=Christianity, till_century=14, include_people omitted or false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 293 OUT857 |
| ✓ | SimpleEinsteinContributionBFCL/simple_python_247 (historical_contrib.get_contrib): scientist=Albert Einstein, date=1915-03-17 (variants), category omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 310 OUT578 |
| ✓ | SimpleHistoricalGdpBFCL/simple_python_237 (get_historical_GDP): country=United States/US, start_year=1960, end_year=2000.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 315 OUT359 |
| ✓ | SimpleMriReportBFCL/simple_python_218 (patient.get_mri_report): patient_id=546382, mri_type omitted or brain, status=concluded.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 324 OUT350 |
| ✓ | SimpleNeuronalActivityBFCL/simple_python_220 (calculate_neuronal_activity): input_synaptic_rate=200, weight=0.5, decay_rate=0.1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 334 OUT597 |
| ✓ | SimpleReadingPreferenceBFCL/simple_python_225 (psych_research.get_preference): category=reading, option_one=digital[reading], option_two=physical book(s), demographic omitted or all.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 297 OUT480 |
| ✓ | SimpleRelativityInventionBFCL/simple_python_248 (science_history.get_invention): invention_name=theory of relativity, want_year=true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 287 OUT767 |
| ✓ | SimpleRenewableEmissionSavingsBFCL/simple_python_202 (calculate_emission_savings): energy_type=renewable, usage_duration=3, region=California/CA. EXPECTED MODEL MISS (kept, do not loosen): the prompt says "renewable energy sources" generically; models often over-specify a concrete type (e.g. "solar"). Ground truth wants "renewable".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 315 OUT1 341 |
| ✓ | SimpleZodiacCompatibilityBFCL/simple_python_226 (get_zodiac_compatibility): sign1=Aries, sign2=Gemini, scale omitted or percentage.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 295 OUT703 |
Tool Calling - Simple: Law, Finance & Records (set 1)
10 passed |
100.0% |
IN73 011 OUT7 096 | |
| ✓ | SimpleAirQualityByDateBFCL/simple_python_196 (air_quality): location London, date reformatted to 08-16-2022.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 287 OUT1 036 |
| ✓ | SimpleCivilCaseDetailsWithDissentBFCL/simple_python_167 (law.civil.get_case_details): case_title Roe v. Wade, include_dissent true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 287 OUT653 |
| ✓ | SimpleCompoundedInterestQuarterlyBFCL/simple_python_153 (calculate_compounded_interest): principal 5000, rate 0.03, time 5, n 4 (quarterly).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 323 OUT699 |
| ✓ | SimpleCrimeRateByCityBFCL/simple_python_164 (get_crime_rate): city San Francisco, state California/CA, type violent/omitted, year 2020.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 297 OUT945 |
| ✓ | SimpleCriminalRecordsByPersonBFCL/simple_python_158 (get_criminal_records): name Mr. X, location New York, NY, from_year 2012, to_year 2015.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 320 OUT655 |
| ✓ | SimpleDetailedWeatherForecastHoursBFCL/simple_python_185 (detailed_weather_forecast): location New York, duration 72 hours, include_precipitation true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 290 OUT665 |
| ✓ | SimpleFutureValueByDurationBFCL/simple_python_155 (calculate_future_value): initial_investment 1000, interest_rate 0.05, duration 2, compounded omitted or 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 309 OUT842 |
| ✓ | SimpleLawsuitDetailsWithExtraFieldsBFCL/simple_python_178 (get_lawsuit_details): case_number LAX2019080202, court_location Los Angeles, additional_details omitted or full default list.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 307 OUT514 |
| ✓ | SimpleLawsuitSearchByFilingDateBFCL/simple_python_168 (lawsuit_search): company Google/GOOG, start_date 01-01-2021, location California, status ongoing/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 316 OUT484 |
| ✓ | SimpleLawyerExperienceMonthsBFCL/simple_python_175 (lawyer.get_experience): name John Doe, law_type Bankruptcy.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 275 OUT603 |
Tool Calling - Simple: Law, Finance & Records (set 2)
9 passed1 failed |
90.0% |
IN70 820 OUT4 893 | |
| ✓ | SimpleAirQualityIndexByTimeBFCL/simple_python_197 (get_air_quality_index): location San Diego, time 12pm/12:00.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 055 OUT575 |
| ✓ | SimpleCivilCasesRetrieveTheftBFCL/simple_python_165 (civil_cases.retrieve): year 2020, crime_type theft, location Los Angeles.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 070 OUT329 |
| ✓ | SimpleFindNearbyDivorceLawyerBFCL/simple_python_166 (lawyer.find_nearby): city Chicago IL, specialty [Divorce], fee 400. NOTE: single-call contract kept intentionally; the model occasionally emits a spurious second call (no other visible tool fits this query), which is a model-reliability signal, not a test defect.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 082 OUT742 |
| ✓ | SimpleGenerateLawContractRentalBFCL/simple_python_162 (generate_law_contract): parties [John, Alice], contract_type Rental Agreement, location California/CA.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 069 OUT351 |
| ✓ | SimpleLawsuitDetailsFindByCompanyBFCL/simple_python_176 (lawsuit_details.find): company_name Apple Inc., year 2010, case_type Patent/IPR.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 081 OUT559 |
| ✓ | SimpleLawsuitDetailsWithVerdictBFCL/simple_python_181 (get_lawsuit_details): case_number 123456-ABC, court_location Los Angeles, with_verdict true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 093 OUT378 |
| ✓ | SimpleLocalNurseryAnnualPlantsBFCL/simple_python_193 (local_nursery.find): location Toronto, plant_types [Annual].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 059 OUT328 |
| × | SimplePropertyRecordsWithOwnerBFCL/simple_python_163 (property_records.get): address 123 main street, parcel_number 1234567890, county Santa Clara, include_owner true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.county?.ToLower()).AnyOf(["santa clara"]) |
Failed | IN7 142 OUT521 |
| ✓ | SimpleRoiSavingsAccountBFCL/simple_python_150 (calculate_roi): deposit 1000, annual_interest_rate 0.03, years 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 103 OUT403 |
| ✓ | SimpleWeatherForecastDetailedDaysBFCL/simple_python_189 (weather_forecast_detailed): location New York, days 3, details true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 066 OUT707 |
Tool Calling - Simple: Law, Finance & Records (set 3)
7 passed2 failed1 broken |
70.0% |
IN75 531 OUT11 097 | |
| ✓ | SimpleCaseInfoVictimBFCL/simple_python_160 (get_case_info): docket 2022/AL2562, court California/CA, info_type victim.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 202 OUT847 |
| ✓ | SimpleCrimeStatuteTheftBFCL/simple_python_161 (crime_statute_lookup): jurisdiction California/CA, crime theft, detail_level detailed.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 178 OUT757 |
| ! | SimpleCurrentWeatherCelsiusBFCL/simple_python_186 (current_weather_condition): city Tokyo, country Japan, measurement c/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN7 178 OUT3 139 |
| × | SimpleEnvironmentalAirQualityDaysBFCL/simple_python_199 (environmental_data.air_quality_index): location San Jose, days 3.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 910 OUT801 |
| ✓ | SimpleFelonyCrimeRecordBFCL/simple_python_156 (crime_record.get_record): case_number CA123456, county San Diego, details true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 195 OUT572 |
| × | SimpleHighestGrossingBankBFCL/simple_python_151 (highest_grossing_banks): country U.S variants, year 2020, top_n 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.top_n == 1
Values:
args.top_n = null |
Failed | IN7 177 OUT874 |
| ✓ | SimpleHistoricalFraudCaseBFCL/simple_python_170 (law_case_search.find_historical): subject fraud, from_year 2010, to_year 2015.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 194 OUT448 |
| ✓ | SimpleLawsuitCasesByCompanyYearBFCL/simple_python_177 (get_lawsuit_cases): company_name Facebook, year 2018, status all/omitted. EXPECTED MODEL MISS (kept, do not loosen): "Patent" is the case TYPE, not the status (enum is open/closed/all); status should be omitted/"all". Models often wrongly set status="Patent".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 168 OUT1 291 |
| ✓ | SimpleLawsuitInfoByCaseNumberBFCL/simple_python_182 (lawsuit_info): case_number XYZ123, year omitted/2023, location omitted/all.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 157 OUT617 |
| ✓ | SimplePlantsForSlopeErosionBFCL/simple_python_194 (get_plants_for_slope): slope_type hill/steep/moderate, num_results 3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 172 OUT1 751 |
Tool Calling - Simple: Law, Finance & Records (set 4)
10 passed |
100.0% |
IN73 536 OUT5 429 | |
| ✓ | SimpleActDetailsAmendmentBFCL/simple_python_159 (get_act_details): act_name Criminal Law Amendment Act, amendment_year 2013.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 328 OUT747 |
| ✓ | SimpleCheckFeloniesByNameBFCL/simple_python_157 (criminal_history.check_felonies): full_name John Doe, birth_date 01-01-1980, state California/CA.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 368 OUT536 |
| ✓ | SimpleCourtCaseSearchNoFullTextBFCL/simple_python_169 (court_case.search): docket_number 123456, location Texas, full_text false/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 355 OUT427 |
| ✓ | SimpleCurrentWeatherTempHumidityBFCL/simple_python_187 (get_current_weather): location Seattle, include_temperature true/omitted, include_humidity true/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 303 OUT688 |
| ✓ | SimpleLawsuitCheckClosedStatusBFCL/simple_python_184 (lawsuit.check_case): case_id 1234, closed_status true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 341 OUT435 |
| ✓ | SimpleLawsuitsSearchByCompanyLocationBFCL/simple_python_180 (lawsuits_search): company_name Google, location California/CA, year 2020, case_type all/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 336 OUT364 |
| ✓ | SimpleMutualFundBalanceBFCL/simple_python_152 (calculate_mutual_fund_balance): investment_amount 50000, annual_yield 0.05, years 3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 366 OUT549 |
| ✓ | SimpleSlopeGradientCoordinatesBFCL/simple_python_192 (calculate_slope_gradient): point1 [40.7128,-74.006], point2 [34.0522,-118.2437], unit degree/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 431 OUT839 |
| ✓ | SimpleStateLawCaseYearRangeBFCL/simple_python_173 (law_case_search): topic land disputes, year_range [2015,2021], location New York, judicial_system state.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 377 OUT401 |
| ✓ | SimpleTopConstitutionalCasesBFCL/simple_python_174 (get_top_cases): field_of_law constitutional law, top_number 10, country China/CN.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 331 OUT443 |
Tool Calling - Simple: Law, Finance & Records (set 5)
8 passed2 failed |
80.0% |
IN79 655 OUT7 556 | |
| ✓ | SimpleCarbonFootprintLifestyleBFCL/simple_python_195 (calculate_carbon_footprint): daily_miles 20, meat_meals_per_week 3, annual_trash_weight 500, flights_per_year omitted/0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 263 OUT693 |
| ✓ | SimpleDailyWaterIntakeWeightBFCL/simple_python_198 (calculate_daily_water_intake): weight 70, activity_level omitted/moderate, climate omitted/temperate.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 189 OUT618 |
| ✓ | SimpleFetchLawCaseByNumberBFCL/simple_python_171 (fetch_law_case_details): case_number 43403, court New York, year 2018.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 233 OUT319 |
| ✓ | SimpleFutureValueByPresentValueBFCL/simple_python_154 (calculate_future_value): present_value 5000, annual_interest_rate 0.05, years 10, compounds_per_year omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 242 OUT856 |
| ✓ | SimpleHumidityForecastDaysBFCL/simple_python_188 (weather.humidity_forecast): location Miami, days 7, min_humidity omitted/0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 198 OUT622 |
| × | SimpleLatestCourtCaseCompaniesBFCL/simple_python_179 (find_latest_court_case): company1 Apple, company2 Samsung, country USA/omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 974 OUT1 323 |
| ✓ | SimpleLawsuitSearchByCountyBFCL/simple_python_183 (lawsuit_search): entity Apple, county Santa Clara County, state California/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 188 OUT448 |
| ✓ | SimpleLegalCaseFetchDetailsBFCL/simple_python_172 (legal_case.fetch): case_id R vs Adams, details true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 198 OUT481 |
| × | SimpleParkElevationAreaBFCL/simple_python_190 (park_information): park_name Yellowstone, information [Elevation, Area] (order-independent).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 961 OUT1 620 |
| ✓ | SimpleTallestMountainsRadiusBFCL/simple_python_191 (locate_tallest_mountains): location Denver, radius 50, amount 5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 209 OUT576 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 1)
1 passed |
100.0% |
IN4 008 OUT293 | |
| ✓ | SimpleAnswerStringLogistic1BFCL/live_simple_32-9-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L113: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 008 OUT293 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 2)
1 passed |
100.0% |
IN3 976 OUT215 | |
| ✓ | SimpleAnswerStringLogistic2BFCL/live_simple_33-10-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L114: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN3 976 OUT215 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 3)
1 skipped |
- |
IN0 OUT0 | |
| – | SimpleAnswerStringValidBFCL/live_simple_34-11-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L115: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Harness-semantic mismatch (item flaw for our runner): the query ('...Valid?') is directly answerable, and our system prompt permits answering in plain text when no tool is needed, so the model reasonably returns 0 tool calls instead of routing to the answer_string meta-tool. Not a deterministic tool-routing test. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Simple: Live Answer Tool (isolated) (set 4)
1 passed |
100.0% |
IN4 157 OUT467 | |
| ✓ | SimpleAnswerStringWhatIsBFCL/live_simple_35-12-0 (answer.string): answer states logistic regression is not present / cannot be answered. ISOLATED from ToolCalling-Simple-L116: answer_string is a universal "respond to the user" collider (it can answer ANY query), so alongside specific tools it makes the 27B oscillate (Completed=false loops that broke the co-located tests). Solo file => only this tool is visible.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 157 OUT467 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 1)
2 passed1 failed |
66.7% |
IN14 737 OUT1 072 | |
| ✓ | SimpleChangeFoodBurgerBFCL/live_simple_20-4-0 (change_food): food_item='burger', modification_request='no onions, extra cheese'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 899 OUT185 |
| × | SimpleThinqAirCleanHighBFCL/live_simple_40-17-0 (ThinQ_Connect): AIR_CLEAN mode, HIGH wind, monitoring on, air clean POWER_ON. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. windStrength broadened (high|strongest|strong) for "strongest wind strength". EXPECTED MODEL MISS: 27B sets airConJobMode=AIR_CLEAN but omits airCleanOperationMode=POWER_ON — the prompt asks to "start the air purifying function", so POWER_ON is the fair GT; kept strict (a real tool-use miss).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.airCleanOperationMode?.ToLower()).AnyOf(["power_on"]) |
Failed | IN4 939 OUT255 |
| ✓ | SimpleWeatherTelAvivBFCL/live_simple_4-3-0 (get_current_weather): location='Tel Aviv, Israel', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 899 OUT632 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 10)
2 passed1 failed |
66.7% |
IN13 950 OUT2 759 | |
| ✓ | SimpleChafodOrderPizzaBFCL/live_simple_23-5-1 (ChaFod): TheFod='PIZZA'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 626 OUT1 003 |
| ✓ | SimpleUberEatMcdonaldsPizzaBFCL/live_simple_29-7-2 (uber.eat.order): restaurant="McDonald's", items=[pizza], quantities=[1].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 660 OUT470 |
| × | SimpleWeatherShanghaiBFCL/live_simple_13-3-9 (get_current_weather): location='Shanghai, China', unit=celsius. Prompt is Chinese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["shanghai, china"]) |
Failed | IN4 664 OUT1 286 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 11)
3 passed |
100.0% |
IN15 100 OUT1 146 | |
| ✓ | SimpleChafodSwitchBurgerBFCL/live_simple_24-5-2 (ChaFod): TheFod='BURGER'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 005 OUT213 |
| ✓ | SimpleFindBeerSierraNevadaBFCL/live_simple_48-21-0 (find_beer): brewery='Sierra Nevada', taste=bitter, aroma=hoppy, color=pale, style=lager; defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 065 OUT449 |
| ✓ | SimpleWeatherChennaiBFCL/live_simple_15-3-11 (get_current_weather): location='Chennai, Tamil Nadu', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 030 OUT484 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 12)
3 passed |
100.0% |
IN15 215 OUT1 612 | |
| ✓ | SimpleChafodGreensSaladBFCL/live_simple_25-5-3 (ChaFod): TheFod='SALAD' (greens mapped to salad).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 030 OUT258 |
| ✓ | SimpleFindBeerCreekPorterBFCL/live_simple_49-21-1 (find_beer): brewery='Creek', taste='slightly bitter', aroma=chocolatey, color=dark, style=porter; defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 122 OUT332 |
| ✓ | SimpleWeatherQuintanaRooBFCL/live_simple_19-3-15 (get_current_weather): location='Quintana Roo, Mexico', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 063 OUT1 022 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 13)
2 passed |
100.0% |
IN9 605 OUT1 085 | |
| ✓ | SimpleLexListExportsV2BFCL/live_simple_30-8-0 (aws.lexv2_models.list_exports): botId='my-bot-id', botVersion='v2', sortBy=ASC or omitted, others omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 838 OUT459 |
| ✓ | SimpleWeatherLangSonBFCL/live_simple_16-3-12 (get_current_weather): location='Lang Son, Vietnam', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 767 OUT626 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 14)
2 passed |
100.0% |
IN9 600 OUT925 | |
| ✓ | SimpleLexListExportsDescBFCL/live_simple_31-8-1 (aws.lexv2_models.list_exports): botId='B12345', botVersion='v1', sortBy=DESC, others omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 842 OUT631 |
| ✓ | SimpleWeatherBostonBFCL/live_simple_17-3-13 (get_current_weather): location='Boston, MA', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 758 OUT294 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 15)
2 passed |
100.0% |
IN8 714 OUT967 | |
| ✓ | SimpleGetUserInfoBlackBFCL/live_simple_0-0-0 (get_user_info): user_id=7890, special='black'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 365 OUT389 |
| ✓ | SimpleWeatherMoscowBFCL/live_simple_18-3-14 (get_current_weather): location='Moscow, Russia', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 349 OUT578 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 16)
3 passed |
100.0% |
IN14 597 OUT2 113 | |
| ✓ | SimpleFetchWeatherCoordsBFCL/live_simple_39-16-0 (fetch_weather_data): latitude=37.8651, longitude=-119.5383; url/units/language defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 879 OUT961 |
| ✓ | SimpleGithubStarGorillaBFCL/live_simple_1-1-0 (github_star): repos='ShishirPatil/gorilla,gorilla-llm/gorilla-cli', aligned=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 881 OUT350 |
| ✓ | SimpleWeatherBangkokBFCL/live_simple_14-3-10 (get_current_weather): location='Bangkok, Thailand', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 837 OUT802 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 17)
3 passed |
100.0% |
IN13 882 OUT1 307 | |
| ✓ | SimpleCarbonIntensityGBBFCL/live_simple_50-22-0 (get_latest_carbon_intensity): zone='Great Britain' or 'GB'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 590 OUT437 |
| ✓ | SimpleMultiplyThreeTwoBFCL/live_simple_47-20-0 (multiply): a=3, b=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 592 OUT240 |
| ✓ | SimpleWeatherOpenMeteoMariposaBFCL/live_simple_38-15-0 (get_current_weather, open-meteo variant): location='Mariposa, CA', unit=fahrenheit; url default omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 700 OUT630 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 2)
3 passed |
100.0% |
IN14 691 OUT1 005 | |
| ✓ | SimpleChangeFoodChickenSpicyBFCL/live_simple_21-4-1 (change_food): food_item='chicken dish', modification_request='extra spicy'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 916 OUT237 |
| ✓ | SimpleThinqAirDryMidBFCL/live_simple_41-17-1 (ThinQ_Connect): AIR_DRY mode, MID wind (or omitted), all else optional. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. windStrength broadened (mid|medium) for "medium wind strength"; air-clean off-state accepts off/power_off.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 889 OUT228 |
| ✓ | SimpleWeatherDivinopolisBFCL/live_simple_5-3-1 (get_current_weather): location='Divinopolis, MG' (verbatim), unit=fahrenheit or omitted. Prompt is Portuguese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 886 OUT540 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 3)
3 passed |
100.0% |
IN14 717 OUT1 874 | |
| ✓ | SimpleParseAnswerNoDefineBFCL/live_simple_36-13-0 (parseAnswer): answer='The text does not define logistic regression, therefore I cannot answer this question.'1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 914 OUT885 |
| ✓ | SimpleThinqKoreanTarget22BFCL/live_simple_42-17-2 (ThinQ_Connect): coolTargetTemperature=22 required; mode COOL and wind MID optional. Prompt is Korean, copied verbatim. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. Prompt is dehumidify (제습) so GT's optional airConJobMode=COOL is a source oddity — accept cool/dehumidify/air_dry (optional). windStrength MID broadened (mid|medium). The "22도" target-temp is accepted in either coolTargetTemperature or targetTemperature (the two are genuinely ambiguous for "target temperature").1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 920 OUT387 |
| ✓ | SimpleWeatherSanFranciscoBFCL/live_simple_6-3-2 (get_current_weather): location='San Francisco, CA', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 883 OUT602 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 4)
3 passed |
100.0% |
IN14 579 OUT1 078 | |
| ✓ | SimpleParseAnswerNoDefineValidityBFCL/live_simple_37-14-0 (parseAnswer): answer='The text does not define logistic regression, therefore I cannot answer this question.'1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 873 OUT216 |
| ✓ | SimpleThinqCoolHighWindBFCL/live_simple_43-17-3 (ThinQ_Connect): HIGH wind required; COOL mode and coolTargetTemperature=24 optional. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. air-clean off-state broadened (power_off|off) — the model files the unrequested clean-mode as "off".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 868 OUT290 |
| ✓ | SimpleWeatherRigaBFCL/live_simple_7-3-3 (get_current_weather): location='Riga, Latvia'; unit fahrenheit/celsius/omitted all accepted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 838 OUT572 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 5)
3 passed |
100.0% |
IN15 192 OUT1 414 | |
| ✓ | SimpleThinqTurnedOnCoolBFCL/live_simple_44-18-0 (ThinQ_Connect): airConOperationMode=POWER_ON required; COOL currentJobMode optional. NOTE: tool flattened to nullable top-level params (see struct comment) so a valid partial call is one tool call, not a forced retry. windStrength/airClean off-state broadened to synonyms (mid|medium, stop|off|power_off).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 041 OUT586 |
| ✓ | SimpleUberComfortAddisonBFCL/live_simple_2-2-0 (uber.ride): loc='2020 Addison Street, Berkeley, CA, USA', type=comfort, time=600.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 103 OUT320 |
| ✓ | SimpleWeatherLondonBFCL/live_simple_8-3-4 (get_current_weather): location='London, UK' or 'London, England'; unit fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 048 OUT508 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 6)
2 passed1 failed |
66.7% |
IN15 227 OUT1 514 | |
| × | SimpleThinqCoolAirCleanStartBFCL/live_simple_45-18-1 (ThinQ_Connect): airCleanOperationMode=START, airConOperationMode=POWER_ON, coolTargetTemperature=20. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. airCleanOperationMode "air cleaning on" broadened (start|on|power_on). airConOperationMode=POWER_ON and coolTargetTemperature=20 kept strict — both fair GT. EXPECTED MODEL MISS: 27B gets job-mode + air-clean START + temp, but omits airConOperationMode=POWER_ON.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.airConOperationMode?.ToLower()).AnyOf(["power_on"]) |
Failed | IN5 093 OUT612 |
| ✓ | SimpleUberPlusBakerBFCL/live_simple_3-2-1 (uber.ride): loc='221B Baker Street, Berkeley, CA, USA', type=plus, time=600.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 105 OUT398 |
| ✓ | SimpleWeatherHyderabadBFCL/live_simple_9-3-5 (get_current_weather): location='Hyderabad, India', unit=fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 029 OUT504 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 7)
3 passed |
100.0% |
IN15 244 OUT1 473 | |
| ✓ | SimpleThinqFlatCoolCleanBFCL/live_simple_46-19-0 (ThinQ_Connect, flat params): COOL, MID, monitoring on, clean START, POWER_ON, power-save off, target 24C.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 144 OUT662 |
| ✓ | SimpleUberPlusVietnameseBFCL/live_simple_26-6-0 (uber.ride): loc Vietnamese/English form, type=plus, time=10 (minutes). Prompt is Vietnamese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 067 OUT281 |
| ✓ | SimpleWeatherYosemiteMariposaBFCL/live_simple_10-3-6 (get_current_weather): location='Mariposa, CA', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 033 OUT530 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 8)
2 passed1 failed |
66.7% |
IN15 066 OUT1 344 | |
| × | SimpleThinqCoolScheduledBFCL/live_simple_51-23-0 (ThinQ_Connect): COOL, HIGH wind, POWER_ON, power-save on, target 22C, start in 1h30m. NOTE: tool flattened to nullable top-level params (see struct comment) to avoid forced-retry Count>1. airConOperationMode=POWER_ON kept strict ("Start in 1h30m" schedules a power-on). EXPECTED MODEL MISS: 27B gets job-mode COOL + wind + power-save + temp + schedule, but omits airConOperationMode=POWER_ON.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.airConOperationMode?.ToLower()).AnyOf(["power_on"]) |
Failed | IN5 096 OUT337 |
| ✓ | SimpleUberEatPitadaBFCL/live_simple_27-7-0 (uber.eat.order): restaurant='uber pitada', items=[burgers, chicken wings], quantities=[5, 6].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 005 OUT233 |
| ✓ | SimpleWeatherNaplesBFCL/live_simple_11-3-7 (get_current_weather): location='Naples, FL' or 'Naples, Florida'; unit fahrenheit or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 965 OUT774 |
Tool Calling - Simple: Live Assistants, Weather & Devices (set 9)
3 passed |
100.0% |
IN14 050 OUT1 178 | |
| ✓ | SimpleChafodWhopperBurgerBFCL/live_simple_22-5-0 (ChaFod): TheFod='BURGER' (Whopper mapped to burger).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 639 OUT310 |
| ✓ | SimpleUberEatKfcChineseBFCL/live_simple_28-7-1 (uber.eat.order): KFC order, 4 items, quantities [10,50,30,90]. Prompt is Chinese, copied verbatim.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 713 OUT420 |
| ✓ | SimpleWeatherNewYorkBFCL/live_simple_12-3-8 (get_current_weather): location='New York, NY', unit=celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 698 OUT448 |
Tool Calling - Simple: Live Media, Weather & Data (set 2)
8 passed2 failed |
80.0% |
IN80 012 OUT4 120 | |
| ✓ | SimpleCreditQuoteAutoCreditoBFCL/live_simple_67-31-0 (obtener_cotizacion_de_creditos): monto=1000000, plazo=12, producto='auto', año=2024, enganche=0.2.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 696 OUT608 |
| ✓ | SimpleCurrentLocDmsBFCL/live_simple_87-48-0 (get_current_loc): coordinate_format='DMS', include_altitude=true, timeout default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 583 OUT591 |
| × | SimpleGetMoviesMumbaiTodayBFCL/live_simple_59-28-0 (get_movies): city='Mumbai', optionals default/omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 483 OUT817 |
| ✓ | SimplePlayArtistKaarijaTracksBFCL/live_simple_93-54-0 (play_artist): artist_name='Käärijä'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 566 OUT214 |
| ✓ | SimpleRetrieveAnalysisSuperfancyBFCL/live_simple_69-33-0 (analysis_api.AnalysisApi.retrieve_analysis): project/component/vulnerability UUIDs.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 681 OUT397 |
| ✓ | SimpleSensorAlertsPerPage10BFCL/live_simple_70-34-0 (get_sensor_alerts): perPage=10, sensorSerial='Q3CC-CRT3-SZ2G', rest omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 629 OUT409 |
| ✓ | SimpleSitefinityNewsItemJulian12BFCL/live_simple_81-42-0 (sitefinity_create_contentitem): ContentItem='NewsItem', Title='Julian is testing12', rest omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 632 OUT390 |
| ✓ | SimpleTodoAddShopping9pmB25BFCL/live_simple_55-25-1 (todo): type='add', content='go for shopping at 9 pm'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 575 OUT163 |
| × | SimpleTodoDeleteTodoRandomBFCL/live_simple_54-25-0 (todo): type='delete', content='todo random'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.content?.ToLower()).AnyOf(["todo random"]) |
Failed | IN7 577 OUT229 |
| ✓ | SimpleWeatherGetLondonUkBFCL/live_simple_77-38-0 (weather.get): city='London', country='GB' (from UK), units/forecast default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 590 OUT302 |
Tool Calling - Simple: Live Media, Weather & Data (set 3)
10 passed |
100.0% |
IN71 612 OUT4 002 | |
| ✓ | SimpleExtractMillennialsAppleBFCL/live_simple_71-35-0 (extract_parameters_v1): demographics=[millennials], targets=[brand:Apple], metrics=[view], min_date=2022-07-01. NOTE: targets @param clarified to '<attribute>:<entity>' order (was ambiguous 'target:target_type', which made the model emit the reversed 'Apple:brand'); GT 'brand:Apple' kept strict.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 175 OUT1 013 |
| ✓ | SimpleFetchPersonId4BFCL/live_simple_91-52-0 (fetchPerson): id=4.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 137 OUT284 |
| ✓ | SimpleLogFoodChaiTeaBFCL/live_simple_88-49-0 (log_food): food_name='chai tea', portion_amount=16.0, portion_unit='ounces', meal_name='snack'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 205 OUT291 |
| ✓ | SimplePlaySongChaChaChaBFCL/live_simple_94-55-0 (play_song): artist_name='Käärijä', song_name='Cha Cha Cha'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 155 OUT194 |
| ✓ | SimpleSendEmailAndyForecastBFCL/live_simple_78-39-0 (send_email): to='andy@gorilla.ai', subject='Sales Forecast Request', body set, cc/bcc/attachments omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 203 OUT280 |
| ✓ | SimpleSitefinityNewsJulianTestingBFCL/live_simple_82-43-0 (sitefinity_create_contentitem): ContentItem='News', Title='Julian is testing', rest omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 171 OUT346 |
| ✓ | SimpleTemperatureParisFranceBFCL/live_simple_90-51-0 (temperature): location='Paris, France', units/time default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 129 OUT569 |
| ✓ | SimpleTodoManagerAddSleepBFCL/live_simple_57-26-1 (todo_manager.handle_action): type='add', content='go to sleep at 9 pm'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 142 OUT178 |
| ✓ | SimpleTodoManagerDeleteGymBFCL/live_simple_56-26-0 (todo_manager.handle_action): type='delete', content='go to gym'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 132 OUT152 |
| ✓ | SimpleTranslateEnglishFrenchBFCL/live_simple_76-37-0 (language_translator.translate): source='en', target='fr', text='What is your name?'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 163 OUT695 |
Tool Calling - Simple: Live Media, Weather & Data (set 4)
10 passed |
100.0% |
IN53 296 OUT2 744 | |
| ✓ | SimpleOpenMeteoBerkeleyBFCL/live_simple_85-46-0 (open_meteo_api.fetch_weather_data): coordinates=[37.8715,-122.273], units default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 344 OUT560 |
| ✓ | SimpleSearchHotelsSanDiegoBFCL/live_simple_79-40-0 (search_hotels): location='San Diego, CA'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 384 OUT445 |
| ✓ | SimpleSitefinityNewsSleepProductivityBFCL/live_simple_83-44-0 (sitefinity_create_contentitem): full news article with all metadata.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 478 OUT334 |
| ✓ | SimpleSortArrayAscendingBFCL/live_simple_86-47-0 (sort_array): array=[1,54,3,1,76,2343,21], reverse default/omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 354 OUT289 |
| ✓ | SimpleTodoAddGoToGoaBFCL/live_simple_64-29-4 (todo): type='add', content='go to Goa'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 281 OUT228 |
| ✓ | SimpleTodoAddGymTomorrowBFCL/live_simple_61-29-1 (todo): type='add', content='go to gym tomorrow.'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 280 OUT142 |
| ✓ | SimpleTodoAddHiCharlieBFCL/live_simple_65-29-5 (todo): type='add', content='Hi charlie'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 291 OUT211 |
| ✓ | SimpleTodoAddSleep9pmBFCL/live_simple_63-29-3 (todo): type='add', content='go to sleep at 9 pm'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 287 OUT170 |
| ✓ | SimpleTodoCompleteFinancialReportBFCL/live_simple_60-29-0 (todo): type='complete', content='Submit monthly financial report'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 304 OUT222 |
| ✓ | SimpleTodoDeleteRaviBFCL/live_simple_62-29-2 (todo): type='delete', content='ravi'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 293 OUT143 |
Tool Calling - Simple: Live Movies, Music & Booking (set 1)
3 passed |
100.0% |
IN14 588 OUT852 | |
| ✓ | SimpleMoviesTarantinoWhitakerBFCL/live_simple_208-117-0 (Movies_3_FindMovies): directed_by='Quentin Tarantino', cast='Duane Whitaker', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 877 OUT509 |
| ✓ | SimplePlaySongRoarBFCL/live_simple_230-121-0 (play_song): song_id='wjeiruhuq_roar', artist='Katy Perry', quality='high', device_id omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 883 OUT222 |
| ✓ | SimpleSetVolume20BFCL/live_simple_242-126-0 (set_volume): volume=20.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 828 OUT121 |
Tool Calling - Simple: Live Movies, Music & Booking (set 10)
3 passed |
100.0% |
IN15 589 OUT1 383 | |
| ✓ | SimpleHandoverPackageArrivalBFCL/live_simple_257-137-1 (answer_question): statement=package-arrival-time question, urgency/language default. System instructs HANDOVER on question. NOTE: no collider loop observed (single clean call, sibling tests pass); statement is free text -> substring-intent of the key phrase (model drops "Can you tell me").1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 208 OUT468 |
| ✓ | SimpleMoviesNiteshDirectorBFCL/live_simple_217-117-9 (Movies_3_FindMovies): directed_by='Nitesh Tiwari', genre/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 168 OUT642 |
| ✓ | SimpleTtsChineseMaleBFCL/live_simple_227-118-1 (text_to_speech.convert): text=Chinese phrase, language='zh-CN', gender='male', format='wav', speed default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 213 OUT273 |
Tool Calling - Simple: Live Movies, Music & Booking (set 11)
3 passed |
100.0% |
IN14 947 OUT914 | |
| ✓ | SimpleMoviesColaizzoWatkinsBFCL/live_simple_218-117-10 (Movies_3_FindMovies): directed_by='Paul Downs Colaizzo', cast='Michaela Watkins', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 036 OUT451 |
| ✓ | SimpleRequestsGetBuProductsBFCL/live_simple_229-120-0 (requests.get): anchor='user'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 954 OUT230 |
| ✓ | SimpleTtsPrettyGirlDefaultBFCL/live_simple_228-119-0 (text_to_speech.convert): text='I am a pretty girl', language/gender/speed default.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 957 OUT233 |
Tool Calling - Simple: Live Movies, Music & Booking (set 12)
3 passed |
100.0% |
IN14 662 OUT1 170 | |
| ✓ | SimpleBookFlightParisBFCL/live_simple_233-123-0 (book_flight): departure_location='Paris, France', departure_time='12/03/2023 15:00', return_time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 923 OUT390 |
| ✓ | SimpleGetVersionBFCL/live_simple_247-129-0 (version_api.VersionApi.get_version): no arguments.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 835 OUT458 |
| ✓ | SimpleMoviesSimonCurtisDramaBFCL/live_simple_219-117-11 (Movies_3_FindMovies): directed_by='Simon Curtis', genre='Drama', cast='McKinley Belcher III'.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 904 OUT322 |
Tool Calling - Simple: Live Movies, Music & Booking (set 13)
3 passed |
100.0% |
IN15 199 OUT1 861 | |
| ✓ | SimpleAclRetrieveProjectsBFCL/live_simple_248-130-0 (acl_api.AclApi.retrieve_projects): uuid='12', excludeInactive=true, onlyRoot=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 054 OUT625 |
| ✓ | SimpleBookFlightNewYorkToParisBFCL/live_simple_234-123-1 (book_flight): departure_location='New York, NY', departure_time='20/06/2022 17:00', return_time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 104 OUT602 |
| ✓ | SimpleMoviesMackendrickThrillerBFCL/live_simple_220-117-12 (Movies_3_FindMovies): directed_by='Alexander Mackendrick', genre='Thriller', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 041 OUT634 |
Tool Calling - Simple: Live Movies, Music & Booking (set 14)
3 passed |
100.0% |
IN14 821 OUT1 053 | |
| ✓ | SimpleBookFlightJfkBFCL/live_simple_245-127-0 (book_flight): departure_location='JFK', departure_date='16/03/2024', departure_time='14:00'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 995 OUT365 |
| ✓ | SimpleChatWithBunnyBFCL/live_simple_249-131-0 (chat_with_friend): user_message='Hey bunny, How are you doing'. NOTE: no collider loop observed (single clean call, sibling tests pass); user_message is a free-text passthrough -> substring-intent (model may drop the "Hey bunny," greeting).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 910 OUT162 |
| ✓ | SimpleMoviesJamesGrayBFCL/live_simple_221-117-13 (Movies_3_FindMovies): directed_by='James Gray', genre/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 916 OUT526 |
Tool Calling - Simple: Live Movies, Music & Booking (set 15)
2 passed |
100.0% |
IN9 291 OUT1 374 | |
| ✓ | SimpleBookFlightNewYorkMarchBFCL/live_simple_246-128-0 (book_flight): departure_location='New York, NY', departure_time='15/03/2024'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 642 OUT687 |
| ✓ | SimpleMoviesVanOrmanBrownBFCL/live_simple_222-117-14 (Movies_3_FindMovies): directed_by='Thurop Van Orman', cast='Sterling K. Brown', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 649 OUT687 |
Tool Calling - Simple: Live Movies, Music & Booking (set 16)
1 passed1 skipped |
100.0% |
IN4 598 OUT367 | |
| ✓ | SimpleMoviesImaginativeFantasyBFCL/live_simple_223-117-15 (Movies_3_FindMovies): genre='Fantasy' ('Imaginative fiction'), director/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 598 OUT367 |
| – | SimpleRescheduleGymSessionBFCL/live_simple_231-122-0 (reschedule_event): event_identifier='456123', new_datetime='2022-10-30T16:30:00Z'.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: source flaw: relative-date arithmetic is inconsistent — prompt states 'Monday 2022-10-20' (that date is actually a Thursday) and the GT 'next Thursday' resolves to 2022-10-30 (a Sunday), so the exact new_datetime cannot be derived deterministically. |
Skipped | IN0 OUT0 |
Tool Calling - Simple: Live Movies, Music & Booking (set 17)
2 passed |
100.0% |
IN9 266 OUT991 | |
| ✓ | SimpleMoviesTimBurtonDeppBFCL/live_simple_224-117-16 (Movies_3_FindMovies): directed_by='Tim Burton', genre='Offbeat', cast='Johnny Depp'.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 589 OUT667 |
| ✓ | SimpleRescheduleMedAppointmentBFCL/live_simple_232-122-1 (reschedule_event): event_identifier='med123', new_datetime='2021-04-15T13:45:00Z'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 677 OUT324 |
Tool Calling - Simple: Live Movies, Music & Booking (set 18)
2 passed |
100.0% |
IN8 954 OUT474 | |
| ✓ | SimpleMoviesKirillComedyDramaBFCL/live_simple_225-117-17 (Movies_3_FindMovies): directed_by='Kirill Mikhanovsky', genre='Comedy-drama', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 512 OUT311 |
| ✓ | SimpleSetVolume70BFCL/live_simple_244-126-2 (set_volume): volume=70.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 442 OUT163 |
Tool Calling - Simple: Live Movies, Music & Booking (set 2)
3 passed |
100.0% |
IN14 633 OUT885 | |
| ✓ | SimpleMoviesLeitchTuisanoBFCL/live_simple_209-117-1 (Movies_3_FindMovies): directed_by='David Leitch', cast='Lori Pelenise Tuisano', genre dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 902 OUT504 |
| ✓ | SimpleSetVolume30BFCL/live_simple_243-126-1 (set_volume): volume=30.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 864 OUT129 |
| ✓ | SimpleSpotifyFriendsMarshmelloBFCL/live_simple_235-124-0 (play_spotify_song): query='track:Friends artist:Marshmello', shuffle/volume default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 867 OUT252 |
Tool Calling - Simple: Live Movies, Music & Booking (set 3)
3 passed |
100.0% |
IN15 207 OUT947 | |
| ✓ | SimpleFlightStatusIndigo6E123BFCL/live_simple_251-133-0 (flight.status.check): flight_id='6E123', passenger_name='Karan', ticket_number='IND4567'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 102 OUT498 |
| ✓ | SimpleMoviesDramaBFCL/live_simple_210-117-2 (Movies_3_FindMovies): genre='Drama', director/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 052 OUT206 |
| ✓ | SimpleSpotifyDilNu124BFCL/live_simple_236-124-1 (play_spotify_song): query='track:Dil Nu artist:Maninder Buttar', shuffle/volume default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 053 OUT243 |
Tool Calling - Simple: Live Movies, Music & Booking (set 4)
3 passed |
100.0% |
IN15 203 OUT1 192 | |
| ✓ | SimpleFlightStatusIndigoAirlineBFCL/live_simple_252-134-0 (flight.status.check): airlines='Indigo', person_name='Karan', ticket_number='IND4567'. NOTE: tool doc says "if only one name mentioned ... use LNU", so "Karan LNU" is equivalent-correct per the documented convention -> accept both.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 096 OUT550 |
| ✓ | SimpleMoviesComedyCordenBFCL/live_simple_211-117-3 (Movies_3_FindMovies): genre='Comedy' (prompt 'Comic'), cast='James Corden', director dontcare. NOTE: args are correct; single-call contract stays strict. The 27B occasionally emits a 2nd call (over-call) -> legit intermittent fail (kept as signal).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 072 OUT396 |
| ✓ | SimpleSpotifyWreckingBallBFCL/live_simple_237-125-0 (play_spotify_song): query='track:Wrecking Ball artist:Miley Cyrus', device_id/play default. NOTE: GT accepts only the structured 'track:.. artist:..' form (kept strict). The 27B occasionally drops the format (e.g. "wrecking ball Miley Cyrus") -> legit intermittent miss (kept as signal).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 035 OUT246 |
Tool Calling - Simple: Live Movies, Music & Booking (set 5)
3 passed |
100.0% |
IN15 140 OUT1 119 | |
| ✓ | SimpleMoviesComedyEdgarWrightBFCL/live_simple_212-117-4 (Movies_3_FindMovies): directed_by='Edgar Wright', genre='Comedy', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 037 OUT666 |
| ✓ | SimpleSpotifyDilNu125BFCL/live_simple_238-125-1 (play_spotify_song): query='track:Dil Nu artist:Maninder Buttar', device_id/play default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 037 OUT212 |
| ✓ | SimpleUberEatsBigMacBFCL/live_simple_250-132-0 (uber.eat.order): restaurants='McDonald's', items=[Big Mac, McFlurry], quantities=[2,1].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 066 OUT241 |
Tool Calling - Simple: Live Movies, Music & Booking (set 6)
3 passed |
100.0% |
IN15 151 OUT1 317 | |
| ✓ | SimpleMoviesOffbeatTimBurtonBFCL/live_simple_213-117-5 (Movies_3_FindMovies): directed_by='Tim Burton', genre='Offbeat', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 047 OUT890 |
| ✓ | SimpleSpotifyShapeOfYouBFCL/live_simple_239-125-2 (play_spotify_song): query='track:Shape of You artist:Ed Sheeran' or 'Shape of You'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 036 OUT195 |
| ✓ | SimpleUberEatsBurgersWingsBFCL/live_simple_253-135-0 (uber.eat.order): restaurants='McDonald's', items=[burgers, chicken wings], quantities=[5,6].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 068 OUT232 |
Tool Calling - Simple: Live Movies, Music & Booking (set 7)
3 passed |
100.0% |
IN14 969 OUT1 178 | |
| ✓ | SimpleMoviesComicNiteshTiwariBFCL/live_simple_214-117-6 (Movies_3_FindMovies): directed_by='Nitesh Tiwari', genre='Comedy' or 'Comedy-drama', cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 048 OUT639 |
| ✓ | SimpleSpotifyBabySharkBFCL/live_simple_240-125-3 (play_spotify_song): query='Baby Shark' (artist unknown).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 949 OUT241 |
| ✓ | SimpleUserMandatesPendingBFCL/live_simple_254-136-0 (user.mandates): user_id='U123', status='pending'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 972 OUT298 |
Tool Calling - Simple: Live Movies, Music & Booking (set 8)
3 passed |
100.0% |
IN14 918 OUT1 321 | |
| ✓ | SimpleMoviesFantasyTonightBFCL/live_simple_215-117-7 (Movies_3_FindMovies): genre='Fantasy', director/cast dontcare.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 988 OUT732 |
| ✓ | SimpleSpotifyJohnnyPapaBFCL/live_simple_241-125-4 (play_spotify_song): query='Johnny Johnny Yes Papa' (artist unknown).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 945 OUT197 |
| ✓ | SimpleUserMandatesActiveBFCL/live_simple_255-136-1 (user.mandates): user_id='U123456', status='active' or omitted (default).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 985 OUT392 |
Tool Calling - Simple: Live Movies, Music & Booking (set 9)
3 passed |
100.0% |
IN15 568 OUT1 132 | |
| ✓ | SimpleHandoverComplaintStatusBFCL/live_simple_256-137-0 (answer_question): statement=user complaint-status question, urgency/language default. System instructs HANDOVER on question.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 209 OUT394 |
| ✓ | SimpleMoviesActionLeitchKingBFCL/live_simple_216-117-8 (Movies_3_FindMovies): directed_by='David Leitch', genre='Action', cast='Alex King'.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 176 OUT273 |
| ✓ | SimpleTtsPrettyGirlSpanishBFCL/live_simple_226-118-0 (text_to_speech.convert): text='I am a pretty girl' or Spanish translation, language='es-ES', gender/format/speed default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 183 OUT465 |
Tool Calling - Simple: Live Services, Weather & Commands (set 1)
4 passed |
100.0% |
IN18 920 OUT1 530 | |
| ✓ | SimpleExtractChesterJaneBFCL/live_simple_189-114-0 (extractor.extract_information): [Chester,42],[Jane,43] (Jane a year older).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 766 OUT388 |
| ✓ | SimpleServiceIdIroningUnitIntBFCL/live_simple_174-100-0 (get_service_id): ironing service -> service_id=2, unit omitted/1. Thai prompt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 727 OUT373 |
| ✓ | SimpleWeatherNewYorkBFCL/live_simple_193-116-1 (Weather_1_GetWeather): city=New York, date=2023-03-08.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 713 OUT370 |
| ✓ | SimpleWeatherPacificaBFCL/live_simple_192-116-0 (Weather_1_GetWeather): city=Pacifica, date=2023-04-11.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 714 OUT399 |
Tool Calling - Simple: Live Services, Weather & Commands (set 10)
8 passed1 failed |
88.9% |
IN47 823 OUT3 408 | |
| ✓ | SimpleCmdDockerPsBFCL/live_simple_172-99-6 (cmd_controller.execute): is Docker running -> 'docker ps'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 573 OUT167 |
| ✓ | SimpleCmdDockerStartBFCL/live_simple_168-99-2 (cmd_controller.execute): start Docker -> 'docker start'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 573 OUT177 |
| ✓ | SimpleCmdEchoDateBFCL/live_simple_171-99-5 (cmd_controller.execute): today's date -> 'echo %date%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 582 OUT160 |
| ✓ | SimpleCmdEchoTimeBFCL/live_simple_166-99-0 (cmd_controller.execute): current time -> 'echo %time%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 594 OUT344 |
| × | SimpleCmdListCDriveBFCL/live_simple_170-99-4 (cmd_controller.execute): list C drive -> 'dir C:\'.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 916 OUT701 |
| ✓ | SimpleCmdRunPythonScriptBFCL/live_simple_169-99-3 (cmd_controller.execute): run python script with venv interpreter.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 656 OUT244 |
| ✓ | SimpleCmdStartCalcBFCL/live_simple_173-99-7 (cmd_controller.execute): open calculator -> 'start calc'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 575 OUT124 |
| ✓ | SimpleCmdTasklistCodeBFCL/live_simple_167-99-1 (cmd_controller.execute): is VS Code running -> tasklist filter.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 606 OUT862 |
| ✓ | SimpleProvidersChonburiDateRangeBFCL/live_simple_186-111-0 (get_service_providers): ironing, Chonburi, rating>=4.5, 12/03/2024 16:00-18:00.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 748 OUT629 |
Tool Calling - Simple: Live Services, Weather & Commands (set 11)
3 passed1 failed |
75.0% |
IN19 659 OUT2 773 | |
| ✓ | SimpleItemsCircleLogosBFCL/live_simple_162-96-1 (get_items): 4 circle-logo copyright items.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 814 OUT458 |
| ✓ | SimpleItemsLegoMovieBFCL/live_simple_163-96-2 (get_items): 7 Lego-movie copyright items.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 921 OUT377 |
| ✓ | SimpleItemsSuperheroLogosBFCL/live_simple_161-96-0 (get_items): 12 superhero-logo copyright items.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 055 OUT1 137 |
| × | SimpleProvidersChonburiNoLateBFCL/live_simple_187-112-0 (get_service_providers): ironing, Chonburi, rating>=4.5, 12/03/2024, never late. NOTE: the prompt gives a time range "16:00 - 18:00", so end_available_date is expected; the 27B occasionally omits it (~1/3 of runs) -> a legit model-reliability miss, not an assertion flaw.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.end_available_date ?? "").AnyOfContains(["2024-03-12", "2024-12-03"]) |
Failed | IN4 869 OUT801 |
Tool Calling - Simple: Live Services, Weather & Commands (set 12)
2 passed |
100.0% |
IN9 447 OUT797 | |
| ✓ | SimpleCopyrightCocaColaBFCL/live_simple_164-97-0 (get_copyright_info): Coca-Cola logo, The Coca-Cola Company, confidence 100.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 876 OUT282 |
| ✓ | SimpleProfessionalCleaningRating2BFCL/live_simple_183-108-0 (getDataForProfessional): cleaning in Bangkok, rating>=2.0 -> avg_rating=2.0, province_id=1, service_id=1.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 571 OUT515 |
Tool Calling - Simple: Live Services, Weather & Commands (set 13)
3 passed |
100.0% |
IN13 680 OUT1 381 | |
| ✓ | SimpleMpnSpecs3pak7BFCL/live_simple_191-115-1 (raptor.mpn.specs): identifier=3pak7, MPN, include_images omitted/false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 549 OUT617 |
| ✓ | SimpleMpnSpecsAchdImagesBFCL/live_simple_190-115-0 (raptor.mpn.specs): identifier=ACHD, MPN, include_images=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 569 OUT387 |
| ✓ | SimpleProfessionalCleaningRating4BFCL/live_simple_184-109-0 (getDataForProfessional): cleaning in Bangkok, rating>4 -> service_id=1, province_id=1, rating=4.0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 562 OUT377 |
Tool Calling - Simple: Live Services, Weather & Commands (set 14)
2 passed |
100.0% |
IN9 757 OUT831 | |
| ✓ | SimpleExtractChineseNamesBFCL/live_simple_165-98-0 (extractor.extract_information): Chinese text -> [李雷,18],[李丽,21] (sister 3 yrs older).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 878 OUT399 |
| ✓ | SimpleProfessionalIroningChiangMaiBFCL/live_simple_188-113-0 (getDataForProfessional): ironing in Chiang Mai, Sankampang/Ton Pao, rating 4.5.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 879 OUT432 |
Tool Calling - Simple: Live Services, Weather & Commands (set 2)
3 passed |
100.0% |
IN13 110 OUT1 374 | |
| ✓ | SimpleServiceIdIroningUnitSessionBFCL/live_simple_175-101-0 (get_service_id): ironing -> service_id=2, unit omitted/session. Thai prompt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 353 OUT505 |
| ✓ | SimpleWeatherMartinezBFCL/live_simple_194-116-2 (Weather_1_GetWeather): city=Martinez, date=2023-04-25.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 387 OUT410 |
| ✓ | SimpleWeatherPaloAltoBFCL/live_simple_195-116-3 (Weather_1_GetWeather): city=Palo Alto, date=2023-04-25.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 370 OUT459 |
Tool Calling - Simple: Live Services, Weather & Commands (set 3)
3 passed |
100.0% |
IN13 072 OUT1 153 | |
| ✓ | SimpleServiceIdIroningUnitQtyBFCL/live_simple_176-102-0 (get_service_id): ironing -> service_id=2, unit omitted/1. Thai prompt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 339 OUT305 |
| ✓ | SimpleWeatherAlamedaWednesdayBFCL/live_simple_196-116-4 (Weather_1_GetWeather): city=Alameda, upcoming Wednesday (today Mon 2023-04-25) -> 2023-04-27.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 369 OUT410 |
| ✓ | SimpleWeatherStinsonBeachBFCL/live_simple_197-116-5 (Weather_1_GetWeather): city=Stinson Beach, date=2023-04-05.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 364 OUT438 |
Tool Calling - Simple: Live Services, Weather & Commands (set 4)
4 passed |
100.0% |
IN17 403 OUT1 668 | |
| ✓ | SimpleServiceIdCleaningBFCL/live_simple_177-103-0 (get_service_id): cleaning -> service_id=1, unit omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 325 OUT260 |
| ✓ | SimpleServiceIdIroningBFCL/live_simple_178-103-1 (get_service_id): ironing -> service_id=2, unit omitted/1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 327 OUT347 |
| ✓ | SimpleWeatherHealdsburgBFCL/live_simple_198-116-6 (Weather_1_GetWeather): city=Healdsburg, date=2023-03-02.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 380 OUT409 |
| ✓ | SimpleWeatherMarshallBFCL/live_simple_199-116-7 (Weather_1_GetWeather): city=Marshall (prompt 'Marshall, MN'), date=2023-03-05. NOTE: GT city is bare "Marshall"; model faithfully copies the "Marshall, MN" disambiguator from the prompt -> substring-intent (equivalent-correct).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 371 OUT652 |
Tool Calling - Simple: Live Services, Weather & Commands (set 5)
3 passed |
100.0% |
IN13 209 OUT1 369 | |
| ✓ | SimpleServiceIdIroningChiangMaiBFCL/live_simple_179-104-0 (get_service_id): ironing in Chiang Mai -> service_id=2, province_id=2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 374 OUT469 |
| ✓ | SimpleWeatherCampbellBFCL/live_simple_201-116-9 (Weather_1_GetWeather): city=Campbell, date=2023-03-04.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 431 OUT430 |
| ✓ | SimpleWeatherFremontBFCL/live_simple_200-116-8 (Weather_1_GetWeather): city=Fremont, date=2023-03-01.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 404 OUT470 |
Tool Calling - Simple: Live Services, Weather & Commands (set 6)
3 passed |
100.0% |
IN13 396 OUT1 234 | |
| ✓ | SimpleServiceIdIroningBangkokRatingBFCL/live_simple_180-105-0 (get_service_id): ironing in Bangkok, rating>=4 -> service_id=2, province_id=1, rating=4.0.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 468 OUT476 |
| ✓ | SimpleWeatherFosterCityBFCL/live_simple_202-116-10 (Weather_1_GetWeather): city=Foster City, date=2023-04-25.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 461 OUT295 |
| ✓ | SimpleWeatherWashingtonBFCL/live_simple_203-116-11 (Weather_1_GetWeather): city=Washington (prompt 'Washington, DC'), date=2023-03-01. NOTE: GT city is bare "Washington"; model faithfully copies the "Washington, DC" disambiguator from the prompt -> substring-intent (equivalent-correct).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 467 OUT463 |
Tool Calling - Simple: Live Services, Weather & Commands (set 7)
2 passed1 failed |
66.7% |
IN13 722 OUT1 603 | |
| × | SimpleServiceIdIroningChatuchakBFCL/live_simple_181-106-0 (get_service_id): ironing, Bangkok(Chatuchak), rating>=4.5 -> service_id=2, province_id omitted/1, district_name=Chatuchak, rating=4.5.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: (args.district_name?.ToLower()).AnyOf(["chatuchak"]) |
Failed | IN4 586 OUT477 |
| ✓ | SimpleWeatherBerkeleySaturdayBFCL/live_simple_205-116-13 (Weather_1_GetWeather): city=Berkeley, upcoming Saturday (today Tue 2023-04-25) -> 2023-04-29.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 564 OUT625 |
| ✓ | SimpleWeatherRutherfordBFCL/live_simple_204-116-12 (Weather_1_GetWeather): city=Rutherford (prompt 'Rutherford, NJ'), date=2023-04-22. NOTE: GT city is bare "Rutherford"; model faithfully copies the "Rutherford, NJ" disambiguator from the prompt -> substring-intent (equivalent-correct).1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 572 OUT501 |
Tool Calling - Simple: Live Services, Weather & Commands (set 8)
3 passed |
100.0% |
IN13 820 OUT1 597 | |
| ✓ | SimpleProvidersKhlongSamWaBFCL/live_simple_182-107-0 (get_service_providers): ironing, Bangkok, Khlong Sam Wa / Sai Kong, rating>=4.5.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 639 OUT498 |
| ✓ | SimpleWeatherLondonBFCL/live_simple_206-116-14 (Weather_1_GetWeather): city=London (prompt 'London, England'), date=2023-03-05.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 591 OUT634 |
| ✓ | SimpleWeatherSacramentoBFCL/live_simple_207-116-15 (Weather_1_GetWeather): city=Sacramento, date=2023-04-22.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 590 OUT465 |
Tool Calling - Simple: Live Services, Weather & Commands (set 9)
5 passed1 failed |
83.3% |
IN34 206 OUT2 610 | |
| ✓ | SimpleCmdEchoTimeUnitBFCL/live_simple_157-95-14 (cmd_controller.execute): current time -> 'echo %time%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 619 OUT341 |
| × | SimpleCmdListCDriveUnitBFCL/live_simple_156-95-13 (cmd_controller.execute): list C drive -> 'dir C:\'.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 008 OUT945 |
| ✓ | SimpleCmdStartCameraBFCL/live_simple_160-95-17 (cmd_controller.execute): open camera -> 'start microsoft.windows.camera:'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 614 OUT195 |
| ✓ | SimpleCmdTaskkillCodeBFCL/live_simple_158-95-15 (cmd_controller.execute): kill VS Code -> 'taskkill /IM code.exe /F'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 614 OUT176 |
| ✓ | SimpleCmdTimeout10BFCL/live_simple_159-95-16 (cmd_controller.execute): timer 10s -> 'timeout 10'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 634 OUT204 |
| ✓ | SimpleProvidersChonburiTomorrowBFCL/live_simple_185-110-0 (get_service_providers): ironing, Chonburi, rating>=4.5, tomorrow of 20230929 -> 2023-09-30.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN4 717 OUT749 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 1)
3 passed |
100.0% |
IN13 884 OUT1 291 | |
| ✓ | SimpleClassifyGreetingsAndLinksBFCL/live_simple_105-62-0 (classify): clean_hello=['hello'], faq_link_accounts_start=['link my account','connect my accounts'].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 611 OUT314 |
| ✓ | SimpleCmdDockerVersionBFCL/live_simple_141-94-0 (cmd_controller.execute): command='docker --version'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 576 OUT233 |
| ✓ | SimpleReqNodesPekAciBFCL/live_simple_128-83-0 (requests.get): nodes list url with ip 192.168.11.33, fabricName='PEK-ACI'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 697 OUT744 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 10)
4 passed |
100.0% |
IN19 605 OUT1 316 | |
| ✓ | SimpleCmdShutdownBFCL/live_simple_150-95-7 (cmd_controller.execute): shutdown the pc via 'shutdown /s /t 0'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 866 OUT200 |
| ✓ | SimpleProcessImageMoondreamBFCL/live_simple_122-78-0 (process_data): image.png + long question + model 'vikhyatk/moondream2'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 978 OUT322 |
| ✓ | SimpleReqDataLimit50BFCL/live_simple_135-88-0 (requests.get): data endpoint url, params limit=50.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 897 OUT193 |
| ✓ | SimpleSumNumbersListBFCL/live_simple_116-72-0 (sum_numbers): numbers_list=[133, 34].1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 864 OUT601 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 11)
4 passed |
100.0% |
IN20 023 OUT2 930 | |
| ✓ | SimpleCmdWifiDisconnectBFCL/live_simple_151-95-8 (cmd_controller.execute): disconnect wifi via 'netsh wlan disconnect'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 893 OUT201 |
| ✓ | SimplePipelineFurnitureImageBFCL/live_simple_123-79-0 (pipeline): analyze thewowstyle image with furniture/minimalist question.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 169 OUT991 |
| ✓ | SimpleReqTopologySalesAppBFCL/live_simple_136-89-0 (requests.get): topology url ip 192.168.1.1, filter accountName:AcmeCorp AND applicationName:SalesApp.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 050 OUT377 |
| ✓ | SimpleTemperatureHyderabadBFCL/live_simple_118-74-0 (get_temperature): units=Celsius, location='Hyderabad, India', time omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 911 OUT1 361 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 12)
4 passed |
100.0% |
IN21 311 OUT1 450 | |
| ✓ | SimpleCmdDirDAndEchoBFCL/live_simple_152-95-9 (cmd_controller.execute): 'dir D:\ && echo testing.txt'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 272 OUT273 |
| ✓ | SimpleReqAnomaliesSummaryBFCL/live_simple_139-92-0 (requests.get): anomalies summary url ip 192.168.1.1, params fabric='network222'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 399 OUT561 |
| ✓ | SimpleRescheduleAlice8pmBFCL/live_simple_137-90-0 (reschedule): identifier='Alice-One-one-One', 2023-11-01T20:00:00, Europe/London.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 306 OUT224 |
| ✓ | SimpleWeatherForecastParisBFCL/live_simple_120-76-0 (weather.forecast): Paris, France 2023-04-03..05, Celsius, defaults omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 334 OUT392 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 13)
2 passed1 failed |
66.7% |
IN17 443 OUT1 925 | |
| ✓ | SimpleCmdDelPuttyBFCL/live_simple_153-95-10 (cmd_controller.execute): remove putty shortcut via del with quoted path.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 818 OUT205 |
| × | SimpleDynamicPricingSaaSBFCL/live_simple_113-69-0 (calculate_dynamic_pricing): geolocation='34.0522, -118.2437', base=100, min=90.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 772 OUT1 353 |
| ✓ | SimpleRescheduleBob4pmBFCL/live_simple_138-91-0 (reschedule): identifier='Bob-123', 2023-11-01T16:00:00, Europe/London.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 853 OUT367 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 14)
3 passed |
100.0% |
IN14 094 OUT1 090 | |
| ✓ | SimpleCmdStartYoutubeBFCL/live_simple_154-95-11 (cmd_controller.execute): open youtube rickroll via start command.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 711 OUT348 |
| ✓ | SimpleGetTicketsCompanyOpenBFCL/live_simple_124-80-0 (get_tickets): customer (company name in prompt), status=open, limit=10.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 696 OUT376 |
| ✓ | SimpleGreetPartnerJeffBFCL/live_simple_140-93-0 (greet_partner): user='Emily', partner='Jeff', relationship='husband'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 687 OUT366 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 15)
2 passed1 failed |
66.7% |
IN14 467 OUT1 213 | |
| ✓ | SimpleCalculateTaxNewYorkBFCL/live_simple_104-61-2 (calculate_tax): purchase_amount=250, state='NY', all optionals omitted/default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 832 OUT582 |
| ✓ | SimpleCmdDateBFCL/live_simple_155-95-12 (cmd_controller.execute): what is the date via 'date /t' or 'echo %date%'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 805 OUT378 |
| × | SimpleConcatJohnDoeBFCL/live_simple_125-81-0 (concat_strings): strings=['John','Doe'], separator=' '. EXPECTED MODEL MISS: 'combine ... into a full name' requires separator=' ' (John Doe). The 27B tends to omit separator (=> 'JohnDoe'); that is a legit model failure of the measured competence, so the ' ' assertion stays strict.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.separator.AnyOf([" "])
Values:
args.separator = null |
Failed | IN4 830 OUT253 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 2)
3 passed |
100.0% |
IN16 106 OUT2 277 | |
| ✓ | SimpleCmdDirDesktopBFCL/live_simple_142-94-1 (cmd_controller.execute): command='dir Desktop'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 041 OUT150 |
| ✓ | SimpleRecordBankNineIntentsBFCL/live_simple_106-63-0 (record): classify 20 bank queries into 9 intents (2 buckets empty).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 963 OUT1 391 |
| ✓ | SimpleReqNodesFabEdBFCL/live_simple_129-83-1 (requests.get): nodes list url with ip 3.321.3232.2, fabricName='fab-ed'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 102 OUT736 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 3)
4 passed |
100.0% |
IN21 436 OUT1 469 | |
| ✓ | SimpleCmdDockerPsBFCL/live_simple_143-95-0 (cmd_controller.execute): command='docker ps'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 217 OUT131 |
| ✓ | SimpleRecordTelephoneHoursAccountsBFCL/live_simple_107-64-0 (record): classify 7 queries into telephone-banking / bank-hours / accounts.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 617 OUT722 |
| ✓ | SimpleTelemetryGlobalFabricStatusBFCL/live_simple_126-82-0 (telemetry.flowrules.interfaceInfo.get): fabric='Global-Fabric', node=1200, pod=3, ethernet, status.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 299 OUT325 |
| ✓ | SimpleUpdateProfileEmailAgeBFCL/live_simple_114-70-0 (update_user_profile): user_id=12345, email+age set, bio omitted, notify default.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 303 OUT291 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 4)
2 passed2 failed |
50.0% |
IN23 523 OUT2 351 | |
| ✓ | SimpleCmdKillFirefoxBFCL/live_simple_144-95-1 (cmd_controller.execute): close firefox via taskkill.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 081 OUT167 |
| × | SimpleRecordMoneyTransactionsCreditBFCL/live_simple_108-65-0 (record): classify 10 queries into money-movement / transactions / credit-limit / link-accounts. EXPECTED MODEL MISS: the 27B classifies 'show me all outgoing wire transfers' as a transaction-view (get_transactions_start) rather than money_movement_start per GT, so the get_transactions count comes out as 5 instead of 4. GT bucketing kept strict; this is a legit model classification failure.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.money_movement_start.Count == 1
Values:
args.money_movement_start.Count = 0 |
Failed | IN5 520 OUT956 |
| × | SimpleReverseSayHiBFCL/live_simple_117-73-0 (reverse_input): input_value='say hi'.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 763 OUT885 |
| ✓ | SimpleTelemetryTestDeStatusBFCL/live_simple_127-82-1 (telemetry.flowrules.interfaceInfo.get): fabric='test-de', node=5, pod=1, ethernet, status.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 159 OUT343 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 5)
4 passed |
100.0% |
IN20 872 OUT2 276 | |
| ✓ | SimpleCmdDirCDriveBFCL/live_simple_145-95-2 (cmd_controller.execute): list C drive => 'dir C:\'. EXPECTED MODEL MISS: for the terse/typo'd 'LIST C DRIIVE' the 27B currently returns 'dir d:' — the wrong drive. The accepted-command list is broadened with the no-backslash 'dir c:' variant for fairness, but a wrong drive letter is a legit model failure.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 115 OUT716 |
| ✓ | SimplePrimeMinistersThreeBFCL/live_simple_119-75-0 (GetPrimeMinisters): countries=[Australia, Canada, India].1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 139 OUT421 |
| ✓ | SimpleRecordAutoWithdrawIncomeBFCL/live_simple_109-66-0 (record): classify 5 loan/withdraw queries; outofscope empty.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 347 OUT675 |
| ✓ | SimpleReqInterfaceStatsNetwork1BFCL/live_simple_130-84-0 (requests.get): interfaceInfo stats, ip 192.120.45.67, Network1/node 12/pod 10/ethernet/statistics.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 271 OUT464 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 6)
4 passed |
100.0% |
IN21 534 OUT1 955 | |
| ✓ | SimpleCmdStartHuggingfaceBFCL/live_simple_146-95-3 (cmd_controller.execute): open huggingface via start command.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 292 OUT535 |
| ✓ | SimpleGenerateSalaryChartBFCL/live_simple_121-77-0 (generate_chart): 5 job labels and salaries in order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 446 OUT404 |
| ✓ | SimpleRecordAtmLocationOnlyBFCL/live_simple_110-67-0 (record): single query 'Need ATM location' -> atm_finder_start only.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 291 OUT235 |
| ✓ | SimpleReqInterfaceSpeedNetwork1BFCL/live_simple_131-84-1 (requests.get): interfaceInfo Speed, ip 192.120.45.67, Network1/node 12/pod 10/ethernet/Speed.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 505 OUT781 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 7)
1 passed2 failed |
33.3% |
IN17 524 OUT1 555 | |
| × | SimpleCmdKillTimerBFCL/live_simple_147-95-4 (cmd_controller.execute): remove timer.exe via taskkill.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: args.command.ToLower().AnyOf(["taskkill /f /im timer.exe"]) |
Failed | IN4 865 OUT162 |
| ✓ | SimpleRecordBalanceAndAtmBFCL/live_simple_111-67-1 (record): 10 queries -> 8 balance + 2 atm; other buckets empty.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN5 172 OUT580 |
| × | SimpleReqDeviceConnectorVersionsBFCL/live_simple_132-85-0 (requests.get): device connector versions JSON url, no params.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN7 487 OUT813 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 8)
2 passed1 failed |
66.7% |
IN14 857 OUT1 228 | |
| × | SimpleCmdWhatFileInCBFCL/live_simple_148-95-5 (cmd_controller.execute): 'what is the file in c' => 'dir C:\'. EXPECTED MODEL MISS: the 27B mis-selects the 'record' classifier (dropping the request into its 'outofscope' bucket) instead of cmd_controller_execute. record is a legitimate distractor here (not a looping universal collider), so this stays as a real tool-selection failure — no isolation needed.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: call.ToolName == expectedTool
Values:
call.ToolName = "record"
expectedTool = "cmd_controller_execute" |
Failed | IN4 937 OUT442 |
| ✓ | SimpleRecordWeatherOutOfScopeBFCL/live_simple_112-68-0 (record): single query 'what is the weather like' -> outofscope only.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 958 OUT381 |
| ✓ | SimpleReqRecommendationsDacBFCL/live_simple_133-86-0 (requests.get): recommendations url, params advisoryId='dac'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 962 OUT405 |
Tool Calling - Simple: Live Shell, Classification & APIs (set 9)
3 passed |
100.0% |
IN13 871 OUT737 | |
| ✓ | SimpleCalculateSum133And34BFCL/live_simple_115-71-0 (calculate_sum): number1=133, number2=34.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 607 OUT280 |
| ✓ | SimpleCmdEchoHiBFCL/live_simple_149-95-6 (cmd_controller.execute): say hi via echo command.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN4 581 OUT174 |
| ✓ | SimpleReqInsightsGroupsBFCL/live_simple_134-87-0 (requests.get): insights groups url, params limit=20, sort='desc'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 683 OUT283 |
Tool Calling - Simple: Math, Physics & Geometry (set 1)
11 passed1 failed |
91.7% |
IN92 482 OUT8 115 | |
| ✓ | SimpleAlgebraQuadraticRootsBFCL/simple_python_3 (algebra.quadratic_roots): a=1, b=-3, c=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 395 OUT600 |
| × | SimpleAreaUnderCurveXSquaredBFCL/simple_python_13 (calculate_area_under_curve): function y=x^2, interval [1,3], method omitted/trapezoidal.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 252 OUT1 224 |
| ✓ | SimpleCircleCircumferenceInchesBFCL/simple_python_7 (calculate_circumference): radius=4, unit inches/in.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 376 OUT228 |
| ✓ | SimpleDerivativeCubicPolyBFCL/simple_python_14 (calculate_derivative): function 3x^2+2x-1, x_value omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 383 OUT414 |
| ✓ | SimpleElectricFieldFromChargeBFCL/simple_python_39 (calculate_electric_field): charge=2, distance=3, permitivity omitted/8.854e-12.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 384 OUT1 022 |
| ✓ | SimpleFinalVelocityFromDistanceBFCL/simple_python_30 (kinematics.final_velocity_from_distance): acceleration=4, distance=300, initial_velocity omitted/0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 411 OUT602 |
| ✓ | SimpleFinalVelocityFromHeightBFCL/simple_python_25 (calculate_final_velocity): height=150, initial_velocity omitted/0, gravity omitted/9.81.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 392 OUT1 008 |
| ✓ | SimpleGeometryAreaCircleBFCL/simple_python_8 (geometry.area_circle): radius=10, units omitted or 'meters'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 365 OUT377 |
| ✓ | SimpleGetPrimeFactors450BFCL/simple_python_17 (get_prime_factors): number=450, formatted omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 355 OUT662 |
| ✓ | SimpleMagneticFieldLoopBFCL/simple_python_40 (calculate_magnetic_field): current=5, radius=4, permeability omitted/12.57e10.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 396 OUT859 |
| ✓ | SimpleMathGcd40And50BFCL/simple_python_19 (math.gcd): num1=40, num2=50.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 379 OUT580 |
| ✓ | SimpleTriangleAreaWithUnitBFCL/simple_python_0 (calculate_triangle_area): base=10, height=5, unit omitted or 'units'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 394 OUT539 |
Tool Calling - Simple: Math, Physics & Geometry (set 2)
11 passed1 failed |
91.7% |
IN92 165 OUT8 647 | |
| ✓ | SimpleAverageVelocityBFCL/simple_python_26 (calculate_velocity): distance=50, duration=2, unit omitted or 'km/h'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 359 OUT284 |
| ✓ | SimpleCalculusDerivativeAtValueBFCL/simple_python_16 (calculus.derivative): function 2x^2, value=1, function_variable omitted or 'x'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 360 OUT716 |
| ✓ | SimpleElectricFieldStrengthBFCL/simple_python_44 (calculate_electric_field_strength): charge=0.01, distance=4, medium omitted or 'vacuum'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 360 OUT839 |
| ✓ | SimpleFinalVelocityConstAccelBFCL/simple_python_31 (calculate_final_velocity): initial_velocity=0, acceleration=9.8, time=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 397 OUT442 |
| ✓ | SimpleGeometryCalculateAreaCircleBFCL/simple_python_9 (geometry.calculate_area_circle): radius=5, unit omitted or 'units'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 334 OUT431 |
| ✓ | SimpleGeometryCircumferenceBFCL/simple_python_12 (geometry.circumference): radius=3, units omitted or 'cm'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 327 OUT539 |
| ✓ | SimpleIntegrateCubeSimpsonBFCL/simple_python_15 (integrate): function x^3, start_x=-2, end_x=3, method simpson.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 392 OUT648 |
| × | SimpleMagneticFieldStrengthWireBFCL/simple_python_43 (calculate_magnetic_field_strength): current=20, distance=10, permeability omitted/12.57e-7.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 216 OUT1 747 |
| ✓ | SimpleMathHcfBFCL/simple_python_20 (math.hcf): number1=36, number2=24.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 345 OUT650 |
| ✓ | SimpleNumberAnalysisPrimeFactorsBFCL/simple_python_18 (number_analysis.prime_factors): number=123456.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 341 OUT969 |
| ✓ | SimpleRightTriangleAreaCmBFCL/simple_python_10 (calculate_area): base=6, height=10, unit omitted or 'cm'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 376 OUT653 |
| ✓ | SimpleSolveQuadraticEquation265BFCL/simple_python_4 (solve_quadratic_equation): a=2, b=6, c=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 358 OUT729 |
Tool Calling - Simple: Math, Physics & Geometry (set 3)
8 passed1 failed |
88.9% |
IN63 701 OUT5 831 | |
| × | SimpleBoilingMeltingPointsBFCL/simple_python_47 (get_boiling_melting_points): substance='water', sea_level=5000.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 175 OUT1 340 |
| ✓ | SimpleDisplacementBFCL/simple_python_28 (calculate_displacement): initial_velocity=10, time=5, acceleration=9.8.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 702 OUT448 |
| ✓ | SimpleElectrostaticPotentialBFCL/simple_python_38 (calculate_electrostatic_potential): charge1=1e-9, charge2=2e-9, distance=0.05, constant omitted/8.99e9.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 716 OUT990 |
| ✓ | SimpleFinalSpeedFromHeightBFCL/simple_python_32 (calculate_final_speed): initial_velocity=0, height=100, gravity omitted or 9.8.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 672 OUT381 |
| ✓ | SimpleNumberTheoryGcdBFCL/simple_python_21 (number_theory.gcd): number1=36, number2=48.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 685 OUT478 |
| ✓ | SimplePrimeFactorizeDictionaryBFCL/simple_python_23 (prime_factorize): number=60, return_type='dictionary'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 676 OUT367 |
| ✓ | SimpleRouteEstimateTimeStopsBFCL/simple_python_37 (route.estimate_time): SF -> LA, stops {Santa Barbara, Monterey} in either order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 694 OUT495 |
| ✓ | SimpleSolveQuadraticAllRootsBFCL/simple_python_5 (solve_quadratic): a=3, b=-11, c=-4, root_type='all' (user asked for all roots).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 705 OUT968 |
| ✓ | SimpleTriangleAreaNoUnitBFCL/simple_python_11 (calculate_triangle_area): base=10, height=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 676 OUT364 |
Tool Calling - Simple: Math, Physics & Geometry (set 4)
9 passed |
100.0% |
IN59 738 OUT6 661 | |
| ✓ | SimpleCalculateDensityBFCL/simple_python_48 (calculate_density): mass=45, volume=15, unit omitted or 'kg/m^3'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 640 OUT408 |
| ✓ | SimpleElectromagneticForceBFCL/simple_python_41 (electromagnetic_force): charge1=5, charge2=7, distance=3, medium_permittivity omitted/8.854e-12.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 642 OUT697 |
| ✓ | SimpleFinalSpeedFreeFallTimeBFCL/simple_python_29 (calculate_final_speed): initial_speed omitted/0, time=5, gravity omitted/-9.81.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 627 OUT1 029 |
| ✓ | SimpleMathFactorial5BFCL/simple_python_1 (math.factorial): number=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 600 OUT372 |
| ✓ | SimpleMathGcd12And15BFCL/simple_python_22 (math.gcd): num1=12, num2=15.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 633 OUT604 |
| ✓ | SimpleShortestDrivingDistanceBFCL/simple_python_36 (get_shortest_driving_distance): origin=New York City, destination=Washington D.C., unit omitted/km.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 625 OUT391 |
| ✓ | SimpleSolveQuadraticRoots253BFCL/simple_python_6 (solve_quadratic): a=2, b=5, c=3.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 638 OUT711 |
| ✓ | SimpleThermoEnergyVaporizationBFCL/simple_python_45 (thermo.calculate_energy): mass=100, phase_transition=vaporization, substance omitted/water.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 670 OUT1 172 |
| ✓ | SimpleTravelItineraryTokyoBFCL/simple_python_34 (travel_itinerary_generator): destination=Tokyo, days=7, daily_budget=100, exploration_type=nature.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 663 OUT1 277 |
Tool Calling - Simple: Math, Physics & Geometry (set 5)
8 passed |
100.0% |
IN51 550 OUT4 642 | |
| ✓ | SimpleAbsolutePressureBFCL/simple_python_49 (calc_absolute_pressure): atm_pressure=1, gauge_pressure=2.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 437 OUT624 |
| ✓ | SimpleFinalTemperatureMixBFCL/simple_python_46 (calculate_final_temperature): mass1=20, temperature1=30, mass2=15, temperature2=60, shc omitted/4.2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 475 OUT717 |
| ✓ | SimpleFinalVelocityAccelTimeBFCL/simple_python_27 (final_velocity): initial_velocity=10, acceleration=2, time=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 469 OUT423 |
| ✓ | SimpleGetDirectionsFastestBFCL/simple_python_33 (get_directions): start=Sydney, end=Melbourne, route_type omitted or 'fastest'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 422 OUT329 |
| ✓ | SimpleMathGcd12And18BFCL/simple_python_24 (math.gcd): num1=12, num2=18.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 423 OUT484 |
| ✓ | SimpleMathHypotenuseBFCL/simple_python_2 (math.hypot): x=4, y=5, z omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 433 OUT362 |
| ✓ | SimpleResonantFrequencyLCBFCL/simple_python_42 (calculate_resonant_frequency): inductance=0.05 (50mH), capacitance=0.0001 (100uF), round_off omitted/2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 460 OUT1 145 |
| ✓ | SimpleVeganRestaurantFindBFCL/simple_python_35 (vegan_restaurant.find_nearby): location='New York, NY', operating_hours=23 (11 PM).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 431 OUT558 |
Tool Calling - Simple: Science, Data & Biology (set 1)
11 passed2 failed |
84.6% |
IN111 267 OUT8 090 | |
| ✓ | SimpleArraySortBFCL/simple_python_87 (array_sort): list=[5,3,4,1,2], order=ascending.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 213 OUT206 |
| ✓ | SimpleAverageTemperatureBFCL/simple_python_78 (average_temperature): location=Austin, days=3, temp_unit=Celsius.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 197 OUT437 |
| ✓ | SimpleBacteriaEvolutionRateBFCL/simple_python_74 (calculate_bacteria_evolution_rate): start=5000, dup_freq=1, duration=6, gen_time omitted or 20.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 230 OUT499 |
| ✓ | SimpleBmiKgCmBFCL/simple_python_84 (calculate_bmi): weight=85, height=180, unit omitted or 'metric'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 216 OUT583 |
| ✓ | SimpleCreateHistogramBFCL/simple_python_79 (create_histogram): data=[85,90,88,92,86,89,91], bins=5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 273 OUT450 |
| ✓ | SimpleCreatureFitnessBFCL/simple_python_72 (calculate_fitness): trait_values=[0.8,0.7], trait_contributions=[0.4,0.6].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 276 OUT477 |
| ✓ | SimpleDbFetchStudentsBFCL/simple_python_89 (db_fetch_records): database=StudentDB, table=students, conditions={department:Science, school:Bluebird High School}, fetch_limit omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 227 OUT564 |
| ✓ | SimpleEntropyChangeSubstanceBFCL/simple_python_50 (entropy_change.calculate): substance=ice, mass=1, init=0, final=100, pressure omitted or 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 265 OUT1 623 |
| ✓ | SimpleGenerateDnaSequenceBFCL/simple_python_71 (generate_DNA_sequence): length=100, preferences=[G,C] in either order.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 216 OUT449 |
| ✓ | SimpleGpsCoordinateDistanceBFCL/simple_python_83 (calculate_distance): coord1=[33.4484,-112.074], coord2=[34.0522,-118.2437], unit=miles.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 305 OUT1 101 |
| ✓ | SimpleImdbMoviesByActorBFCL/simple_python_92 (imdb.find_movies_by_actor): actor='Leonardo DiCaprio', year=2010, category omitted or 'all'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 210 OUT406 |
| × | SimplePopulationProjectionBFCL/simple_python_73 (population_projections): country=United States/USA, years=20, growth_rate omitted or 1.2.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN12 452 OUT1 054 |
| × | SimpleRestaurantFindNearbyBFCL/simple_python_77 (restaurant.find_nearby): location='Los Angeles, CA', dietary_preference=[Vegan].1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: (args.location?.ToLower()).AnyOf(["los angeles, ca"]) |
Failed | IN8 187 OUT241 |
Tool Calling - Simple: Science, Data & Biology (set 2)
13 passed |
100.0% |
IN106 655 OUT6 265 | |
| ✓ | SimpleBmiMetersBFCL/simple_python_88 (calculate_BMI): weight_kg=70, height_m=1.75.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 202 OUT716 |
| ✓ | SimpleClockAngleBFCL/simple_python_98 (calculate_clock_angle): hours=6, minutes=30, round_to omitted or 2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 186 OUT640 |
| ✓ | SimpleDatabaseQueryConditionsBFCL/simple_python_96 (database.query): table=user, conditions=[{age,>,25},{job,=,engineer}].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 245 OUT334 |
| ✓ | SimpleElephantPopulationBFCL/simple_python_75 (elephant_population_estimate): current=35000, growth_rate=0.015, years=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 226 OUT740 |
| ✓ | SimpleEntropyChangeIsothermalBFCL/simple_python_51 (calculate_entropy_change): initial_temp=300, final_temp=400, heat_capacity=5, isothermal omitted or true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 234 OUT568 |
| ✓ | SimpleEvolutionPredictionDarwinBFCL/simple_python_76 (prediction.evolution): species=Homo Sapiens, years=50, model=Darwin.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 200 OUT583 |
| ✓ | SimpleFindRestaurantsThaiBFCL/simple_python_80 (find_restaurants): location=Manhattan variants, food_type=Thai, number=5, dietary_requirements=[vegan].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 214 OUT665 |
| ✓ | SimpleGeoDistanceCitiesBFCL/simple_python_85 (geo_distance.calculate): start='Boston, MA', end='Washington, D.C.', units omitted or 'miles'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 196 OUT485 |
| ✓ | SimpleMathFactorialBFCL/simple_python_97 (math.factorial): number=5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 149 OUT189 |
| ✓ | SimplePlotSineWaveBFCL/simple_python_99 (plot_sine_wave): start=0.0, end=6.2832, frequency=5, amplitude omitted or 1, phase_shift omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 200 OUT394 |
| ✓ | SimpleTheaterMovieReleasesBFCL/simple_python_93 (get_theater_movie_releases): location=LA, timeframe=7, format=IMAX.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 191 OUT406 |
| ✓ | SimpleTriangleAreaMetersBFCL/simple_python_95 (calc_area_triangle): base=5, height=3.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN8 175 OUT255 |
| ✓ | SimpleUpdateUserInfoBFCL/simple_python_94 (update_user_info): user_id=43523, update_info={name:John Doe, email:johndoe@email.com}, database omitted or CustomerInfo. NOTE: single-call contract kept intentionally; the model very occasionally emits a spurious second call (args are correct ~3/3 sampled runs), a model-reliability signal rather than a test defect.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN8 237 OUT290 |
Tool Calling - Simple: Science, Data & Biology (set 3)
11 passed1 failed |
91.7% |
IN95 426 OUT7 187 | |
| ✓ | SimpleCalculateAverageBFCL/simple_python_82 (calculate_average): numbers=[12,15,18,20,21,26,30].1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 660 OUT539 |
| ✓ | SimpleCellDensityBFCL/simple_python_57 (calculate_cell_density): optical_density=0.6, dilution=5, calibration_factor omitted or 1e9.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 632 OUT665 |
| ✓ | SimpleCellInfoHumanBFCL/simple_python_55 (biology.get_cell_info): cell_type=human, detailed=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 595 OUT929 |
| × | SimpleCellProteinsMembraneBFCL/simple_python_56 (cellbio.get_proteins): cell_compartment=plasma membrane, include_description omitted/true/false.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 562 OUT1 269 |
| ✓ | SimpleCityDistanceTrainBFCL/simple_python_86 (city_distance.find_shortest): start=New York, end=Los Angeles, transportation=train, allow_transfer=true.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 640 OUT381 |
| ✓ | SimpleEmployeeFetchDataBFCL/simple_python_90 (employee.fetch_data): company='ABC Ltd.', employee_id=345, data_field=[Personal Info, Job History].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 641 OUT246 |
| ✓ | SimpleFetchDnaSequenceBFCL/simple_python_53 (fetch_DNA_sequence): DNA_id=DNA123, format omitted or 'fasta', upstream omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 599 OUT214 |
| ✓ | SimpleGetRestaurantSushiBFCL/simple_python_91 (get_restaurant): cuisine=sushi, location=Boston, condition=open on Sundays.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 614 OUT461 |
| ✓ | SimpleHeatCapacityAirBFCL/simple_python_52 (calc_heat_capacity): temp=298, volume=10, gas omitted or 'air'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 643 OUT633 |
| ✓ | SimpleMolecularWeightBFCL/simple_python_59 (calculate_molecular_weight): compound=C6H12O6, to_unit=grams/mole or g/mol.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 626 OUT564 |
| ✓ | SimpleOrganelleFunctionLookupBFCL/simple_python_58 (cell_biology.function_lookup): molecule=ATP synthase, organelle=mitochondria, specific_function=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 615 OUT730 |
| ✓ | SimpleProteinSequenceBFCL/simple_python_54 (get_protein_sequence): gene=BRCA1, species omitted or 'Homo sapiens'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 599 OUT556 |
Tool Calling - Simple: Science, Data & Biology (set 4)
12 passed |
100.0% |
IN94 136 OUT6 209 | |
| ✓ | SimpleAnalyzeDnaSequenceBFCL/simple_python_62 (analyze_dna_sequence): sequence + reference_sequence given, mutation_type omitted or 'substitution'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 900 OUT397 |
| ✓ | SimpleDiabetesPredictionBFCL/simple_python_61 (diabetes_prediction): weight=150, height=70 (5ft10in), activity_level=lightly active.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 861 OUT478 |
| ✓ | SimpleFastestRouteAvoidTollsBFCL/simple_python_81 (map_routing.fastest_route): start=San Francisco/SF, end=Los Angeles/LA, avoid_tolls=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 830 OUT360 |
| ✓ | SimpleForestGrowthForecastBFCL/simple_python_68 (forest_growth_forecast): location=Yellowstone National Park, years=5, include_human_impact=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 837 OUT415 |
| ✓ | SimpleGeneticSimilarityBFCL/simple_python_63 (genetics.calculate_similarity): species1=Human, species2=Chimp/Chimpanzee, format omitted or 'percentage'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 828 OUT617 |
| ✓ | SimpleGenotypeFrequencyBFCL/simple_python_64 (calculate_genotype_frequency): allele_frequency=0.3, genotype=AA.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 839 OUT735 |
| ✓ | SimpleIdentifyBirdBFCL/simple_python_67 (identify_bird): color=green, habitat=forest, size=small.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 813 OUT572 |
| ✓ | SimpleMutationTypeFindBFCL/simple_python_60 (mutation_type.find): snp_id=rs6034464, species omitted or 'Homo sapiens'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 843 OUT604 |
| ✓ | SimplePopulationDensityBFCL/simple_python_65 (calculate_density): country=Brazil, year='2022', population=213000000, land_area=8500000.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 886 OUT478 |
| ✓ | SimplePrecipitationStatsBFCL/simple_python_66 (ecology_data.precipitation_stats): location=Amazon rainforest, time_frame=six_months.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 828 OUT329 |
| ✓ | SimpleTurtlePopulationBFCL/simple_python_69 (ecology.get_turtle_population): location=Mississippi river, year=2020, species=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 836 OUT519 |
| ✓ | SimpleVehicleEmissionBFCL/simple_python_70 (calculate_vehicle_emission): vehicle_type=gas, miles_driven=1500, emission_factor omitted or 355.48.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 835 OUT705 |
Tool Calling - Simple: Sports & Games (set 1)
11 passed1 failed |
91.7% |
IN95 920 OUT6 698 | |
| ✓ | SimpleBattingAverageComputeBFCL/simple_python_302 (calculate_batting_average): hits=180, at_bats=600, decimal_places=3 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 667 OUT501 |
| ✓ | SimpleBoardGameInfoTicketBFCL/simple_python_328 (boardgame.get_info): Ticket to Ride, {rating, player count} either order, language omitted/English.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 628 OUT469 |
| ✓ | SimpleFindCardInDeckBFCL/simple_python_335 (find_card_in_deck): rank Queen, suit Hearts, deck omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 629 OUT275 |
| ✓ | SimpleGameListByYearBFCL/simple_python_342 (game_list.get_games): year 2019, multiplayer true, ESRB Everyone.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 650 OUT515 |
| ✓ | SimpleGameWinnerByDateBFCL/simple_python_307 (game_result.get_winner): Lakers/Clippers either order, date 2021-01-28, venue omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 648 OUT530 |
| ✓ | SimpleHighestGameScoreBFCL/simple_python_349 (game_score.highest): Overwatch, PC, region Global or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 641 OUT835 |
| ✓ | SimpleMatchScheduleBFCL/simple_python_308 (sports.match_schedule): Man United, 5 matches, English Premier League or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 646 OUT409 |
| ✓ | SimpleNoteDurationBFCL/simple_python_300 (music.calculate_note_duration): 440/880 Hz, tempo omitted or 120.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 648 OUT353 |
| ✓ | SimplePokerGameWinnerMultiBFCL/simple_python_337 (poker_game_winner): 4 players + per-player hands dict, type omitted/Texas Holdem.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 839 OUT955 |
| × | SimpleSoccerPlayerStatsBFCL/simple_python_303 (soccer_stat.get_player_stats): Cristiano Ronaldo, 2019-2020, league omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN11 654 OUT779 |
| ✓ | SimpleSportRankingPlayerBFCL/simple_python_316 (get_sport_ranking): Tennis, Serena Williams, gender omitted/all/female.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 623 OUT607 |
| ✓ | SimpleTeamRankRegularBFCL/simple_python_317 (get_team_rank): LA Lakers, NBA, 2021, type regular.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 647 OUT470 |
Tool Calling - Simple: Sports & Games (set 2)
9 passed2 failed |
81.8% |
IN85 920 OUT9 733 | |
| ✓ | SimpleBoardGameInfoCatanBFCL/simple_python_330 (board_game_info): Catan, {average_review_rating, age_range}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 123 OUT543 |
| ✓ | SimpleCricketBattingScoreBFCL/simple_python_306 (average_batting_score): Virat Kohli, 10 matches, T20 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 125 OUT497 |
| ✓ | SimpleFifaTeamRankingBFCL/simple_python_318 (get_team_ranking): Germany, 2021, gender men or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 132 OUT2 226 |
| ✓ | SimpleMatchResultsTwoTeamsBFCL/simple_python_325 (sports.match_results): Chicago Bulls vs Los Angeles Lakers, season omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 110 OUT504 |
| ✓ | SimplePokerDetermineWinnerTwoBFCL/simple_python_340 (card_games.poker_determine_winner): John vs Mike with 5-card hands (suit symbols).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 258 OUT1 345 |
| × | SimpleShuffleAndDrawBFCL/simple_python_336 (cards.shuffle_and_draw): num_cards 3.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 808 OUT1 003 |
| ✓ | SimpleSoccerPerformanceBFCL/simple_python_305 (sports_stats.get_performance): Messi, La Liga, 2020-2021, {goals scored, assists made}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 160 OUT916 |
| × | SimpleTeamScheduleGamesBFCL/simple_python_327 (sports_team.get_schedule): Man United, 6 games, Premier League, location omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 877 OUT872 |
| ✓ | SimpleThirdChordBFCL/simple_python_301 (get_third_chord): key C, type major or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 102 OUT841 |
| ✓ | SimpleTopPlayerBySportBFCL/simple_python_323 (sports_ranking.get_top_player): tennis, gender women.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 105 OUT659 |
| ✓ | SimpleVideoGamePlayerStatsBFCL/simple_python_343 (game_stats.fetch_player_statistics): Zelda, user Sam, platform Switch.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 120 OUT327 |
Tool Calling - Simple: Sports & Games (set 3)
7 passed2 failed |
77.8% |
IN64 853 OUT6 539 | |
| ✓ | SimpleCardDrawProbabilityBFCL/simple_python_338 (card_game_probability.calculate): total 52, desired 13 (hearts), cards_drawn 1 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 454 OUT650 |
| ✓ | SimpleCelebrityNetWorthBFCL/simple_python_313 (celebrity_net_worth.get): Messi, currency EUR/euro.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 448 OUT586 |
| ✓ | SimpleDefenseRankingBFCL/simple_python_315 (get_defense_ranking): season 2021, top 1 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 447 OUT442 |
| × | SimpleFindAthleteBFCL/simple_python_311 (sports_db.find_athlete): Lebron James, sport Basketball, team omitted.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 831 OUT1 208 |
| ✓ | SimpleGameItemStatsBFCL/simple_python_344 (get_game_item_stats): BotW, Guardian Sword+, stat power/power rating.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 483 OUT704 |
| ✓ | SimpleMonopolyOddsBFCL/simple_python_329 (monopoly_odds_calculator): number 7, dice_number 2, dice_faces 6 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 452 OUT508 |
| × | SimpleNflPlayerRecordBFCL/simple_python_309 (nfl_data.player_record): Tom Brady, 2020, team omitted or Tampa Bay Buccaneers.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN9 854 OUT1 234 |
| ✓ | SimpleSportsRankingSeasonIntBFCL/simple_python_319 (sports_ranking): Man United, Premier League, season omitted or 2023.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 438 OUT585 |
| ✓ | SimpleTeamLatestScoreBFCL/simple_python_324 (team_score.get_latest): Los Angeles Lakers, include_opponent true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 446 OUT622 |
Tool Calling - Simple: Sports & Games (set 4)
7 passed |
100.0% |
IN41 841 OUT4 892 | |
| ✓ | SimpleCareerStatsBFCL/simple_python_310 (get_career_stats): LeBron James, team omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 954 OUT1 128 |
| ✓ | SimpleCelebrityAchievementsBFCL/simple_python_314 (sports_celebrity.get_major_achievements): Messi, sports Football/Soccer or omitted, team all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 972 OUT763 |
| ✓ | SimpleChessTopPlayersBFCL/simple_python_331 (board_game.chess.get_top_players): New York, min rating 2300, number_of_players 10 or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 979 OUT396 |
| ✓ | SimpleDeckOddsSuitBFCL/simple_python_341 (deck_of_cards.odds): suit hearts, deck_type without_joker or normal.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 971 OUT477 |
| ✓ | SimpleGameValuationBFCL/simple_python_345 (game_valuation): Super Mario Bros., 1985, condition Like New/New.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 991 OUT571 |
| ✓ | SimpleTeamPositionDetailedBFCL/simple_python_320 (sports_ranking.get_team_position): Golden State Warriors, 2022-2023, detailed true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 007 OUT916 |
| ✓ | SimpleTeamScoreWithStatsBFCL/simple_python_326 (get_team_score): Los Angeles Lakers, NBA, include_player_stats any.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 967 OUT641 |
Tool Calling - Simple: Sports & Games (set 5)
6 passed1 failed |
85.7% |
IN43 573 OUT3 975 | |
| ✓ | SimpleChessRatingBFCL/simple_python_332 (chess.rating): Magnus Carlsen, variant classical or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 772 OUT370 |
| ✓ | SimpleCollectablesInSeasonBFCL/simple_python_346 (get_collectables_in_season): Animal Crossing: New Horizons, Spring, item_type all or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 795 OUT762 |
| ✓ | SimpleDetailedWeatherForecastBFCL/simple_python_333 (detailed_weather_forecast): London UK, 3 days, {high/low temp, humidity, precipitation}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 817 OUT462 |
| ✓ | SimpleFullHouseProbabilityBFCL/simple_python_339 (poker_probability.full_house): defaults deck_size 52, hand_size 5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 773 OUT566 |
| ✓ | SimplePlayerStatisticYearBFCL/simple_python_312 (player_statistic): Ronaldo, year 2021, team omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 779 OUT440 |
| × | SimpleSoccerLastMatchBFCL/simple_python_347 (soccer.get_last_match): Liverpool F.C., include_stats true.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN8 846 OUT915 |
| ✓ | SimpleSportsRankingSeasonStrBFCL/simple_python_321 (sports_ranking): Barcelona, La Liga, 2021.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 791 OUT460 |
Tool Calling - Simple: Sports & Games (set 6)
4 passed |
100.0% |
IN20 202 OUT2 540 | |
| ✓ | SimpleBlackjackWinnerBFCL/simple_python_334 (blackjack.check_winner): player [A,10], dealer [10,9], ace_value 1.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 080 OUT915 |
| ✓ | SimpleCreatePlayerProfileBFCL/simple_python_348 (create_player_profile): StarPlayer, class Mage, starting_level 5.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 057 OUT221 |
| ✓ | SimpleCurrentTeamRankingBFCL/simple_python_322 (sports_ranking.get_current): Liverpool, Premier League, season omitted or 2023-2024.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN5 023 OUT364 |
| ✓ | SimplePlayerLastGameStatsBFCL/simple_python_304 (player_stats.getLastGame): LeBron James, Lakers, metrics {Points, Rebounds}.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN5 042 OUT1 040 |
Tool Calling - Simple: Stats, Finance & ML (set 1)
8 passed |
100.0% |
IN51 703 OUT5 081 | |
| ✓ | SimpleCardProbabilityBFCL/simple_python_112 (calculate_probability): total_outcomes=52, favorable_outcomes=4 (king from deck), round_to omitted or 2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 411 OUT372 |
| ✓ | SimpleCompoundInterestNBFCL/simple_python_131 (calculate_compound_interest): principal=10000, rate=0.05, time=10, n=4.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 509 OUT855 |
| ✓ | SimpleInvestmentProfitBFCL/simple_python_134 (investment.predictProfit): investment_amount=5000, annual_return=0.07, years=5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 454 OUT867 |
| ✓ | SimpleLightTravelTimeBFCL/simple_python_100 (light_travel_time): distance_in_light_years=4, speed_of_light omitted or default.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 417 OUT273 |
| ✓ | SimpleQuarterlyDividendBFCL/simple_python_128 (finance.calculate_quarterly_dividend_per_share): total_payout=50000000, outstanding_shares=100000000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 465 OUT815 |
| ✓ | SimpleRandomForestTrainBFCL/simple_python_109 (random_forest.train): n_estimators=100, max_depth=5, data='my_data'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 443 OUT354 |
| ✓ | SimpleStockPriceByDateBFCL/simple_python_142 (get_stock_price): company_name Amazon/AMZN, date=2022-03-11, exchange NASDAQ or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 445 OUT856 |
| ✓ | SimpleTTestStatsBFCL/simple_python_118 (stats.t_test): array_1=[10,15,12,14,11], array_2=[18,16,17,20,22], alpha=0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 559 OUT689 |
Tool Calling - Simple: Stats, Finance & ML (set 2)
7 passed1 failed |
87.5% |
IN57 488 OUT5 857 | |
| ✓ | SimpleCalculateSpeedBFCL/simple_python_101 (calculate_speed): distance=450, time=20, to_unit=km/h.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 717 OUT551 |
| ✓ | SimpleCardProbabilitySingleBFCL/simple_python_116 (probabilities.calculate_single): total_outcomes=52, event_outcomes=4 (king), round omitted or 2.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 705 OUT643 |
| × | SimpleCompoundInterestFreqBFCL/simple_python_136 (compound_interest): principal=10000, annual_rate=5.0, compounding_freq=monthly, time_in_years=5.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 310 OUT1 851 |
| ✓ | SimpleDiscountedCashFlowBFCL/simple_python_129 (calculate_discounted_cash_flow): coupon_payment=100, period=5, discount_rate=0.04, face_value omitted or 1000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 737 OUT749 |
| ✓ | SimpleHousePriceBedroomsBFCL/simple_python_110 (predict_house_price): bedrooms=3, bathrooms=2, area=1800, location San Francisco.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 738 OUT519 |
| ✓ | SimpleStockPriceDaysBFCL/simple_python_143 (get_stock_price): company=AAPL, days=5, exchange NASDAQ or omitted.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 698 OUT448 |
| ✓ | SimpleStockReturnProjectedBFCL/simple_python_137 (calculate_stock_return): investment_amount=5000, annual_growth_rate=0.06, holding_period=5, dividends omitted or false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 750 OUT612 |
| ✓ | SimpleTTestIndBFCL/simple_python_119 (hypothesis_testing.ttest_ind): sample1=[22,33,42,12,34], sample2=[23,45,44,14,38], significance_level=0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 833 OUT484 |
Tool Calling - Simple: Stats, Finance & ML (set 3)
8 passed |
100.0% |
IN52 210 OUT5 306 | |
| ✓ | SimpleCompoundedInterestAnnuallyBFCL/simple_python_145 (calculate_compounded_interest): principal=5000, interest_rate=0.05, period=10, compounding_frequency omitted or Annually.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 523 OUT549 |
| ✓ | SimpleDistanceEarthMoonBFCL/simple_python_102 (calculate_distance): body1=Earth, body2=Moon, unit mi/miles/mile.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 479 OUT633 |
| ✓ | SimpleHeartRatioProbabilityBFCL/simple_python_117 (probability_of_event): success_outcomes=13 (hearts), total_outcomes=52, format_as_ratio=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 511 OUT861 |
| ✓ | SimpleHousePriceRoomsBFCL/simple_python_125 (predict_house_price): area=2500, rooms=5, year=1990, location San Francisco/SF.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 538 OUT765 |
| ✓ | SimpleMutualFundReturnBFCL/simple_python_139 (estimate_mutual_fund_return): yearly_yield=5.0, investment_amount=2000, years=3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 539 OUT898 |
| ✓ | SimpleReturnOnEquityBFCL/simple_python_132 (calculate_return_on_equity): net_income=2000000, shareholder_equity=10000000, dividends_paid=200000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 581 OUT848 |
| ✓ | SimpleStockPriceCurrentMultiBFCL/simple_python_149 (get_stock_price): company_names=[Apple, Microsoft] (or tickers AAPL, MSFT).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 461 OUT261 |
| ✓ | SimpleTwoSampleTTestRunBFCL/simple_python_120 (run_two_sample_ttest): group1=[3,4,5,6,4], group2=[7,8,9,8,7], equal_variance=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 578 OUT491 |
Tool Calling - Simple: Stats, Finance & ML (set 4)
7 passed1 failed |
87.5% |
IN56 345 OUT5 340 | |
| ✓ | SimpleBinomialProbDistBFCL/simple_python_114 (prob_dist.binomial): trials=10, successes=5, p omitted or 0.5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 574 OUT906 |
| ✓ | SimpleFutureValueMonthlyBFCL/simple_python_133 (finance.predict_future_value): present_value=5000, annual_interest_rate=0.05, compounding_periods_per_year=12, time_years=3.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 616 OUT888 |
| ✓ | SimpleGeometryAreaTriangleBFCL/simple_python_104 (geometry.area_triangle): base=6, height=10, unit omitted or 'square meters'.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 557 OUT245 |
| ✓ | SimpleMathPowerBFCL/simple_python_105 (math.power): base=3, exponent=4, mod omitted or 1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 550 OUT521 |
| ✓ | SimpleNPVCashFlowsBFCL/simple_python_127 (calculate_NPV): cash_flows=[200,300,400,500], discount_rate=0.1, initial_investment=2000.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 665 OUT1 094 |
| × | SimpleReturnOnInvestmentBFCL/simple_python_135 (calculate_return_on_investment): purchase_price=20, sale_price=25, dividend=2.1 AI query
2 tool callscategory:ToolCallinglevel:Easysource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN10 073 OUT652 |
| ✓ | SimpleStockPriceDataTypeBFCL/simple_python_146 (stock_price): company Amazon/AMZN, days=3, data_type omitted or Close.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 551 OUT445 |
| ✓ | SimpleTwoSampleTTestFloatBFCL/simple_python_123 (hypothesis_testing.two_sample_t_test): group1=[12.4,15.6,11.2,18.9], group2=[10.5,9.8,15.2,13.8], alpha omitted or 0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 759 OUT589 |
Tool Calling - Simple: Stats, Finance & ML (set 5)
7 passed1 failed |
87.5% |
IN60 578 OUT6 627 | |
| × | SimpleAreaUnderCurvePolyBFCL/simple_python_103 (mathematics.calculate_area_under_curve): polynomial=[3.0,2.0,-4.0], limits=[-1.0,2.0].1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 3 |
Failed | IN13 858 OUT2 374 |
| ✓ | SimpleBinomialCalcBFCL/simple_python_115 (calculate_binomial_probability): number_of_trials=8, number_of_successes=5, probability_of_success omitted or 0.5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 632 OUT1 058 |
| ✓ | SimpleCalculateBmiBFCL/simple_python_107 (calculate_bmi): weight=70, height=175, system omitted or metric.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN6 652 OUT480 |
| ✓ | SimpleFutureValueInvestmentBFCL/simple_python_148 (finance.calculate_future_value): initial_investment=20000, rate_of_return=0.08, years=5, contribution omitted or 0.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 683 OUT487 |
| ✓ | SimpleLinearRegressionBFCL/simple_python_108 (run_linear_regression): predictors=[Age,Income,Education], target=Purchase_Amount, standardize=true.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 669 OUT339 |
| ✓ | SimpleNPVInvestmentBFCL/simple_python_130 (finance_calculator.npv): cash_flows=[-50000,10000,15000,20000,25000,30000], discount_rate=0.08, years omitted or empty.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 767 OUT1 203 |
| ✓ | SimpleStockPricesDurationBFCL/simple_python_147 (get_stock_prices): companies=[Microsoft,Google], duration='2 weeks'.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 621 OUT247 |
| ✓ | SimpleTTestDatasetsBFCL/simple_python_124 (t_test): dataset_A=[12,24,36], dataset_B=[15,30,45], alpha omitted or 0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN6 696 OUT439 |
Tool Calling - Simple: Stats, Finance & ML (set 6)
10 passed |
100.0% |
IN71 471 OUT5 451 | |
| ✓ | SimpleBinomialCoin100BFCL/simple_python_121 (calc_binomial_prob): num_trials=100, num_success=60, prob_success=0.5.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 153 OUT593 |
| ✓ | SimpleCagrBFCL/simple_python_140 (calculate_cagr): initial_value=2000, final_value=3000, period_in_years=4.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 172 OUT592 |
| ✓ | SimpleChiSquaredTestBFCL/simple_python_122 (chi_squared_test): table=[[10,20],[30,40]], alpha omitted or 0.05.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 154 OUT419 |
| ✓ | SimpleDiceRollBFCL/simple_python_113 (probability.dice_roll): desired_number=6, number_of_rolls=2, die_sides omitted or 6.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 130 OUT569 |
| ✓ | SimpleMarketPerformanceBFCL/simple_python_144 (market_performance.get_data): indexes=[S&P 500, Dow Jones], days=5, detailed omitted or true/false.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 137 OUT496 |
| ✓ | SimpleMetalPriceBFCL/simple_python_141 (get_metal_price): metal=Gold, measure=ounce.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 096 OUT529 |
| ✓ | SimpleNormalVariateBFCL/simple_python_111 (random.normalvariate): mu=0, sigma=1.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 117 OUT392 |
| ✓ | SimplePortfolioFutureValueBFCL/simple_python_138 (portfolio_future_value): stock=X, invested_amount=5000, expected_annual_return=0.05, years=7.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN7 175 OUT927 |
| ✓ | SimpleRandomForestClassifierBFCL/simple_python_106 (train_random_forest_classifier): dataset=your_dataset_name, max_depth=5, n_estimators=100.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN7 156 OUT228 |
| ✓ | SimpleRSquaredBFCL/simple_python_126 (linear_regression.get_r_squared): dataset_path=C:/data/cars.csv, independent_variables=[engine_size,fuel_economy], dependent_variable=car_price.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN7 181 OUT706 |
Tool Calling - Tool Selection: Area (Triangle/Circle)
1 passed |
100.0% |
IN4 535 OUT625 | |
| ✓ | SelectTriangleAreaHeronToolBFCL/multiple_1 (math.triangle_area_heron): triangle area from three side lengths (Heron).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 535 OUT625 |
Tool Calling - Tool Selection: Art & Music
8 passed1 failed |
88.9% |
IN105 609 OUT4 528 | |
| ✓ | SelectAnalyzeStructureToolBFCL/multiple_162 (analyze_structure): dynamic structural analysis of specific floors.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 764 OUT379 |
| ✓ | SelectConcertSearchToolBFCL/multiple_166 (concert.search): concert search by genre/location/date/price.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 730 OUT438 |
| × | SelectGuitarPriceToolBFCL/multiple_165 (guitar_price.find): guitar price by model/condition/location (not instrument_price.get).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.model.ToLower() == "gibson les paul" |
Failed | IN11 730 OUT517 |
| ✓ | SelectInstrumentPriceToolBFCL/multiple_164 (instrument_price.get): instrument price by brand/model/finish (not guitar_price.find).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 738 OUT604 |
| ✓ | SelectMelodyGeneratorToolBFCL/multiple_167 (music_generator.generate_melody): melody generation with key/note/length/tempo. Tightened per methodology: prompt explicitly gives "120 beats per minute", so tempo is required exactly 120 (ground truth also accepts omission, which equals the schema default).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 767 OUT310 |
| ✓ | SelectMetropolitanArtworksToolBFCL/multiple_163 (metropolitan_museum.get_top_artworks): top artworks sorted by popularity.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 732 OUT557 |
| ✓ | SelectMusicalScaleToolBFCL/multiple_169 (musical_scale): musical scale for a key.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 706 OUT475 |
| ✓ | SelectSculptureInfoToolBFCL/multiple_160 (get_sculpture_info): sculpture info by artist with detail.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 717 OUT610 |
| ✓ | SelectSongLyricsToolBFCL/multiple_168 (get_song_lyrics): lyrics by song title/artist.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 725 OUT638 |
Tool Calling - Tool Selection: Calculators & Directories
7 passed1 failed |
87.5% |
IN81 752 OUT5 958 | |
| ✓ | SelectCalculateGcdToolBFCL/multiple_35 (calculate_gcd): greatest common divisor, not lcm.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 591 OUT510 |
| ✓ | SelectCurrencyGetRateToolBFCL/multiple_17 (currency_conversion.get_rate): rate at a specific date, not convert/latest-rate.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 610 OUT666 |
| ✓ | SelectGeodistanceToolBFCL/multiple_65 (geodistance.find): distance between cities, not timezone difference/flights.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 580 OUT646 |
| ✓ | SelectLibrarySearchBookToolBFCL/multiple_38 (library.search_book): search for a book, not reserve one.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 592 OUT671 |
| ✓ | SelectLinearRegressionToolBFCL/multiple_53 (linear_regression): linear regression, not random forest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 605 OUT819 |
| ✓ | SelectMagneticFieldCalculateToolBFCL/multiple_41 (magnetic_field.calculate): magnetic field via Ampere's law, not electric field/force.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 623 OUT693 |
| × | SelectSolveQuadraticEquationToolBFCL/multiple_28 (solve.quadratic_equation): solve a quadratic, not rgb/string-reverse.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN14 572 OUT1 521 |
| ✓ | SelectStockForecastToolBFCL/multiple_55 (stock_forecast): stock price prediction, not weather forecast.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 579 OUT432 |
Tool Calling - Tool Selection: Finance & Legal
5 passed5 failed |
50.0% |
IN165 426 OUT9 307 | |
| ✓ | SelectCagrCalculationToolBFCL/multiple_130 (calculate_cagr): compound annual growth rate from initial/final/years, not other finance tools.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 172 OUT730 |
| ✓ | SelectCaseInfoToolBFCL/multiple_135 (get_case_info): retrieve case info (victim) by docket/court.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 148 OUT837 |
| × | SelectCrimeRateToolBFCL/multiple_136 (get_crime_rate): crime rate by city/state/type/year.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.type.ToLower() == "violent" |
Failed | IN13 140 OUT890 |
| ✓ | SelectCrimeRecordToolBFCL/multiple_134 (crime_record.get_record): detailed crime record by case number/county.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 141 OUT662 |
| × | SelectFutureValueToolBFCL/multiple_132 (finance.calculate_future_value): future value from rate/initial/years, not compound_interest.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN19 954 OUT2 137 |
| × | SelectLawsuitDetailsToolBFCL/multiple_139 (lawsuit_details.find): patent lawsuit details by company/year/type.1 AI query
3 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 3 |
Failed | IN26 772 OUT835 |
| × | SelectLawsuitSearchToolBFCL/multiple_137 (lawsuit_search): lawsuit search by company/date/location/status. Tightened per methodology: prompt explicitly asks for "ongoing" lawsuits, so status is required exactly "ongoing" (ground truth also accepts omission, but an explicitly-stated value must be passed).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN19 913 OUT832 |
| ✓ | SelectLegalCaseFetchToolBFCL/multiple_138 (legal_case.fetch): fetch detailed legal case by id.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 136 OUT708 |
| × | SelectMarketPerformanceToolBFCL/multiple_131 (market_performance.get_data): index performance over N days.1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN19 890 OUT669 |
| ✓ | SelectMutualFundBalanceToolBFCL/multiple_133 (calculate_mutual_fund_balance): mutual fund balance from investment/yield/years.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 160 OUT1 007 |
Tool Calling - Tool Selection: Finance & Markets
8 passed |
100.0% |
IN82 450 OUT5 708 | |
| ✓ | SelectAvgClosingPriceToolBFCL/multiple_56 (avg_closing_price): average closing price, not volume/total revenue.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 275 OUT357 |
| ✓ | SelectCompoundInterestCalcToolBFCL/multiple_49 (calc_Compound_Interest): compound interest, not future value/simple interest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 324 OUT652 |
| ✓ | SelectCurrencyConversionToolBFCL/multiple_52 (currency_conversion): currency conversion, not unit conversion.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 285 OUT624 |
| ✓ | SelectDividendDataToolBFCL/multiple_54 (corporate_finance.dividend_data): dividend history, not generic stock market data.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 276 OUT589 |
| ✓ | SelectHousePriceForecastToolBFCL/multiple_50 (house_price_forecast): house price forecast, not weather/stock forecast.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 269 OUT520 |
| ✓ | SelectPropertyDepreciationToolBFCL/multiple_14 (finance.property_depreciation): depreciation, not loan/inflation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 324 OUT935 |
| ✓ | SelectRevenueForecastToolBFCL/multiple_13 (corporate_finance.revenue_forecast): revenue forecast, not product price.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 316 OUT356 |
| ✓ | SelectYoyGrowthRateToolBFCL/multiple_12 (corporate_finance.calculate_YOY_growth_rate): YOY growth, not ROE/ROA.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 381 OUT1 675 |
Tool Calling - Tool Selection: Food & Travel
4 passed3 failed |
57.1% |
IN74 140 OUT5 802 | |
| × | SelectCurrencyConvertToolBFCL/multiple_95 (currency_exchange.convert): currency conversion, not unit_conversion. live_conversion is required true: prompt says "using current exchange rate" (ground truth = [true]).1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN14 023 OUT1 705 |
| × | SelectDrivingDistanceToolBFCL/multiple_104 (get_shortest_driving_distance): driving distance, not instrument price / biology / scientist lookup.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN13 999 OUT1 237 |
| ✓ | SelectHotelBookToolBFCL/multiple_93 (hotel.book): book a hotel room, not car.rental.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 241 OUT319 |
| × | SelectRecipeFindToolBFCL/multiple_91 (recipe.find): recipe by main ingredient + limit, not restaurant.find.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.ingredientLimit == 4
Values:
args.ingredientLimit = 5 |
Failed | IN9 210 OUT622 |
| ✓ | SelectRecipeSearchToolBFCL/multiple_89 (recipe_search): find a recipe from ingredients, not restaurant_search/ingredient_replace.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 227 OUT552 |
| ✓ | SelectRestaurantForGroupToolBFCL/multiple_90 (restaurant.find_group): restaurant for a group, not events.find_event.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 220 OUT306 |
| ✓ | SelectWalmartVeganToolBFCL/multiple_92 (walmart.vegan_products): the named store is Walmart, not Safeway/Whole Foods.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 220 OUT1 061 |
Tool Calling - Tool Selection: Games, Food & Utilities
10 passed |
100.0% |
IN137 979 OUT5 049 | |
| ✓ | SelectBestGroceryStoreToolBFCL/multiple_188 (grocery_store.find_best): best grocery store by rating/products.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 824 OUT458 |
| ✓ | SelectClosestRestaurantToolBFCL/multiple_185 (restaurant_search.find_closest): closest restaurant by cuisine/amenities.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 791 OUT598 |
| ✓ | SelectFindRecipeToolBFCL/multiple_186 (find_recipe): recipe by restrictions/type/time.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 802 OUT422 |
| ✓ | SelectGamePlayerStatsToolBFCL/multiple_180 (game_stats.fetch_player_statistics): player stats by game/username/platform.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 794 OUT266 |
| ✓ | SelectMultiplayerGameFinderToolBFCL/multiple_182 (multiplayer_game_finder): multiplayer game by platform/rating.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 798 OUT514 |
| ✓ | SelectRecipeCaloriesToolBFCL/multiple_183 (recipe_info.get_calories): calories of a recipe from a website.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 789 OUT610 |
| ✓ | SelectRecipeSearchToolBFCL/multiple_184 (recipe_search): recipe by restriction/ingredients/servings.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 799 OUT357 |
| ✓ | SelectSoccerLastMatchToolBFCL/multiple_181 (soccer.get_last_match): last match with stats for a club.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 793 OUT377 |
| ✓ | SelectTimezoneConvertToolBFCL/multiple_189 (timezone.convert): convert a time between timezones.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 798 OUT873 |
| ✓ | SelectWholeFoodsPriceToolBFCL/multiple_187 (whole_foods.check_price): price of items at a Whole Foods location.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 791 OUT574 |
Tool Calling - Tool Selection: Geometry, Finance & Services
7 passed |
100.0% |
IN58 976 OUT3 222 | |
| ✓ | SelectAirQualityForecastToolBFCL/multiple_63 (air_quality_forecast): air quality, not weather/news.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 409 OUT363 |
| ✓ | SelectFinancialCompoundInterestToolBFCL/multiple_57 (financial.compound_interest): compound interest, not simple interest.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 472 OUT797 |
| ✓ | SelectGeometryRectangleToolBFCL/multiple_31 (geometry_rectangle.calculate): rectangle area+perimeter, not square/circle.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 425 OUT286 |
| ✓ | SelectLawyerFinderToolBFCL/multiple_59 (lawyer_finder): find a lawyer by specialization, not rent a car.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 405 OUT266 |
| ✓ | SelectLibrarySearchBooksToolBFCL/multiple_68 (library.search_books): search a specific library, not Google Books/Open Library.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 411 OUT300 |
| ✓ | SelectMapsShortestPathToolBFCL/multiple_27 (maps.shortest_path): shortest path with transport mode, not route-time estimation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 433 OUT899 |
| ✓ | SelectTrafficEstimateToolBFCL/multiple_66 (traffic_estimate): traffic for a time period, not raw distance/weather.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN8 421 OUT311 |
Tool Calling - Tool Selection: History & Knowledge
9 passed1 failed |
90.0% |
IN136 021 OUT8 044 | |
| ✓ | SelectDiscovererToolBFCL/multiple_155 (get_discoverer): discoverer of a discovery, with detail.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 576 OUT591 |
| × | SelectEarliestReferenceToolBFCL/multiple_157 (get_earliest_reference): earliest reference of a person from a source.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.source.ToLower() == "historical records" |
Failed | IN13 578 OUT1 308 |
| ✓ | SelectEventDateToolBFCL/multiple_153 (get_event_date): date of a historical event.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 571 OUT634 |
| ✓ | SelectHistoricalContribToolBFCL/multiple_156 (historical_contrib.get_contrib): scientist's contribution on a date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 608 OUT697 |
| ✓ | SelectHistoryKeyEventsToolBFCL/multiple_152 (history.get_key_events): key war events by country/years.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 621 OUT1 201 |
| ✓ | SelectMostFollowedToolBFCL/multiple_151 (social_media_analytics.most_followed): most-followed by topic/sub-topics.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 610 OUT871 |
| ✓ | SelectNeuronalActivityToolBFCL/multiple_150 (calculate_neuronal_activity): neuronal activity from synaptic rate/weight/decay.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 632 OUT715 |
| ✓ | SelectPaintNeededToolBFCL/multiple_159 (calculate_paint_needed): paint quantity for a wall.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 639 OUT496 |
| ✓ | SelectPapalBiographyToolBFCL/multiple_158 (religious_history.get_papal_biography): papal biography with contributions.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 589 OUT844 |
| ✓ | SelectUsPresidentToolBFCL/multiple_154 (US_president.in_year): president in a given year. Tightened per methodology: prompt explicitly asks for the "full name", so full_name is required true (ground truth also accepts omission, but an explicitly-requested option must be passed).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 597 OUT687 |
Tool Calling - Tool Selection: History, Demographics & Art
7 passed1 failed2 skipped |
87.5% |
IN91 872 OUT5 481 | |
| × | SelectArtworkPriceToolBFCL/multiple_74 (art_auction.fetch_artwork_price): artwork price, not library book search. Tightened per methodology: prompt explicitly asks "on all auction platforms", so require platform == "all" (ground truth also accepts omission, but an explicitly-given value must pass).1 AI query
4 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 4 |
Failed | IN16 651 OUT1 842 |
| ✓ | SelectArtworkSearchToolBFCL/multiple_77 (artwork_search.find): artwork search, not tourist_attraction/park search.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 747 OUT603 |
| – | SelectCustomSculptureToolBFCL/multiple_76 (sculpture.create_custom): custom sculpture (material), not painting (color).0 AI queries
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Skipped: Ambiguous action-vs-question item (item flaw, not model behavior): the prompt mixes an order ('I want to order a custom bronze sculpture of a horse') with a question ('What material options are available?'), so a competent model may either call sculpture_create_custom or answer the question in text - not deterministic. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | SelectEuropeanMonarchsToolBFCL/multiple_70 (european_history.get_monarchs): monarch lookup, not events/culture.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 748 OUT600 |
| ✓ | SelectExhibitionInfoToolBFCL/multiple_79 (exhibition_info): museum exhibitions, not restaurant_info.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 754 OUT462 |
| – | SelectMuseumInfoToolBFCL/multiple_78 (museum_info): museum info with a feature list, not generic tourist_spot_info.0 AI queries
0 tool callscategory:ToolCallinglevel:Hardsource:BFCL Skipped: Ambiguous single-call item (item flaw, not model behavior): a dedicated exhibition_info tool coexists with museum_info, so the 'exhibitions' part can reasonably route either to museum_info's features or to a separate exhibition_info call; a competent model may make one or two calls - not deterministic. Kept for traceability. |
Skipped | IN0 OUT0 |
| ✓ | SelectPaintColorTrendsToolBFCL/multiple_75 (paint_color.trends): paint color trend, not weather/house-price trends.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 732 OUT407 |
| ✓ | SelectPopulationByCategoryToolBFCL/multiple_71 (get_population): population by category, not bureau statistics/demographics.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 756 OUT571 |
| ✓ | SelectReligionOriginToolBFCL/multiple_73 (religion.get_origin): founder/origin, not core beliefs.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 737 OUT476 |
| ✓ | SelectStatePopulationToolBFCL/multiple_72 (us_history.population_by_state_year): state population, not GDP.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 747 OUT520 |
Tool Calling - Tool Selection: Isolated Collisions: Exhibition, Hotel & Currency
1 passed2 failed |
33.3% |
IN21 023 OUT2 145 | |
| × | SelectBookLuxuryHotelToolBFCL/multiple_191 (book_hotel): luxury room with view, stay_duration variant.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: args.room_type.ToLower() == "luxury" |
Failed | IN7 063 OUT312 |
| × | SelectCurrencyConversionToolBFCL/multiple_192 (currency_conversion.convert): convert an amount between currencies.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.from_currency == "EUR"
Values:
args.from_currency = "Euros" |
Failed | IN6 979 OUT670 |
| ✓ | SelectExhibitionToolBFCL/multiple_161 (find_exhibition): top-rated sculpture exhibition by location/art_form.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN6 981 OUT1 163 |
Tool Calling - Tool Selection: Knowledge & Media
7 passed3 failed |
70.0% |
IN135 823 OUT8 447 | |
| ✓ | SelectBattleDetailsToolBFCL/multiple_18 (european_history.battle_details): battle details, not war/leader info.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 747 OUT544 |
| ✓ | SelectDatabaseModifyColumnsToolBFCL/multiple_10 (database.modify_columns): delete columns, not create a backup.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 804 OUT319 |
| ✓ | SelectFiveFactorModelToolBFCL/multiple_69 (five_factor_model.analyse): Big Five personality, not MBTI.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 818 OUT1 261 |
| × | SelectGameRewardsToolBFCL/multiple_26 (game_rewards.get): rewards, not scores/missions list.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 771 OUT1 371 |
| ✓ | SelectGenerateSoundWaveToolBFCL/multiple_21 (generate_sound_wave): generate a sound, not play a file.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 771 OUT623 |
| ✓ | SelectMostPointsSingleGameToolBFCL/multiple_22 (sports_data.basketball.most_points_single_game): single-game record, not season/career.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 747 OUT489 |
| ✓ | SelectPlayerStatsToolBFCL/multiple_23 (basketball.player_stats.get): player stats, not game/team stats.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 769 OUT447 |
| ✓ | SelectReligionSchismsToolBFCL/multiple_19 (religion_history.get_schisms): schisms, not councils/reformations.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 739 OUT819 |
| × | SelectSculpturePriceToolBFCL/multiple_20 (sculpture_price.calculate): commission price, not sculptor info/availability.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 794 OUT1 308 |
| × | SelectVideoGameStorePriceToolBFCL/multiple_25 (video_games.store_price): store price, not store currency/on-sale.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 863 OUT1 266 |
Tool Calling - Tool Selection: Legal, Environment & Services
9 passed1 failed |
90.0% |
IN127 938 OUT6 201 | |
| ✓ | SelectAirQualityToolBFCL/multiple_144 (air_quality): AQI by location/date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 151 OUT510 |
| ✓ | SelectEmissionsToolBFCL/multiple_145 (calculate_emissions): annual CO2 from distance/fuel/efficiency.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 186 OUT1 300 |
| ✓ | SelectHumidityForecastToolBFCL/multiple_142 (weather.humidity_forecast): humidity forecast by location/days.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 138 OUT445 |
| ✓ | SelectLawsuitCheckCaseToolBFCL/multiple_141 (lawsuit.check_case): verify a case's closed status by id.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 157 OUT530 |
| ✓ | SelectLawsuitsSearchToolBFCL/multiple_140 (lawsuits_search): lawsuits by company/location/year.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 152 OUT271 |
| ✓ | SelectMapDirectionsToolBFCL/multiple_147 (map_service.get_directions): directions avoiding features.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN12 153 OUT332 |
| ✓ | SelectRestaurantFindNearbyToolBFCL/multiple_146 (restaurant.find_nearby): nearby restaurants by cuisine/distance.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 148 OUT283 |
| ✓ | SelectSentimentAnalysisToolBFCL/multiple_149 (sentiment_analysis): sentiment of a review text.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 158 OUT614 |
| × | SelectSlopeGradientToolBFCL/multiple_143 (calculate_slope_gradient): slope gradient between two coordinates. Tightened per methodology: prompt explicitly asks for the gradient "in degree", so unit is required exactly "degree" (ground truth also accepts omission, which equals the schema default).1 AI query
2 tool callscategory:ToolCallinglevel:Hardsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN18 568 OUT1 412 |
| ✓ | SelectStockInfoToolBFCL/multiple_148 (get_stock_info): detailed stock information by company.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 127 OUT504 |
Tool Calling - Tool Selection: Local Services & Weather
9 passed |
100.0% |
IN86 764 OUT4 413 | |
| ✓ | SelectHumidityTemperatureForecastToolBFCL/multiple_60 (humidity_temperature_forecast): humidity/temperature, not events/movies.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 632 OUT459 |
| ✓ | SelectLandscapeArchitectToolBFCL/multiple_61 (landscape_architect.find_specialty): landscape architect, not home renovation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 647 OUT344 |
| ✓ | SelectLawyerSearchToolBFCL/multiple_58 (lawyer.search): find a lawyer, not a doctor.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 618 OUT301 |
| ✓ | SelectLibraryFindNearbyToolBFCL/multiple_48 (library.find_nearby): find a library, not a store.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 643 OUT447 |
| ✓ | SelectNatureParkToolBFCL/multiple_62 (nature_park.find_nearby): find a nature park, not a restaurant.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 645 OUT730 |
| ✓ | SelectRideHailingToolBFCL/multiple_39 (ride_hailing.get_rides): find a ride, not a grocery delivery.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 641 OUT306 |
| ✓ | SelectRoutePlannerToolBFCL/multiple_24 (route_planner.calculate_route): plan a route, not chess-club lookup.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 657 OUT651 |
| ✓ | SelectTranslateToolBFCL/multiple_67 (translate): translate text, not sentiment/word-count.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 634 OUT635 |
| ✓ | SelectUvIndexToolBFCL/multiple_64 (uv_index.get_future): UV index, not rainfall/snowfall.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 647 OUT540 |
Tool Calling - Tool Selection: Mathematics
8 passed |
100.0% |
IN75 269 OUT4 079 | |
| ✓ | SelectCalculateIntegralToolBFCL/multiple_33 (calculate_integral): definite integral, not derivative.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 413 OUT344 |
| ✓ | SelectCosineSimilarityToolBFCL/multiple_47 (cosine_similarity.calculate): cosine similarity, not correlation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 432 OUT1 024 |
| ✓ | SelectDiceRollProbabilityToolBFCL/multiple_51 (dice_roll_probability): dice sum probability, not coin/card.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 407 OUT527 |
| ✓ | SelectFunctionsIntersectToolBFCL/multiple_29 (functions.intersect): intersection of two functions, not zeros of one.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 409 OUT421 |
| ✓ | SelectMathLcmToolBFCL/multiple_34 (math.lcm): least common multiple, not gcd/sqrt.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 399 OUT496 |
| ✓ | SelectQuadraticRootsToolBFCL/multiple_11 (math_roots.quadratic): quadratic roots from coefficients, not cubic/polynomial.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 411 OUT675 |
| ✓ | SelectRectangleAreaToolBFCL/multiple_30 (rectangle.area): rectangle area, not circle/triangle area.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 397 OUT380 |
| ✓ | SelectSortListToolBFCL/multiple_46 (sort_list): sort a list, not filter/sum.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 401 OUT212 |
Tool Calling - Tool Selection: Mathematics & Geometry
7 passed |
100.0% |
IN70 313 OUT3 434 | |
| ✓ | SelectCircleAreaToolBFCL/multiple_97 (geometry.area_circle): circle area, not plot_sine_wave.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 026 OUT319 |
| ✓ | SelectCircumferenceToolBFCL/multiple_98 (geometry.circumference): circle circumference, not the reference/time/melody tools.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 017 OUT298 |
| ✓ | SelectDatabaseQueryToolBFCL/multiple_119 (database.query): structured DB query with conditions, not the score/ROI/probability distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN10 108 OUT398 |
| ✓ | SelectDerivativeToolBFCL/multiple_99 (calculus.derivative): derivative at a point, not get_personality_traits.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 040 OUT813 |
| ✓ | SelectHcfToolBFCL/multiple_100 (math.hcf): highest common factor, not the scale-progression distractor.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 036 OUT540 |
| ✓ | SelectSolveQuadraticToolBFCL/multiple_96 (solve_quadratic_equation): solve quadratic, not stock/portfolio tools.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 049 OUT814 |
| ✓ | SelectTriangleAreaToolBFCL/multiple_121 (geometry.area_triangle): triangle area, not map directions / invention lookup.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN10 037 OUT252 |
Tool Calling - Tool Selection: Mixed Domains
7 passed1 skipped |
100.0% |
IN68 830 OUT4 466 | |
| ✓ | SelectAverageGradeToolBFCL/multiple_9 (calculate_average): selects average, not standard_deviation/highest_grade.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 853 OUT593 |
| ✓ | SelectCapacitanceToolBFCL/multiple_6 (capacitance_calculator.calculate): selects capacitance, not resistance/magnetic_field. Tightened per methodology: prompt explicitly states "dielectric constant K is 1.0", so K is required exactly 1.0 (ground truth also accepts omission, but an explicitly-given value must be passed).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 855 OUT915 |
| ✓ | SelectCapitalToolBFCL/multiple_2 (country_info.capital): selects capital lookup, not largest_city/population.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 754 OUT246 |
| ✓ | SelectDisplacementToolBFCL/multiple_4 (kinematics.calculate_displacement): selects displacement, not final_speed.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 847 OUT765 |
| ✓ | SelectEuclideanDistanceToolBFCL/multiple_3 (EuclideanDistance.calculate): selects distance, not angleToXAxis.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 799 OUT574 |
| ✓ | SelectRealEstateSearchToolBFCL/multiple_8 (realestate.find_properties): property search with budget, not property_valuation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 858 OUT700 |
| ✓ | SelectWeatherByCoordinatesDateToolBFCL/multiple_5 (weather.get_by_coordinates_date): historical-by-coordinates, not by-city/forecast.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 864 OUT673 |
| – | SelectWildlifeGrowthToolBFCL/multiple_7 (wildlife_population.assess_growth): population growth, not ecological_impact.0 AI queries
0 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Skipped: Ambiguous single-call item (item flaw, not model behavior): the prompt asks for BOTH population growth AND impact on woodland, and two distinct tools genuinely fit (wildlife_population_assess_growth + ecological_impact_analyze); a competent human would reasonably make two calls, so a single-call ground truth is unfair. Kept for traceability. |
Skipped | IN0 OUT0 |
Tool Calling - Tool Selection: Music, Sports & Games
8 passed1 broken |
88.9% |
IN90 206 OUT6 810 | |
| ✓ | SelectBoardGameRecommendToolBFCL/multiple_86 (BoardGameGeek.recommend): BoardGameGeek store, not AmazonGameStore.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 019 OUT466 |
| ✓ | SelectConcertTicketToolBFCL/multiple_81 (concert.book_ticket): concert ticket, not festival ticket.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 042 OUT271 |
| ! | SelectGameUpdateToolBFCL/multiple_87 (games.update.find): game update/patch, not reviews/price.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN10 030 OUT3 287 |
| ✓ | SelectMusicGenerateToolBFCL/multiple_82 (music.generate): musical piece from key/tempo, not raw audio.generate.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 008 OUT229 |
| ✓ | SelectMusicShopToolBFCL/multiple_80 (music_shop.find_nearby): music shop with lessons, not gym.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 020 OUT386 |
| ✓ | SelectPlayerCountToolBFCL/multiple_88 (video_games.get_player_count): active player count, not sales.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 027 OUT623 |
| ✓ | SelectPlayerGoalsToolBFCL/multiple_83 (player_stats.get_all_time_goals): per-player goals, not team/league top scorer.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 008 OUT525 |
| ✓ | SelectSoccerScoresToolBFCL/multiple_85 (soccer_scores.get_scores): soccer scores, not basketball.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 024 OUT569 |
| ✓ | SelectTopGoalScorersToolBFCL/multiple_84 (getTopGoalScorers): top goal scorers, not top assists.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN10 028 OUT454 |
Tool Calling - Tool Selection: Physical Sciences
10 passed |
100.0% |
IN113 064 OUT9 598 | |
| ✓ | SelectBiologicalEnergyToolBFCL/multiple_43 (biological.calc_energy): energy from moles of substance, not biomass/work.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 291 OUT685 |
| ✓ | SelectBiotSavartLawToolBFCL/multiple_40 (electromagnetism.biot_savart_law): magnetic field via Biot-Savart, not Ampere's law.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 294 OUT1 309 |
| ✓ | SelectConeVolumeToolBFCL/multiple_32 (geometry.calculate_cone_volume): cone volume, not cone mass.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 274 OUT745 |
| ✓ | SelectFinalTemperatureToolBFCL/multiple_42 (calculate_final_temperature): mixing two gas samples, not mass calculation.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 335 OUT767 |
| ✓ | SelectFinalVelocityToolBFCL/multiple_37 (kinematics.final_velocity): final velocity from initial velocity + time, not wave velocity/distance.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 284 OUT785 |
| ✓ | SelectGeologyEraToolBFCL/multiple_45 (geology.get_era): geological era date, not historical event date.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 267 OUT1 044 |
| ✓ | SelectPopulationGeneticsNeToolBFCL/multiple_16 (population_genetics.calculate_ne): effective population size, not range-shift/carrying-capacity.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 334 OUT1 124 |
| ✓ | SelectSolarFarmPotentialToolBFCL/multiple_15 (solarFarm.potential): solar (not wind) energy potential from coordinates + panel area.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 390 OUT871 |
| ✓ | SelectSpeedFromRestToolBFCL/multiple_36 (kinematics.calculate_speed_from_rest): speed from rest over a distance, not acceleration.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 310 OUT1 278 |
| ✓ | SelectWeightInSpaceToolBFCL/multiple_44 (calculate.weight_in_space): weight on a planet, not unit/currency conversion.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 285 OUT990 |
Tool Calling - Tool Selection: Physics & Biology
12 passed1 failed |
92.3% |
IN202 682 OUT8 696 | |
| × | SelectCellProteinsToolBFCL/multiple_109 (cellbio.get_proteins): proteins by cell compartment, not find_exhibition / mutation lookup.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN22 622 OUT1 404 |
| ✓ | SelectDensityToolBFCL/multiple_107 (calculate_density): mass/volume density, not mix_paint_color.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 997 OUT341 |
| ✓ | SelectDisplacementToolBFCL/multiple_102 (calculate_displacement): kinematic displacement, not poker/musical distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 015 OUT424 |
| ✓ | SelectElectricFieldToolBFCL/multiple_106 (calculate_electric_field_strength): electric field, not paint/cooking/group distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 993 OUT926 |
| ✓ | SelectEvolutionPredictionToolBFCL/multiple_114 (prediction.evolution): evolutionary-rate prediction, not modify_painting / calculate_probability.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 005 OUT797 |
| ✓ | SelectFinalSpeedToolBFCL/multiple_103 (calculate_final_speed): free-fall final speed, not chess/court/event distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 985 OUT595 |
| ✓ | SelectFitnessToolBFCL/multiple_113 (calculate_fitness): evolutionary fitness from trait values/contributions, not chess/walmart/lawyer distractors. Percentages map to decimal contributions [0.4, 0.6].1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN15 065 OUT723 |
| ✓ | SelectForestGrowthToolBFCL/multiple_112 (forest_growth_forecast): forest growth forecast, not get_scientist_for_discovery. include_human_impact is required true: prompt says "including human impact" (ground truth = [true]).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 998 OUT639 |
| ✓ | SelectGenotypeFrequencyToolBFCL/multiple_111 (calculate_genotype_frequency): genotype frequency, not factorial / find_flute.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 000 OUT757 |
| ✓ | SelectHeatCapacityToolBFCL/multiple_108 (calc_heat_capacity): gas heat capacity, not discounted_cash_flow.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 016 OUT562 |
| ✓ | SelectLightTravelTimeToolBFCL/multiple_120 (light_travel_time): light travel time, not grocery/hotel/exchange distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN14 983 OUT476 |
| ✓ | SelectMagneticFieldToolBFCL/multiple_105 (calculate_magnetic_field): magnetic field of a loop, not concert/lawsuit distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN14 999 OUT638 |
| ✓ | SelectMutationTypeToolBFCL/multiple_110 (mutation_type.find): SNP mutation type, not get_collectables_in_season.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN15 004 OUT414 |
Tool Calling - Tool Selection: Sports & Card Games
10 passed |
100.0% |
IN134 181 OUT6 057 | |
| ✓ | SelectChessTopPlayersToolBFCL/multiple_177 (board_game.chess.get_top_players): top chess players by location/rating.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 423 OUT475 |
| ✓ | SelectDefenseRankingToolBFCL/multiple_173 (get_defense_ranking): NBA best-defence ranking by season.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 421 OUT586 |
| ✓ | SelectFindAthleteToolBFCL/multiple_172 (sports_db.find_athlete): athlete profile by name/sport.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 406 OUT705 |
| ✓ | SelectFindCardToolBFCL/multiple_178 (find_card_in_deck): find a card by rank/suit in default deck.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 417 OUT206 |
| ✓ | SelectGameWinnerToolBFCL/multiple_171 (game_result.get_winner): winner of a game between two teams on a date.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN13 436 OUT908 |
| ✓ | SelectPokerFullHouseToolBFCL/multiple_179 (poker_probability.full_house): probability of a full house (default deck/hand).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 407 OUT724 |
| ✓ | SelectSoccerPlayerStatsToolBFCL/multiple_170 (soccer_stat.get_player_stats): player stats by name/season.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 434 OUT739 |
| ✓ | SelectSportsRankingToolBFCL/multiple_174 (sports_ranking): team ranking in a league.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 404 OUT626 |
| ✓ | SelectTeamScheduleToolBFCL/multiple_176 (sports_team.get_schedule): team schedule for N games in a league.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 428 OUT725 |
| ✓ | SelectTopPlayerToolBFCL/multiple_175 (sports_ranking.get_top_player): top player by sport/gender.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN13 405 OUT363 |
Tool Calling - Tool Selection: Statistics, Finance & Health
5 passed1 failed |
83.3% |
IN56 326 OUT3 427 | |
| ✓ | SelectBmiMetersToolBFCL/multiple_117 (calculate_BMI): BMI in kg/m, not soccer / hotel_booking distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN9 367 OUT276 |
| ✓ | SelectGcdToolBFCL/multiple_101 (math.gcd): greatest common divisor, not get_top_cases.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN9 344 OUT374 |
| × | SelectHotelPricingToolBFCL/multiple_94 (hotel_room_pricing.get): hotel room pricing, not car/flight pricing.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: args.roomType.ToLower() == "suite with queen size bed" |
Failed | IN9 375 OUT636 |
| ✓ | SelectReturnOnEquityToolBFCL/multiple_128 (calculate_return_on_equity): ROE from income/equity/dividends, not library/lyrics/law distractors. dividends_paid is required 200000: prompt explicitly supplies "dividends paid of $200,000".1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 445 OUT596 |
| ✓ | SelectSingleProbabilityToolBFCL/multiple_124 (probabilities.calculate_single): single-event probability, not math.power / fetch_DNA_sequence.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 353 OUT520 |
| ✓ | SelectTwoSampleTtestToolBFCL/multiple_125 (run_two_sample_ttest): two-group t-test, not restaurant/personality distractors. equal_variance is required true: prompt says "assuming equal variance" (ground truth = [true, ""]).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN9 442 OUT1 025 |
Tool Calling - Tool Selection: Statistics, Finance & Services
6 passed2 failed |
75.0% |
IN103 021 OUT6 353 | |
| × | SelectBmiMetricToolBFCL/multiple_116 (calculate_bmi): BMI in kg/cm with unit option, not celebrity_net_worth.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 257 OUT1 430 |
| ✓ | SelectCalculateProbabilityToolBFCL/multiple_123 (calculate_probability): probability from favorable/total, not lawsuit/house-price distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 367 OUT387 |
| × | SelectCompoundInterestToolBFCL/multiple_129 (compound_interest): compound interest, not lawsuits_search.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN17 326 OUT1 520 |
| ✓ | SelectFindRestaurantsToolBFCL/multiple_115 (find_restaurants): restaurant search with count + dietary, not instrument/stock/sports.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 400 OUT410 |
| ✓ | SelectLinearRegressionToolBFCL/multiple_122 (run_linear_regression): regression with predictors, not recipe/itinerary/cooking distractors. standardize is required true: prompt says "Also apply standardization" (ground truth = [true]).1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN11 417 OUT238 |
| ✓ | SelectMoviesByActorToolBFCL/multiple_118 (imdb.find_movies_by_actor): movies by actor+year, not flight.book / lawsuits_search.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 389 OUT726 |
| ✓ | SelectQuarterlyDividendToolBFCL/multiple_127 (finance.calculate_quarterly_dividend_per_share): dividend per share, not lyrics/paint/movie distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Easysource:BFCL |
Passed | IN11 421 OUT555 |
| ✓ | SelectTTestToolBFCL/multiple_126 (t_test): two-dataset t-test, not event_finder / triangle-area distractors.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN11 444 OUT1 087 |
Tool Calling - Tool Selection: Travel & Biology
7 passed1 failed |
87.5% |
IN104 200 OUT4 942 | |
| ✓ | SelectBookHotelToolBFCL/multiple_190 (book_hotel): book a room by hotel/location/room_type/start/nights.1 AI query
1 tool callcategory:ToolCallinglevel:Hardsource:BFCL |
Passed | IN12 281 OUT314 |
| × | SelectCellProteinsToolBFCL/multiple_196 (cellbio.get_proteins): proteins in a cell compartment.1 AI query
2 tool callscategory:ToolCallinglevel:Mediumsource:BFCL Assertion expression evaluated to false: response.ToolCalls.Count == 1
Values:
response.ToolCalls.Count = 2 |
Failed | IN18 485 OUT1 269 |
| ✓ | SelectDistanceDurationToolBFCL/multiple_193 (maps.get_distance_duration): distance/duration between two places.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 231 OUT448 |
| ✓ | SelectForestGrowthToolBFCL/multiple_199 (forest_growth_forecast): forest growth forecast with human impact.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 234 OUT685 |
| ✓ | SelectGenotypeFrequencyToolBFCL/multiple_198 (calculate_genotype_frequency): genotype frequency from allele frequency.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 242 OUT629 |
| ✓ | SelectHeatCapacityToolBFCL/multiple_195 (calc_heat_capacity): heat capacity for a gas by temp/volume. Tightened per methodology: prompt explicitly states the gas is "air", so gas is required exactly "air" (ground truth also accepts omission, which equals the schema default).1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 258 OUT827 |
| ✓ | SelectMuseumHoursToolBFCL/multiple_194 (get_museum_hours): museum opening hours on a day.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 223 OUT517 |
| ✓ | SelectMutationTypeToolBFCL/multiple_197 (mutation_type.find): gene mutation type by SNP id.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN12 246 OUT253 |
Tool Calling - Tool Selection: Triangle Properties
1 passed |
100.0% |
IN4 613 OUT664 | |
| ✓ | SelectTrianglePropertiesToolBFCL/multiple_0 (triangle_properties.get): retrieves triangle dimensions/properties from 3 sides.1 AI query
1 tool callcategory:ToolCallinglevel:Mediumsource:BFCL |
Passed | IN4 613 OUT664 |