Benchmark summary
IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Instruction Following - Alphabet & Consonants
3 passed5 failed11 broken |
15.8% |
IN18 102 OUT535 107 | |
| × | AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN952 OUT40 601 |
| × | AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN990 OUT7 144 |
| ✓ | AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN926 OUT4 112 |
| ✓ | AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN947 OUT7 346 |
| ✓ | AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN942 OUT4 330 |
| × | AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN997 OUT13 306 |
| ! | ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN979 OUT6 344 |
| ! | ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN933 OUT62 144 |
| ! | ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN946 OUT12 329 |
| × | ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN919 OUT8 280 |
| ! | ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN993 OUT6 728 |
| × | ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN918 OUT8 332 |
| ! | ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN969 OUT62 144 |
| ! | ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN955 OUT14 092 |
| ! | ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN933 OUT62 144 |
| ! | ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN917 OUT62 144 |
| ! | ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 006 OUT62 144 |
| ! | ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN918 OUT29 299 |
| ! | ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN962 OUT62 144 |
Instruction Following - Brackets & Quotes
13 passed6 failed1 skipped |
68.4% |
IN17 971 OUT44 306 | |
| × | Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedParentheses(answer) |
Failed | IN951 OUT1 138 |
| × | Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedParentheses(answer) |
Failed | IN926 OUT1 685 |
| ✓ | Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN909 OUT1 902 |
| × | Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedParentheses(answer) |
Failed | IN1 000 OUT1 411 |
| ✓ | Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN898 OUT3 252 |
| ✓ | Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 006 OUT1 458 |
| ✓ | Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN944 OUT923 |
| × | Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN932 OUT3 231 |
| ✓ | Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN907 OUT2 133 |
| × | Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN915 OUT951 |
| ✓ | Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN922 OUT3 477 |
| ✓ | Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN922 OUT1 460 |
| × | Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN1 004 OUT3 007 |
| ✓ | Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN983 OUT5 878 |
| – | QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN952 OUT2 908 |
| ✓ | QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN942 OUT3 807 |
| ✓ | QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN952 OUT2 627 |
| ✓ | QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN912 OUT1 449 |
| ✓ | QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN994 OUT1 609 |
Instruction Following - Bullet Points
15 passed2 failed |
88.2% |
IN16 059 OUT30 579 | |
| ✓ | NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN923 OUT921 |
| ✓ | NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN931 OUT1 756 |
| × | NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSomeBullets(answer) |
Failed | IN983 OUT1 578 |
| ✓ | NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN905 OUT610 |
| ✓ | NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 009 OUT1 581 |
| ✓ | SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN912 OUT1 314 |
| × | SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer) |
Failed | IN919 OUT824 |
| ✓ | SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN994 OUT2 218 |
| ✓ | SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN946 OUT2 345 |
| ✓ | SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN959 OUT2 754 |
| ✓ | SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN984 OUT648 |
| ✓ | SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN922 OUT1 231 |
| ✓ | SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN950 OUT3 247 |
| ✓ | SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN923 OUT2 386 |
| ✓ | SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN930 OUT2 518 |
| ✓ | SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN946 OUT2 010 |
| ✓ | SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN923 OUT2 638 |
Instruction Following - Custom Transformations
8 passed3 failed |
72.7% |
IN9 954 OUT18 306 | |
| × | CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharacterReverse(answer) |
Failed | IN902 OUT469 |
| ✓ | CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN907 OUT360 |
| ✓ | CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN918 OUT820 |
| ✓ | CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN925 OUT1 954 |
| ✓ | DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN901 OUT4 299 |
| × | EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckEuropeanCapitals(answer) |
Failed | IN914 OUT4 938 |
| ✓ | Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT1 561 |
| ✓ | Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN888 OUT406 |
| × | ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckReverseNewline(answer) |
Failed | IN887 OUT1 050 |
| ✓ | SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN896 OUT1 868 |
| ✓ | WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN902 OUT581 |
Instruction Following - Indentation & Emoji
6 passed3 failed8 skipped |
66.7% |
IN8 313 OUT21 932 | |
| – | Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN899 OUT803 |
| ✓ | LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN891 OUT931 |
| ✓ | LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN900 OUT1 409 |
| ✓ | LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN942 OUT767 |
| ✓ | LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN918 OUT2 000 |
| ✓ | LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN967 OUT1 392 |
| × | LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer) |
Failed | IN940 OUT3 708 |
| × | LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharCountUniqueWords(answer) |
Failed | IN910 OUT2 745 |
| × | LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharCountUniqueWords(answer) |
Failed | IN946 OUT8 177 |
Instruction Following - Keyword & Conjunction Counts
15 passed2 failed1 broken |
83.3% |
IN17 214 OUT44 269 | |
| ✓ | Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN986 OUT3 905 |
| ✓ | Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN982 OUT1 877 |
| ✓ | Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN945 OUT1 838 |
| ✓ | Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 008 OUT1 622 |
| ✓ | Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN983 OUT1 166 |
| × | KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "kaleidoscope", "nebula", "whisper", "labyrinth", "paradox") |
Failed | IN1 002 OUT3 210 |
| ✓ | KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN924 OUT3 023 |
| ✓ | KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 007 OUT4 675 |
| ! | KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 031 OUT3 448 |
| × | KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "quickly", "transform", "abundant", "bridge", "through") |
Failed | IN951 OUT2 813 |
| ✓ | Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN893 OUT993 |
| ✓ | Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN980 OUT533 |
| ✓ | Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN951 OUT2 728 |
| ✓ | Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN946 OUT4 030 |
| ✓ | Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN914 OUT1 934 |
| ✓ | NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN901 OUT1 703 |
| ✓ | NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN898 OUT1 413 |
| ✓ | NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN912 OUT3 358 |
Instruction Following - Keyword In Nth Sentence
6 passed2 failed1 broken |
66.7% |
IN8 358 OUT43 413 | |
| ! | KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 018 OUT4 171 |
| ✓ | KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT2 591 |
| ✓ | KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN899 OUT1 126 |
| ✓ | KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN893 OUT2 747 |
| ✓ | KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN936 OUT4 224 |
| × | KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "trichosis", 22) |
Failed | IN960 OUT17 223 |
| ✓ | KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN912 OUT2 684 |
| ✓ | KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN918 OUT2 507 |
| × | KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 246, 254) |
Failed | IN908 OUT6 140 |
Instruction Following - Lists, Lines & Options
13 passed5 failed1 broken |
68.4% |
IN17 862 OUT37 835 | |
| ✓ | List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN962 OUT2 006 |
| × | List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?") |
Failed | IN902 OUT723 |
| × | List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?") |
Failed | IN916 OUT1 255 |
| ✓ | List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN936 OUT899 |
| ✓ | List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN903 OUT963 |
| × | ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN1 010 OUT7 130 |
| ✓ | ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN942 OUT6 277 |
| ✓ | ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT3 123 |
| ! | NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN916 OUT3 140 |
| ✓ | NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN956 OUT2 136 |
| × | NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNewLineWords(answer) |
Failed | IN927 OUT3 723 |
| × | NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNewLineWords(answer) |
Failed | IN937 OUT2 471 |
| ✓ | NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN986 OUT641 |
| ✓ | Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN908 OUT170 |
| ✓ | Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN932 OUT817 |
| ✓ | Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN1 017 OUT863 |
| ✓ | Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN922 OUT184 |
| ✓ | Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN960 OUT818 |
| ✓ | OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN916 OUT496 |
Instruction Following - Names, Pronouns & Punctuation
11 passed3 failed2 broken |
68.8% |
IN15 676 OUT83 566 | |
| ✓ | PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 049 OUT1 057 |
| ✓ | PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 052 OUT2 047 |
| ✓ | PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN1 048 OUT1 021 |
| ✓ | PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 015 OUT689 |
| ✓ | PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 065 OUT2 290 |
| ! | Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN892 OUT14 924 |
| × | Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPronouns(answer, 39) |
Failed | IN907 OUT7 064 |
| ✓ | Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN971 OUT2 318 |
| ✓ | Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN973 OUT1 384 |
| ✓ | Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN901 OUT2 145 |
| ✓ | Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 002 OUT10 389 |
| × | Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN1 008 OUT4 343 |
| × | Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN961 OUT3 234 |
| ! | Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN949 OUT20 505 |
| ✓ | Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN923 OUT4 379 |
| ✓ | Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN960 OUT5 777 |
Instruction Following - Palindromes & Letter Patterns
8 passed5 failed4 broken5 skipped |
47.1% |
IN16 098 OUT206 574 | |
| ✓ | NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN931 OUT1 404 |
| ✓ | NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN904 OUT1 320 |
| × | NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN903 OUT4 716 |
| × | NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN994 OUT5 193 |
| × | NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN938 OUT3 330 |
| – | OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN927 OUT1 146 |
| ✓ | Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN949 OUT29 118 |
| ✓ | Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN921 OUT1 366 |
| ✓ | Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN918 OUT9 549 |
| ✓ | Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN937 OUT2 799 |
| × | Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer) |
Failed | IN1 041 OUT2 849 |
| ✓ | Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 117 OUT5 995 |
| ! | PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN889 OUT62 144 |
| ! | PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN921 OUT62 144 |
| ! | PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN940 OUT5 822 |
| ! | PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN886 OUT6 003 |
| × | PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer) |
Failed | IN982 OUT1 676 |
Instruction Following - Prompt Repetition
3 passed6 failed |
33.3% |
IN8 515 OUT20 831 | |
| ✓ | RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN930 OUT580 |
| × | RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "Use Python code: construct a 3-layers feedforward network with 8 input neurons, 20 hidden neurons, and 2 output neurons.") |
Failed | IN946 OUT2 470 |
| ✓ | RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 001 OUT1 166 |
| × | RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer) |
Failed | IN1 025 OUT7 814 |
| × | RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer) |
Failed | IN892 OUT409 |
| × | RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6) |
Failed | IN919 OUT2 199 |
| × | RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15) |
Failed | IN930 OUT3 014 |
| × | RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7) |
Failed | IN955 OUT1 434 |
| ✓ | RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN917 OUT1 745 |
Instruction Following - Ratios & Stop Words
12 passed6 failed2 broken |
60.0% |
IN19 171 OUT130 319 | |
| × | RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN922 OUT2 373 |
| ✓ | RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN943 OUT679 |
| × | RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN905 OUT1 579 |
| ✓ | RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN948 OUT1 642 |
| × | RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN1 029 OUT1 047 |
| ! | RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN981 OUT20 349 |
| × | RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN958 OUT13 827 |
| ✓ | RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN999 OUT5 807 |
| × | RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN982 OUT1 193 |
| ✓ | RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN977 OUT1 588 |
| ✓ | RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN909 OUT414 |
| ✓ | RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN950 OUT2 910 |
| ✓ | RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 030 OUT3 598 |
| ✓ | RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN949 OUT3 373 |
| ✓ | RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN896 OUT712 |
| × | RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckStopWords(answer, 20) |
Failed | IN926 OUT1 499 |
| ✓ | RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN935 OUT1 222 |
| ✓ | RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN983 OUT2 567 |
| ✓ | RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 002 OUT1 796 |
| ! | RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN947 OUT62 144 |
Instruction Following - Sentence Progression
3 passed7 failed3 broken |
23.1% |
IN12 229 OUT89 242 | |
| ✓ | AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 002 OUT4 998 |
| ✓ | AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN920 OUT4 781 |
| × | AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN950 OUT15 263 |
| × | AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN938 OUT7 252 |
| × | AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN930 OUT4 614 |
| ! | SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN943 OUT2 673 |
| ! | SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN959 OUT5 162 |
| ✓ | SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN899 OUT10 391 |
| × | SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4) |
Failed | IN934 OUT1 177 |
| × | SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 1) |
Failed | IN903 OUT6 596 |
| ! | SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN938 OUT15 430 |
| × | SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4) |
Failed | IN921 OUT7 029 |
| × | SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5) |
Failed | IN992 OUT3 876 |
Instruction Following - Sentence Words & Overlap
4 failed1 broken9 skipped |
0.0% |
IN4 736 OUT131 033 | |
| – | RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| ! | RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN920 OUT25 004 |
| × | RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN933 OUT6 732 |
| × | RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN934 OUT10 415 |
| × | RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN992 OUT49 256 |
| × | RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN957 OUT39 626 |
Instruction Following - Templates & Whitespace
13 passed5 failed |
72.2% |
IN16 513 OUT31 936 | |
| × | NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN888 OUT484 |
| × | NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN889 OUT1 599 |
| × | NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN886 OUT547 |
| × | NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN925 OUT1 160 |
| ✓ | OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN903 OUT510 |
| ✓ | OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN911 OUT1 310 |
| ✓ | OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN984 OUT2 838 |
| ✓ | OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN921 OUT627 |
| ✓ | Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN929 OUT1 447 |
| ✓ | Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN938 OUT1 407 |
| ✓ | Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN913 OUT1 348 |
| ✓ | Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN925 OUT4 299 |
| ✓ | Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN899 OUT1 285 |
| ✓ | ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT3 969 |
| ✓ | TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN926 OUT2 901 |
| ✓ | TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN912 OUT1 849 |
| × | TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckTitleCase(answer) |
Failed | IN952 OUT3 239 |
| ✓ | TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN898 OUT1 117 |
Instruction Following - Vowels & Word Repeats
4 passed4 failed7 broken |
26.7% |
IN14 038 OUT281 880 | |
| ✓ | LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN970 OUT12 632 |
| × | LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4) |
Failed | IN907 OUT2 458 |
| ✓ | LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN902 OUT2 192 |
| ✓ | LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 018 OUT1 678 |
| ✓ | LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN900 OUT3 981 |
| ! | ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN900 OUT6 861 |
| ! | ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN898 OUT9 248 |
| ! | ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN932 OUT62 144 |
| ! | ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN895 OUT62 144 |
| × | ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN961 OUT11 219 |
| × | ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN1 007 OUT5 607 |
| ! | ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN891 OUT62 144 |
| × | VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN946 OUT18 634 |
| ! | VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN998 OUT14 257 |
| ! | VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN913 OUT6 681 |
Instruction Following - Word Counts
9 passed6 failed4 broken |
47.4% |
IN17 936 OUT69 829 | |
| × | Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5) |
Failed | IN919 OUT4 449 |
| ! | Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN930 OUT5 058 |
| × | Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 17) |
Failed | IN909 OUT5 060 |
| × | Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5) |
Failed | IN899 OUT9 679 |
| × | Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 25) |
Failed | IN923 OUT2 270 |
| ✓ | UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 102 OUT2 436 |
| ✓ | UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN972 OUT1 982 |
| ✓ | UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN888 OUT1 615 |
| ! | UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN897 OUT5 022 |
| ✓ | UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 049 OUT2 055 |
| ! | UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN893 OUT7 183 |
| ! | UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN891 OUT3 810 |
| × | UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckUniqueWordCount(answer, 170) |
Failed | IN991 OUT1 184 |
| ✓ | WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 004 OUT3 014 |
| ✓ | WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN905 OUT1 637 |
| ✓ | WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN969 OUT3 662 |
| ✓ | WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN936 OUT2 927 |
| ✓ | WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN903 OUT2 254 |
| × | WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?") |
Failed | IN956 OUT4 532 |
Instruction Following - Word Position & Order
9 passed3 failed6 broken6 skipped |
50.0% |
IN17 088 OUT179 385 | |
| × | KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "giggle", 22, 33) |
Failed | IN922 OUT3 395 |
| × | KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "beneath", 23, 34) |
Failed | IN927 OUT3 824 |
| ! | KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN990 OUT3 411 |
| ✓ | KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN972 OUT3 532 |
| ✓ | LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN913 OUT1 286 |
| ! | LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN911 OUT62 144 |
| ! | LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN918 OUT2 923 |
| × | LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN1 016 OUT2 902 |
| ✓ | LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN968 OUT6 606 |
| ! | ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN908 OUT3 044 |
| ! | ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN945 OUT62 144 |
| ✓ | ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN981 OUT2 095 |
| ✓ | ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN913 OUT2 933 |
| ! | ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN999 OUT5 676 |
| ✓ | ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN922 OUT2 461 |
| – | StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN992 OUT2 493 |
| ✓ | WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN977 OUT2 120 |
| ✓ | WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN914 OUT6 396 |