Benchmark summary
IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Instruction Following - Alphabet & Consonants
4 passed4 failed11 broken |
21.1% |
IN18 102 OUT564 700 | |
| × | AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN952 OUT10 786 |
| × | AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN990 OUT11 761 |
| ✓ | AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN926 OUT5 432 |
| ✓ | AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN947 OUT14 177 |
| ✓ | AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN942 OUT5 923 |
| × | AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN997 OUT17 615 |
| ! | ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN979 OUT62 144 |
| ! | ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN933 OUT9 763 |
| ! | ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN946 OUT62 144 |
| ✓ | ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN919 OUT29 765 |
| ! | ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN993 OUT20 700 |
| ! | ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN918 OUT62 144 |
| ! | ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN969 OUT62 144 |
| ! | ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN955 OUT62 144 |
| ! | ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN933 OUT18 186 |
| × | ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN917 OUT29 673 |
| ! | ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 006 OUT62 144 |
| ! | ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN918 OUT6 429 |
| ! | ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN962 OUT11 626 |
Instruction Following - Brackets & Quotes
12 passed5 failed2 broken1 skipped |
63.2% |
IN17 971 OUT59 146 | |
| × | Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedParentheses(answer) |
Failed | IN951 OUT1 092 |
| ✓ | Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN926 OUT1 634 |
| ✓ | Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN909 OUT2 208 |
| ✓ | Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 000 OUT1 844 |
| ✓ | Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN898 OUT3 262 |
| ✓ | Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 006 OUT931 |
| ! | Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN944 OUT8 390 |
| × | Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN932 OUT2 944 |
| × | Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN907 OUT3 875 |
| ✓ | Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN915 OUT884 |
| × | Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN922 OUT1 532 |
| × | Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN922 OUT1 072 |
| ✓ | Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 004 OUT2 702 |
| ✓ | Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN983 OUT7 616 |
| – | QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN952 OUT3 144 |
| ✓ | QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN942 OUT5 654 |
| ! | QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN952 OUT6 170 |
| ✓ | QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN912 OUT584 |
| ✓ | QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN994 OUT3 608 |
Instruction Following - Bullet Points
12 passed3 failed2 broken |
70.6% |
IN16 059 OUT52 677 | |
| ✓ | NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN923 OUT2 036 |
| ✓ | NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN931 OUT2 842 |
| × | NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSomeBullets(answer) |
Failed | IN983 OUT1 152 |
| ✓ | NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN905 OUT1 540 |
| ✓ | NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 009 OUT1 076 |
| ✓ | SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN912 OUT2 030 |
| × | SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer) |
Failed | IN919 OUT1 124 |
| × | SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer) |
Failed | IN994 OUT1 788 |
| ✓ | SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN946 OUT1 035 |
| ✓ | SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN959 OUT5 137 |
| ✓ | SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN984 OUT1 282 |
| ✓ | SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN922 OUT1 833 |
| ✓ | SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN950 OUT5 598 |
| ✓ | SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN923 OUT10 436 |
| ! | SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN930 OUT4 471 |
| ! | SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN946 OUT7 578 |
| ✓ | SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN923 OUT1 719 |
Instruction Following - Custom Transformations
6 passed5 failed |
54.5% |
IN9 954 OUT46 119 | |
| × | CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharacterReverse(answer) |
Failed | IN902 OUT699 |
| ✓ | CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN907 OUT393 |
| × | CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCsvQuotes(answer) |
Failed | IN918 OUT966 |
| ✓ | CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN925 OUT1 643 |
| ✓ | DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN901 OUT16 067 |
| × | EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckEuropeanCapitals(answer) |
Failed | IN914 OUT21 586 |
| ✓ | Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT1 465 |
| ✓ | Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN888 OUT185 |
| × | ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckReverseNewline(answer) |
Failed | IN887 OUT1 622 |
| × | SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceAlphabet(answer) |
Failed | IN896 OUT1 060 |
| ✓ | WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN902 OUT433 |
Instruction Following - Indentation & Emoji
5 passed2 failed2 broken8 skipped |
55.6% |
IN8 313 OUT87 733 | |
| – | Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN899 OUT905 |
| × | LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer) |
Failed | IN891 OUT908 |
| ✓ | LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN900 OUT1 337 |
| ✓ | LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN942 OUT1 334 |
| ✓ | LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN918 OUT1 873 |
| ✓ | LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN967 OUT2 091 |
| × | LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer) |
Failed | IN940 OUT2 147 |
| ! | LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN910 OUT14 994 |
| ! | LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN946 OUT62 144 |
Instruction Following - Keyword & Conjunction Counts
15 passed3 failed |
83.3% |
IN17 214 OUT49 856 | |
| ✓ | Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN986 OUT5 224 |
| ✓ | Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN982 OUT4 054 |
| ✓ | Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN945 OUT1 332 |
| ✓ | Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 008 OUT1 594 |
| ✓ | Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN983 OUT2 379 |
| × | KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "kaleidoscope", "nebula", "whisper", "labyrinth", "paradox") |
Failed | IN1 002 OUT3 488 |
| ✓ | KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN924 OUT1 996 |
| × | KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "door", "bread", "blue", "lamp", "river") |
Failed | IN1 007 OUT2 935 |
| ✓ | KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 031 OUT4 141 |
| ✓ | KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN951 OUT2 135 |
| ✓ | Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN893 OUT609 |
| ✓ | Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN980 OUT500 |
| ✓ | Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN951 OUT4 042 |
| ✓ | Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN946 OUT2 344 |
| ✓ | Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN914 OUT1 984 |
| ✓ | NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN901 OUT2 904 |
| × | NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 7) |
Failed | IN898 OUT1 937 |
| ✓ | NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN912 OUT6 258 |
Instruction Following - Keyword In Nth Sentence
7 passed2 broken |
77.8% |
IN8 358 OUT38 257 | |
| ! | KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 018 OUT5 443 |
| ✓ | KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT2 291 |
| ✓ | KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN899 OUT1 092 |
| ✓ | KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN893 OUT2 163 |
| ✓ | KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN936 OUT3 811 |
| ! | KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN960 OUT5 371 |
| ✓ | KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN912 OUT5 775 |
| ✓ | KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN918 OUT3 282 |
| ✓ | KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN908 OUT9 029 |
Instruction Following - Lists, Lines & Options
14 passed3 failed2 broken |
73.7% |
IN17 862 OUT153 912 | |
| × | List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "SEPARATOR") |
Failed | IN962 OUT2 607 |
| × | List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?") |
Failed | IN902 OUT941 |
| ✓ | List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN916 OUT2 007 |
| ✓ | List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN936 OUT944 |
| ✓ | List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN903 OUT1 482 |
| ✓ | ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 010 OUT5 009 |
| ! | ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN942 OUT62 144 |
| ✓ | ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN914 OUT3 500 |
| ✓ | NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN916 OUT4 131 |
| ! | NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN956 OUT57 418 |
| × | NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNewLineWords(answer) |
Failed | IN927 OUT3 347 |
| ✓ | NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN937 OUT4 359 |
| ✓ | NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN986 OUT2 702 |
| ✓ | Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN908 OUT282 |
| ✓ | Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN932 OUT807 |
| ✓ | Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN1 017 OUT629 |
| ✓ | Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN922 OUT150 |
| ✓ | Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN960 OUT588 |
| ✓ | OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN916 OUT865 |
Instruction Following - Names, Pronouns & Punctuation
14 passed2 failed |
87.5% |
IN15 676 OUT54 981 | |
| ✓ | PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 049 OUT1 035 |
| ✓ | PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 052 OUT1 594 |
| ✓ | PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN1 048 OUT1 008 |
| ✓ | PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 015 OUT810 |
| ✓ | PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 065 OUT2 948 |
| ✓ | Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN892 OUT2 853 |
| ✓ | Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN907 OUT3 982 |
| ✓ | Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN971 OUT4 224 |
| ✓ | Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN973 OUT918 |
| ✓ | Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN901 OUT5 555 |
| × | Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN1 002 OUT8 870 |
| ✓ | Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 008 OUT3 297 |
| × | Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN961 OUT2 958 |
| ✓ | Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN949 OUT4 931 |
| ✓ | Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN923 OUT5 157 |
| ✓ | Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN960 OUT4 841 |
Instruction Following - Palindromes & Letter Patterns
7 passed4 failed6 broken5 skipped |
41.2% |
IN16 098 OUT269 011 | |
| × | NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN931 OUT2 441 |
| ✓ | NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN904 OUT2 193 |
| × | NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN903 OUT4 863 |
| × | NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN994 OUT15 145 |
| ! | NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN938 OUT21 940 |
| – | OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN927 OUT939 |
| ✓ | Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN949 OUT13 915 |
| ✓ | Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN921 OUT1 197 |
| ! | Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN918 OUT22 540 |
| ✓ | Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN937 OUT4 551 |
| ✓ | Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 041 OUT3 731 |
| × | Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer) |
Failed | IN1 117 OUT4 456 |
| ! | PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN889 OUT31 160 |
| ! | PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN921 OUT62 144 |
| ! | PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN940 OUT62 144 |
| ! | PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN886 OUT5 043 |
| ✓ | PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN982 OUT10 609 |
Instruction Following - Prompt Repetition
1 passed8 failed |
11.1% |
IN8 515 OUT24 486 | |
| × | RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "Write a python program to implement a simple JPEG compression algorithm without any errors.") |
Failed | IN930 OUT863 |
| × | RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "Use Python code: construct a 3-layers feedforward network with 8 input neurons, 20 hidden neurons, and 2 output neurons.") |
Failed | IN946 OUT3 112 |
| × | RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "The random, uniform generation of derangements is very simple. Since Dn ≈ n!/e, the obvious procedure of generating a random permutation and check if it is a derangement or not, and generating a new permutation in the negative case, is straight-forward and guarantees a linear time complexity, at least on the average; actually, as we will see, the average complexity is µ1 ≈ e(n − 1).") |
Failed | IN1 001 OUT708 |
| × | RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer) |
Failed | IN1 025 OUT4 930 |
| × | RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer) |
Failed | IN892 OUT744 |
| × | RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6) |
Failed | IN919 OUT5 831 |
| × | RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15) |
Failed | IN930 OUT5 323 |
| × | RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7) |
Failed | IN955 OUT2 156 |
| ✓ | RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN917 OUT819 |
Instruction Following - Ratios & Stop Words
13 passed3 failed4 broken |
65.0% |
IN19 171 OUT186 130 | |
| ! | RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN922 OUT7 545 |
| ✓ | RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN943 OUT523 |
| ✓ | RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN905 OUT1 882 |
| ✓ | RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN948 OUT3 968 |
| ✓ | RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 029 OUT1 468 |
| ! | RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN981 OUT5 246 |
| ! | RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN958 OUT62 144 |
| × | RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN999 OUT8 523 |
| × | RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN982 OUT3 678 |
| ✓ | RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN977 OUT3 923 |
| × | RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN909 OUT126 |
| ✓ | RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN950 OUT3 687 |
| ! | RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 030 OUT62 144 |
| ✓ | RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN949 OUT3 263 |
| ✓ | RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN896 OUT1 111 |
| ✓ | RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN926 OUT3 313 |
| ✓ | RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN935 OUT5 799 |
| ✓ | RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN983 OUT2 073 |
| ✓ | RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 002 OUT1 855 |
| ✓ | RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN947 OUT3 859 |
Instruction Following - Sentence Progression
4 passed6 failed3 broken |
30.8% |
IN12 229 OUT229 987 | |
| ! | AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 002 OUT62 144 |
| ✓ | AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN920 OUT6 606 |
| × | AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN950 OUT22 882 |
| × | AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN938 OUT12 318 |
| × | AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN930 OUT1 309 |
| ✓ | SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN943 OUT9 046 |
| ✓ | SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN959 OUT11 477 |
| ✓ | SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN899 OUT4 995 |
| ! | SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN934 OUT5 682 |
| ! | SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN903 OUT62 144 |
| × | SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3) |
Failed | IN938 OUT10 742 |
| × | SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4) |
Failed | IN921 OUT7 477 |
| × | SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5) |
Failed | IN992 OUT13 165 |
Instruction Following - Sentence Words & Overlap
1 passed1 failed3 broken9 skipped |
20.0% |
IN4 736 OUT67 367 | |
| – | RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| ! | RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN920 OUT17 706 |
| ✓ | RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN933 OUT9 773 |
| ! | RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN934 OUT16 839 |
| ! | RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN992 OUT9 062 |
| × | RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN957 OUT13 987 |
Instruction Following - Templates & Whitespace
13 passed5 failed |
72.2% |
IN16 513 OUT41 793 | |
| × | NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN888 OUT311 |
| × | NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN889 OUT833 |
| × | NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN886 OUT982 |
| × | NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text) |
Failed | IN925 OUT812 |
| ✓ | OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN903 OUT1 392 |
| ✓ | OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN911 OUT1 506 |
| ✓ | OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN984 OUT5 143 |
| ✓ | OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN921 OUT1 329 |
| ✓ | Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN929 OUT1 745 |
| ✓ | Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN938 OUT2 010 |
| ✓ | Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN913 OUT1 242 |
| ✓ | Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN925 OUT6 085 |
| ✓ | Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN899 OUT1 215 |
| × | ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 231, 239) |
Failed | IN914 OUT4 966 |
| ✓ | TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN926 OUT4 757 |
| ✓ | TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN912 OUT2 223 |
| ✓ | TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN952 OUT3 942 |
| ✓ | TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN898 OUT1 300 |
Instruction Following - Vowels & Word Repeats
5 passed1 failed9 broken |
33.3% |
IN14 038 OUT284 488 | |
| ✓ | LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN970 OUT7 213 |
| ✓ | LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN907 OUT9 996 |
| ✓ | LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN902 OUT2 377 |
| ✓ | LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 018 OUT2 208 |
| ✓ | LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN900 OUT2 882 |
| ! | ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN900 OUT5 701 |
| ! | ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN898 OUT62 144 |
| ! | ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN932 OUT7 539 |
| × | ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN895 OUT4 126 |
| ! | ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN961 OUT16 007 |
| ! | ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 007 OUT11 386 |
| ! | ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN891 OUT62 144 |
| ! | VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN946 OUT8 482 |
| ! | VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN998 OUT62 144 |
| ! | VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN913 OUT20 139 |
Instruction Following - Word Counts
14 passed4 failed1 broken |
73.7% |
IN17 936 OUT75 944 | |
| ✓ | Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN919 OUT2 476 |
| ✓ | Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN930 OUT7 579 |
| × | Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 17) |
Failed | IN909 OUT7 243 |
| × | Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5) |
Failed | IN899 OUT4 176 |
| × | Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 25) |
Failed | IN923 OUT3 245 |
| ✓ | UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 102 OUT3 213 |
| ✓ | UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN972 OUT2 387 |
| ! | UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN888 OUT3 029 |
| ✓ | UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN897 OUT3 641 |
| ✓ | UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 049 OUT2 119 |
| ✓ | UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN893 OUT8 708 |
| ✓ | UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN891 OUT2 187 |
| ✓ | UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN991 OUT6 748 |
| ✓ | WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 004 OUT3 120 |
| ✓ | WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN905 OUT3 707 |
| ✓ | WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN969 OUT2 551 |
| ✓ | WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN936 OUT4 407 |
| ✓ | WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN903 OUT2 350 |
| × | WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?") |
Failed | IN956 OUT3 058 |
Instruction Following - Word Position & Order
11 passed3 failed4 broken6 skipped |
61.1% |
IN17 088 OUT207 776 | |
| ✓ | KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN922 OUT8 172 |
| ✓ | KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN927 OUT4 074 |
| × | KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "spotty", 24, 35) |
Failed | IN990 OUT10 584 |
| ! | KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN972 OUT9 710 |
| ! | LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN913 OUT3 647 |
| × | LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN911 OUT2 261 |
| ✓ | LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN918 OUT1 510 |
| ✓ | LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 016 OUT10 448 |
| × | LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN968 OUT13 205 |
| ! | ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN908 OUT62 144 |
| ✓ | ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN945 OUT4 087 |
| ✓ | ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN981 OUT2 917 |
| ✓ | ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN913 OUT2 027 |
| ! | ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN999 OUT62 144 |
| ✓ | ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN922 OUT2 538 |
| – | StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN992 OUT3 460 |
| ✓ | WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN977 OUT2 615 |
| ✓ | WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN914 OUT2 233 |