Benchmark summary
IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Instruction Following - Alphabet & Consonants
2 passed17 failed |
10.5% |
IN18 140 OUT6 759 | |
| × | AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN954 OUT183 |
| × | AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN992 OUT138 |
| × | AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN928 OUT114 |
| × | AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN949 OUT70 |
| × | AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN944 OUT34 |
| × | AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer) |
Failed | IN999 OUT104 |
| × | ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN981 OUT48 |
| × | ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN935 OUT2 651 |
| × | ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN948 OUT39 |
| × | ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN921 OUT15 |
| × | ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN995 OUT31 |
| × | ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN920 OUT58 |
| × | ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN971 OUT87 |
| × | ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN957 OUT6 |
| ✓ | ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN935 OUT11 |
| × | ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN919 OUT89 |
| × | ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN1 008 OUT69 |
| ✓ | ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench |
Passed | IN920 OUT8 |
| × | ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer) |
Failed | IN964 OUT3 004 |
Instruction Following - Brackets & Quotes
13 passed5 failed1 broken1 skipped |
68.4% |
IN18 009 OUT10 322 | |
| ✓ | Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN953 OUT10 |
| ✓ | Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN928 OUT541 |
| ✓ | Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN911 OUT311 |
| × | Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedParentheses(answer) |
Failed | IN1 002 OUT156 |
| ! | Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN900 OUT4 760 |
| ✓ | Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 008 OUT11 |
| ✓ | Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN946 OUT13 |
| × | Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN934 OUT184 |
| ✓ | Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN909 OUT241 |
| ✓ | Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN917 OUT40 |
| × | Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN924 OUT104 |
| × | Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN924 OUT62 |
| × | Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer) |
Failed | IN1 006 OUT283 |
| ✓ | Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN985 OUT1 729 |
| – | QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN954 OUT1 070 |
| ✓ | QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN944 OUT362 |
| ✓ | QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN954 OUT90 |
| ✓ | QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN914 OUT38 |
| ✓ | QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN996 OUT317 |
Instruction Following - Bullet Points
16 passed1 failed |
94.1% |
IN16 093 OUT2 821 | |
| ✓ | NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN925 OUT213 |
| ✓ | NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN933 OUT114 |
| ✓ | NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN985 OUT123 |
| ✓ | NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN907 OUT85 |
| ✓ | NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 011 OUT130 |
| ✓ | SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN914 OUT189 |
| × | SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer) |
Failed | IN921 OUT274 |
| ✓ | SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN996 OUT468 |
| ✓ | SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN948 OUT139 |
| ✓ | SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN961 OUT98 |
| ✓ | SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN986 OUT44 |
| ✓ | SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN924 OUT204 |
| ✓ | SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN952 OUT279 |
| ✓ | SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN925 OUT75 |
| ✓ | SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN932 OUT245 |
| ✓ | SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN948 OUT72 |
| ✓ | SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN925 OUT69 |
Instruction Following - Custom Transformations
4 passed7 failed |
36.4% |
IN9 976 OUT6 502 | |
| × | CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharacterReverse(answer) |
Failed | IN904 OUT14 |
| × | CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckCsvCity(answer) |
Failed | IN909 OUT130 |
| ✓ | CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN920 OUT105 |
| ✓ | CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN927 OUT314 |
| ✓ | DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN903 OUT4 355 |
| × | EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckEuropeanCapitals(answer) |
Failed | IN916 OUT39 |
| × | Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckMcq(answer) |
Failed | IN916 OUT484 |
| ✓ | Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN890 OUT18 |
| × | ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckReverseNewline(answer) |
Failed | IN889 OUT700 |
| × | SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceAlphabet(answer) |
Failed | IN898 OUT334 |
| × | WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordReverse(answer) |
Failed | IN904 OUT9 |
Instruction Following - Indentation & Emoji
5 passed4 failed8 skipped |
55.6% |
IN8 331 OUT3 707 | |
| – | Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN901 OUT231 |
| × | LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer) |
Failed | IN893 OUT114 |
| ✓ | LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN902 OUT240 |
| ✓ | LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN944 OUT50 |
| × | LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer) |
Failed | IN920 OUT373 |
| ✓ | LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN969 OUT716 |
| ✓ | LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN942 OUT84 |
| × | LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer) |
Failed | IN912 OUT51 |
| × | LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharCountUniqueWords(answer) |
Failed | IN948 OUT1 848 |
Instruction Following - Keyword & Conjunction Counts
13 passed5 failed |
72.2% |
IN17 250 OUT5 449 | |
| ✓ | Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN988 OUT1 011 |
| ✓ | Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN984 OUT297 |
| ✓ | Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN947 OUT249 |
| ✓ | Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 010 OUT191 |
| ✓ | Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN985 OUT102 |
| × | KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "kaleidoscope", "nebula", "whisper", "labyrinth", "paradox") |
Failed | IN1 004 OUT351 |
| × | KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "meridian", "gossamer", "eclipse", "threshold", "cascade") |
Failed | IN926 OUT148 |
| × | KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "door", "bread", "blue", "lamp", "river") |
Failed | IN1 009 OUT270 |
| ✓ | KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 033 OUT654 |
| ✓ | KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN953 OUT261 |
| ✓ | Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN895 OUT38 |
| ✓ | Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN982 OUT74 |
| × | Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 13) |
Failed | IN953 OUT312 |
| ✓ | Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN948 OUT26 |
| ✓ | Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN916 OUT21 |
| ✓ | NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN903 OUT139 |
| ✓ | NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN900 OUT87 |
| × | NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 36) |
Failed | IN914 OUT1 218 |
Instruction Following - Keyword In Nth Sentence
2 passed4 failed3 broken |
22.2% |
IN8 376 OUT12 111 | |
| ! | KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 020 OUT3 708 |
| × | KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "migratory", 28) |
Failed | IN916 OUT442 |
| ✓ | KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN901 OUT105 |
| ✓ | KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN895 OUT356 |
| ! | KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN938 OUT2 877 |
| ! | KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN962 OUT3 049 |
| × | KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "iritic", 5) |
Failed | IN914 OUT143 |
| × | KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "cactiform", 4) |
Failed | IN920 OUT192 |
| × | KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "betask", 18) |
Failed | IN910 OUT1 239 |
Instruction Following - Lists, Lines & Options
15 passed4 failed |
78.9% |
IN17 900 OUT4 048 | |
| ✓ | List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN964 OUT64 |
| ✓ | List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN904 OUT127 |
| × | List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?") |
Failed | IN918 OUT252 |
| ✓ | List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN938 OUT68 |
| ✓ | List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN905 OUT247 |
| × | ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN1 012 OUT365 |
| ✓ | ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN944 OUT18 |
| × | ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "-") |
Failed | IN916 OUT909 |
| ✓ | NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN918 OUT431 |
| ✓ | NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN958 OUT8 |
| ✓ | NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN929 OUT425 |
| ✓ | NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN939 OUT1 115 |
| ✓ | NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN988 OUT3 |
| ✓ | Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN910 OUT2 |
| ✓ | Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN934 OUT3 |
| ✓ | Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN1 019 OUT3 |
| ✓ | Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN924 OUT2 |
| ✓ | Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN962 OUT3 |
| × | OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNewLineWords(answer) |
Failed | IN918 OUT3 |
Instruction Following - Names, Pronouns & Punctuation
9 passed3 failed4 broken |
56.3% |
IN15 708 OUT79 460 | |
| ✓ | PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 051 OUT146 |
| ! | PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 054 OUT3 072 |
| ✓ | PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN1 050 OUT182 |
| ✓ | PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 017 OUT451 |
| ✓ | PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 067 OUT376 |
| ✓ | Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN894 OUT197 |
| ✓ | Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN909 OUT949 |
| ✓ | Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN973 OUT165 |
| ✓ | Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN975 OUT52 |
| ! | Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN903 OUT4 310 |
| ! | Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 004 OUT62 144 |
| ! | Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN1 010 OUT5 584 |
| × | Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN963 OUT34 |
| ✓ | Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN951 OUT1 443 |
| × | Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN925 OUT57 |
| × | Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer) |
Failed | IN962 OUT298 |
Instruction Following - Palindromes & Letter Patterns
4 passed11 failed2 broken5 skipped |
23.5% |
IN16 132 OUT70 504 | |
| ✓ | NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN933 OUT8 |
| × | NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN906 OUT15 |
| × | NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer) |
Failed | IN905 OUT92 |
| ! | NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN996 OUT3 012 |
| ! | NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN940 OUT62 144 |
| – | OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#. |
Skipped | IN0 OUT0 |
| ✓ | Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN929 OUT24 |
| × | Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer) |
Failed | IN951 OUT323 |
| × | Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer) |
Failed | IN923 OUT373 |
| × | Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer) |
Failed | IN920 OUT1 920 |
| ✓ | Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN939 OUT1 099 |
| × | Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer) |
Failed | IN1 043 OUT238 |
| ✓ | Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 119 OUT313 |
| × | PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer) |
Failed | IN891 OUT29 |
| × | PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer) |
Failed | IN923 OUT11 |
| × | PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer) |
Failed | IN942 OUT128 |
| × | PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer) |
Failed | IN888 OUT771 |
| × | PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer) |
Failed | IN984 OUT4 |
Instruction Following - Prompt Repetition
1 passed8 failed |
11.1% |
IN8 533 OUT276 | |
| × | RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "Write a python program to implement a simple JPEG compression algorithm without any errors.") |
Failed | IN932 OUT60 |
| × | RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "Use Python code: construct a 3-layers feedforward network with 8 input neurons, 20 hidden neurons, and 2 output neurons.") |
Failed | IN948 OUT28 |
| × | RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "The random, uniform generation of derangements is very simple. Since Dn ≈ n!/e, the obvious procedure of generating a random permutation and check if it is a derangement or not, and generating a new permutation in the negative case, is straight-forward and guarantees a linear time complexity, at least on the average; actually, as we will see, the average complexity is µ1 ≈ e(n − 1).") |
Failed | IN1 003 OUT131 |
| ✓ | RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 027 OUT12 |
| × | RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer) |
Failed | IN894 OUT12 |
| × | RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6) |
Failed | IN921 OUT5 |
| × | RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15) |
Failed | IN932 OUT20 |
| × | RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7) |
Failed | IN957 OUT5 |
| × | RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "how to find what value in a quadratic function lands on a certain point. ", 10, 12) |
Failed | IN919 OUT3 |
Instruction Following - Ratios & Stop Words
7 passed13 failed |
35.0% |
IN19 211 OUT18 958 | |
| × | RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN924 OUT2 161 |
| × | RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN945 OUT39 |
| × | RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN907 OUT93 |
| × | RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN950 OUT20 |
| × | RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN1 031 OUT181 |
| × | RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN983 OUT14 298 |
| × | RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer) |
Failed | IN960 OUT95 |
| × | RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN1 001 OUT239 |
| × | RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN984 OUT106 |
| × | RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN979 OUT250 |
| × | RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN911 OUT48 |
| × | RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer) |
Failed | IN952 OUT174 |
| ✓ | RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 032 OUT61 |
| ✓ | RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN951 OUT119 |
| ✓ | RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN898 OUT95 |
| × | RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckStopWords(answer, 20) |
Failed | IN928 OUT102 |
| ✓ | RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN937 OUT102 |
| ✓ | RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN985 OUT93 |
| ✓ | RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 004 OUT530 |
| ✓ | RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN949 OUT152 |
Instruction Following - Sentence Progression
1 passed12 failed |
7.7% |
IN12 255 OUT1 411 | |
| ✓ | AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN1 004 OUT23 |
| × | AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN922 OUT32 |
| × | AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN952 OUT79 |
| × | AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN940 OUT191 |
| × | AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer) |
Failed | IN932 OUT118 |
| × | SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4) |
Failed | IN945 OUT107 |
| × | SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3) |
Failed | IN961 OUT76 |
| × | SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5) |
Failed | IN901 OUT180 |
| × | SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4) |
Failed | IN936 OUT47 |
| × | SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 1) |
Failed | IN905 OUT126 |
| × | SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3) |
Failed | IN940 OUT138 |
| × | SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4) |
Failed | IN923 OUT37 |
| × | SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5) |
Failed | IN994 OUT257 |
Instruction Following - Sentence Words & Overlap
5 failed9 skipped |
0.0% |
IN4 746 OUT202 | |
| – | RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| – | RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent. |
Skipped | IN0 OUT0 |
| × | RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN922 OUT25 |
| × | RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN935 OUT54 |
| × | RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN936 OUT38 |
| × | RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN994 OUT24 |
| × | RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer) |
Failed | IN959 OUT61 |
Instruction Following - Templates & Whitespace
10 passed7 failed1 broken |
55.6% |
IN16 549 OUT10 424 | |
| ✓ | NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN890 OUT2 |
| ✓ | NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN891 OUT96 |
| ✓ | NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN888 OUT14 |
| ✓ | NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN927 OUT53 |
| ✓ | OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN905 OUT98 |
| ✓ | OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN913 OUT394 |
| ✓ | OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN986 OUT472 |
| ✓ | OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN923 OUT223 |
| × | Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckThesis(answer) |
Failed | IN931 OUT151 |
| × | Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckThesis(answer) |
Failed | IN940 OUT171 |
| ✓ | Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN915 OUT791 |
| × | Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckThesis(answer) |
Failed | IN927 OUT1 283 |
| × | Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckThesis(answer) |
Failed | IN901 OUT237 |
| × | ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckThesis(answer) |
Failed | IN916 OUT411 |
| ! | TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN928 OUT4 749 |
| × | TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckTitleCase(answer) |
Failed | IN914 OUT160 |
| × | TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckTitleCase(answer) |
Failed | IN954 OUT1 092 |
| ✓ | TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN900 OUT27 |
Instruction Following - Vowels & Word Repeats
3 passed10 failed2 broken |
20.0% |
IN14 068 OUT71 429 | |
| ! | LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN972 OUT62 144 |
| ! | LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed
Values:
response.Completed = false |
Broken | IN909 OUT7 391 |
| ✓ | LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN904 OUT53 |
| ✓ | LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 020 OUT76 |
| ✓ | LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN902 OUT72 |
| × | ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN902 OUT134 |
| × | ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN900 OUT91 |
| × | ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN934 OUT207 |
| × | ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN897 OUT168 |
| × | ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN963 OUT63 |
| × | ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN1 009 OUT534 |
| × | ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN893 OUT80 |
| × | VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN948 OUT190 |
| × | VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN1 000 OUT56 |
| × | VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer) |
Failed | IN915 OUT170 |
Instruction Following - Word Counts
13 passed6 failed |
68.4% |
IN17 974 OUT7 162 | |
| × | Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5) |
Failed | IN921 OUT48 |
| × | Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 29) |
Failed | IN932 OUT228 |
| × | Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 17) |
Failed | IN911 OUT428 |
| ✓ | Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN901 OUT536 |
| × | Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 25) |
Failed | IN925 OUT333 |
| ✓ | UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 104 OUT267 |
| ✓ | UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN974 OUT82 |
| ✓ | UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN890 OUT3 017 |
| ✓ | UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN899 OUT223 |
| ✓ | UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN1 051 OUT191 |
| ✓ | UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN895 OUT294 |
| ✓ | UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN893 OUT548 |
| ✓ | UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench |
Passed | IN993 OUT420 |
| × | WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 71, 73) |
Failed | IN1 006 OUT83 |
| ✓ | WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN907 OUT66 |
| ✓ | WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN971 OUT32 |
| ✓ | WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN938 OUT132 |
| ✓ | WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench |
Passed | IN905 OUT76 |
| × | WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 90, 100) |
Failed | IN958 OUT158 |
Instruction Following - Word Position & Order
3 passed15 failed6 skipped |
16.7% |
IN17 124 OUT8 290 | |
| × | KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "giggle", 22, 33) |
Failed | IN924 OUT54 |
| × | KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "beneath", 23, 34) |
Failed | IN929 OUT103 |
| × | KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "spotty", 24, 35) |
Failed | IN992 OUT1 928 |
| × | KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "slowly", 25, 36) |
Failed | IN974 OUT2 248 |
| × | LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN915 OUT950 |
| × | LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN913 OUT424 |
| × | LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN920 OUT1 243 |
| × | LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN1 018 OUT59 |
| × | LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query
0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer) |
Failed | IN970 OUT120 |
| × | ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer) |
Failed | IN910 OUT291 |
| ✓ | ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN947 OUT21 |
| ✓ | ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN983 OUT5 |
| × | ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer) |
Failed | IN915 OUT226 |
| × | ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer) |
Failed | IN1 001 OUT178 |
| ✓ | ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench |
Passed | IN924 OUT11 |
| – | StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| – | StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries
0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#. |
Skipped | IN0 OUT0 |
| × | WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordsPosition(answer, "vibrant") |
Failed | IN994 OUT403 |
| × | WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordsPosition(answer, "whisper") |
Failed | IN979 OUT5 |
| × | WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordsPosition(answer, "despite") |
Failed | IN916 OUT21 |