Ai4Users - Qwen 3.8 Flash Next NVFP4 Instruct

Benchmark items
Instruction Following - Alphabet & Consonants 6 passed13 failed	31.6%	IN18 140 OUT995
×	AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer)	Failed	IN954 OUT22
×	AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer)	Failed	IN992 OUT40
×	AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer)	Failed	IN928 OUT8
✓	AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN949 OUT3
×	AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer)	Failed	IN944 OUT24
✓	AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN999 OUT21
×	ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN981 OUT88
×	ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN935 OUT524
×	ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN948 OUT47
×	ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN921 OUT31
×	ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN995 OUT16
×	ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN920 OUT7
✓	ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN971 OUT11
×	ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN957 OUT19
×	ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN935 OUT103
✓	ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN919 OUT9
✓	ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN1 008 OUT11
✓	ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN920 OUT6
×	ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN964 OUT5
Instruction Following - Brackets & Quotes 13 passed6 failed1 skipped	68.4%	IN18 009 OUT3 229
✓	Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN953 OUT62
✓	Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN928 OUT181
✓	Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN911 OUT187
✓	Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 002 OUT154
✓	Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN900 OUT51
✓	Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 008 OUT6
✓	Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN946 OUT7
✓	Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN934 OUT161
×	Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN909 OUT170
×	Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN917 OUT3
×	Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN924 OUT217
×	Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN924 OUT7
×	Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN1 006 OUT338
×	Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN985 OUT56
–	QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
✓	QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN954 OUT1 066
✓	QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN944 OUT194
✓	QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN954 OUT78
✓	QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN914 OUT2
✓	QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN996 OUT289
Instruction Following - Bullet Points 14 passed3 failed	82.4%	IN16 093 OUT3 127
✓	NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN925 OUT217
✓	NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN933 OUT172
✓	NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN985 OUT72
✓	NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN907 OUT121
✓	NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 011 OUT147
✓	SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN914 OUT110
✓	SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN921 OUT247
×	SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer)	Failed	IN996 OUT351
✓	SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN948 OUT420
✓	SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN961 OUT85
✓	SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN986 OUT87
×	SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer)	Failed	IN924 OUT366
✓	SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN952 OUT164
✓	SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN925 OUT131
✓	SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN932 OUT161
×	SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer)	Failed	IN948 OUT202
✓	SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN925 OUT74
Instruction Following - Custom Transformations 5 passed6 failed	45.5%	IN9 976 OUT2 242
×	CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharacterReverse(answer)	Failed	IN904 OUT16
✓	CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN909 OUT119
✓	CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN920 OUT103
✓	CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN927 OUT278
×	DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckDateFormatList(answer)	Failed	IN903 OUT20
×	EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckEuropeanCapitals(answer)	Failed	IN916 OUT32
✓	Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN916 OUT422
×	Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckMultiples(answer)	Failed	IN890 OUT15
×	ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckReverseNewline(answer)	Failed	IN889 OUT887
✓	SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN898 OUT339
×	WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordReverse(answer)	Failed	IN904 OUT11
Instruction Following - Indentation & Emoji 4 passed5 failed8 skipped	44.4%	IN8 331 OUT2 670
–	Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
×	LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer)	Failed	IN901 OUT177
×	LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer)	Failed	IN893 OUT1 036
×	LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer)	Failed	IN902 OUT398
✓	LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN944 OUT44
✓	LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN920 OUT104
✓	LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN969 OUT563
✓	LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN942 OUT71
×	LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)	Failed	IN912 OUT43
×	LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIndentStairs(answer)	Failed	IN948 OUT234
Instruction Following - Keyword & Conjunction Counts 5 passed13 failed	27.8%	IN17 250 OUT3 184
×	Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckConjunctions(answer, 5)	Failed	IN988 OUT422
✓	Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN984 OUT166
✓	Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN947 OUT74
×	Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckConjunctions(answer, 4)	Failed	IN1 010 OUT160
✓	Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN985 OUT97
×	KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "kaleidoscope", "nebula", "whisper", "labyrinth", "paradox")	Failed	IN1 004 OUT461
×	KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "meridian", "gossamer", "eclipse", "threshold", "cascade")	Failed	IN926 OUT176
×	KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "door", "bread", "blue", "lamp", "river")	Failed	IN1 009 OUT542
×	KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "cloud", "rhythm", "fish", "puzzle", "dawn")	Failed	IN1 033 OUT501
×	KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "quickly", "transform", "abundant", "bridge", "through")	Failed	IN953 OUT7
×	Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 2)	Failed	IN895 OUT16
✓	Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN982 OUT79
×	Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 13)	Failed	IN953 OUT136
✓	Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN948 OUT35
×	Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 5)	Failed	IN916 OUT14
×	NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 15)	Failed	IN903 OUT68
×	NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckConjunctions(answer, 4)	Failed	IN900 OUT79
×	NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckNumbersCount(answer, 36)	Failed	IN914 OUT151
Instruction Following - Keyword In Nth Sentence 3 passed6 failed	33.3%	IN8 376 OUT5 632
×	KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "malabathrum", 27)	Failed	IN1 020 OUT2 903
✓	KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN916 OUT592
✓	KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN901 OUT77
×	KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "heterolith", 15)	Failed	IN895 OUT165
✓	KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN938 OUT403
×	KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "trichosis", 22)	Failed	IN962 OUT893
×	KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 89, 99)	Failed	IN914 OUT117
×	KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 81, 93)	Failed	IN920 OUT123
×	KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "betask", 18)	Failed	IN910 OUT359
Instruction Following - Lists, Lines & Options 13 passed6 failed	68.4%	IN17 900 OUT1 761
×	List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "SEPARATOR")	Failed	IN964 OUT42
✓	List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN904 OUT136
✓	List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN918 OUT282
✓	List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN938 OUT21
✓	List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN905 OUT191
×	ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN1 012 OUT89
×	ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")	Failed	IN944 OUT8
×	ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN916 OUT155
✓	NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN918 OUT12
✓	NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN958 OUT10
×	NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNewLineWords(answer)	Failed	IN929 OUT493
✓	NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN939 OUT299
✓	NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN988 OUT7
✓	Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN910 OUT2
✓	Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN934 OUT3
✓	Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN1 019 OUT3
✓	Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN924 OUT2
✓	Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN962 OUT3
×	OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckNewLineWords(answer)	Failed	IN918 OUT3
Instruction Following - Names, Pronouns & Punctuation 11 passed5 failed	68.8%	IN15 708 OUT8 013
✓	PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 051 OUT130
✓	PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 054 OUT589
×	PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Assertion expression evaluated to false: CheckPersonNames(answer, 3)	Failed	IN1 050 OUT9
✓	PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 017 OUT290
✓	PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 067 OUT584
✓	Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN894 OUT153
✓	Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN909 OUT1 023
✓	Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN973 OUT586
✓	Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN975 OUT28
✓	Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN903 OUT738
×	Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN1 004 OUT2 599
×	Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN1 010 OUT29
×	Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN963 OUT70
✓	Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN951 OUT657
✓	Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN925 OUT116
×	Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN962 OUT412
Instruction Following - Palindromes & Letter Patterns 7 passed10 failed5 skipped	41.2%	IN16 132 OUT4 025
✓	NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN933 OUT8
×	NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN906 OUT14
✓	NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN905 OUT64
×	NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN996 OUT173
×	NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN940 OUT221
–	OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
×	Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer)	Failed	IN929 OUT37
×	Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPalindromes(answer)	Failed	IN951 OUT281
✓	Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN923 OUT111
✓	Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN920 OUT1 494
✓	Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN939 OUT917
✓	Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 043 OUT207
✓	Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 119 OUT349
×	PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer)	Failed	IN891 OUT7
×	PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer)	Failed	IN923 OUT13
×	PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer)	Failed	IN942 OUT116
×	PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer)	Failed	IN888 OUT10
×	PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckPrimeLengths(answer)	Failed	IN984 OUT3
Instruction Following - Prompt Repetition 3 passed6 failed	33.3%	IN8 533 OUT601
×	RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatChange(answer, "Write a python program to implement a simple JPEG compression algorithm without any errors.")	Failed	IN932 OUT60
✓	RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN948 OUT32
✓	RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 003 OUT87
✓	RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 027 OUT12
×	RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer)	Failed	IN894 OUT17
×	RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)	Failed	IN921 OUT120
×	RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)	Failed	IN932 OUT214
×	RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7)	Failed	IN957 OUT5
×	RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "how to find what value in a quadratic function lands on a certain point. ", 10, 12)	Failed	IN919 OUT54
Instruction Following - Ratios & Stop Words 8 passed12 failed	40.0%	IN19 211 OUT5 210
×	RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN924 OUT1 789
×	RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN945 OUT5
×	RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN907 OUT30
✓	RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN950 OUT28
×	RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN1 031 OUT14
✓	RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN983 OUT11
×	RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN960 OUT110
×	RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN1 001 OUT304
×	RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN984 OUT107
×	RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN979 OUT198
×	RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN911 OUT30
×	RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN952 OUT1 029
×	RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN1 032 OUT89
✓	RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN951 OUT7
✓	RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN898 OUT36
×	RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckStopWords(answer, 20)	Failed	IN928 OUT3
✓	RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN937 OUT64
✓	RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN985 OUT15
✓	RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 004 OUT1 294
✓	RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN949 OUT47
Instruction Following - Sentence Progression 2 passed11 failed	15.4%	IN12 255 OUT1 446
✓	AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 004 OUT27
✓	AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN922 OUT2
×	AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN952 OUT282
×	AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN940 OUT203
×	AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN932 OUT49
×	SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)	Failed	IN945 OUT123
×	SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)	Failed	IN961 OUT275
×	SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)	Failed	IN901 OUT54
×	SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)	Failed	IN936 OUT40
×	SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 1)	Failed	IN905 OUT80
×	SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)	Failed	IN940 OUT161
×	SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)	Failed	IN923 OUT51
×	SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)	Failed	IN994 OUT99
Instruction Following - Sentence Words & Overlap 5 failed9 skipped	0.0%	IN4 746 OUT101
–	RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
×	RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN922 OUT16
×	RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN935 OUT15
×	RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN936 OUT17
×	RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN994 OUT33
×	RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN959 OUT20
Instruction Following - Templates & Whitespace 12 passed6 failed	66.7%	IN16 549 OUT4 553
✓	NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN890 OUT2
✓	NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN891 OUT7
×	NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text)	Failed	IN888 OUT35
×	NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text)	Failed	IN927 OUT87
✓	OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN905 OUT192
✓	OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN913 OUT305
✓	OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN986 OUT95
✓	OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN923 OUT188
✓	Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN931 OUT34
×	Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckThesis(answer)	Failed	IN940 OUT31
✓	Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN915 OUT822
✓	Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN927 OUT822
✓	Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN901 OUT31
×	ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 231, 239)	Failed	IN916 OUT286
✓	TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN928 OUT701
×	TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckTitleCase(answer)	Failed	IN914 OUT102
×	TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckTitleCase(answer)	Failed	IN954 OUT752
✓	TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN900 OUT61
Instruction Following - Vowels & Word Repeats 3 passed12 failed	20.0%	IN14 068 OUT4 478
×	LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)	Failed	IN972 OUT2 519
×	LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)	Failed	IN909 OUT466
✓	LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN904 OUT44
✓	LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 020 OUT39
✓	LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN902 OUT34
×	ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN902 OUT113
×	ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN900 OUT65
×	ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN934 OUT295
×	ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN897 OUT150
×	ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN963 OUT15
×	ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN1 009 OUT380
×	ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN893 OUT34
×	VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN948 OUT135
×	VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN1 000 OUT77
×	VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN915 OUT112
Instruction Following - Word Counts 11 passed8 failed	57.9%	IN17 974 OUT4 368
×	Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5)	Failed	IN921 OUT38
×	Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 29)	Failed	IN932 OUT497
×	Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 17)	Failed	IN911 OUT154
×	Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5)	Failed	IN901 OUT662
×	Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 25)	Failed	IN925 OUT291
✓	UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 104 OUT124
✓	UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN974 OUT90
✓	UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN890 OUT622
✓	UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN899 OUT237
✓	UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 051 OUT166
✓	UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN895 OUT403
✓	UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN893 OUT232
✓	UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN993 OUT330
✓	WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 006 OUT84
×	WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 57, 61)	Failed	IN907 OUT62
×	WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 24, 28)	Failed	IN971 OUT30
✓	WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN938 OUT111
✓	WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN905 OUT76
×	WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 90, 100)	Failed	IN958 OUT159
Instruction Following - Word Position & Order 3 passed15 failed6 skipped	16.7%	IN17 124 OUT4 200
×	KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "giggle", 22, 33)	Failed	IN924 OUT24
×	KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "beneath", 23, 34)	Failed	IN929 OUT37
×	KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "spotty", 24, 35)	Failed	IN992 OUT94
×	KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "slowly", 25, 36)	Failed	IN974 OUT62
✓	LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN915 OUT44
×	LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer)	Failed	IN913 OUT723
×	LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer)	Failed	IN920 OUT1 942
×	LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer)	Failed	IN1 018 OUT122
✓	LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN970 OUT636
×	ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer)	Failed	IN910 OUT47
×	ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer)	Failed	IN947 OUT16
×	ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer)	Failed	IN983 OUT204
×	ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer)	Failed	IN915 OUT126
✓	ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 001 OUT24
×	ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer)	Failed	IN924 OUT68
–	StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
×	WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordsPosition(answer, "vibrant")	Failed	IN994 OUT15
×	WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordsPosition(answer, "whisper")	Failed	IN979 OUT3
×	WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckWordsPosition(answer, "despite")	Failed	IN916 OUT13

Instruction Following - Alphabet & Consonants 6 passed13 failed

31.6%

IN18 140 OUT995

×

AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckAlphabetLoop(answer)

Failed

IN954 OUT22

×

AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckAlphabetLoop(answer)

Failed

IN992 OUT40

×

AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckAlphabetLoop(answer)

Failed

IN928 OUT8

✓

AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN949 OUT3

×

AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckAlphabetLoop(answer)

Failed

IN944 OUT24

✓

AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN999 OUT21

×

ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN981 OUT88

×

ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN935 OUT524

×

ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN948 OUT47

×

ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN921 OUT31

×

ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN995 OUT16

×

ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN920 OUT7

✓

ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN971 OUT11

×

ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN957 OUT19

×

ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN935 OUT103

✓

ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN919 OUT9

✓

ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN1 008 OUT11

✓

ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN920 OUT6

×

ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN964 OUT5

Instruction Following - Brackets & Quotes 13 passed6 failed1 skipped

68.4%

IN18 009 OUT3 229

✓

Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN953 OUT62

✓

Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN928 OUT181

✓

Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN911 OUT187

✓

Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 002 OUT154

✓

Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN900 OUT51

✓

Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 008 OUT6

✓

Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN946 OUT7

✓

Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN934 OUT161

×

Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN909 OUT170

×

Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN917 OUT3

×

Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN924 OUT217

×

Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN924 OUT7

×

Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN1 006 OUT338

×

Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN985 OUT56

–

QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

✓

QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN954 OUT1 066

✓

QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN944 OUT194

✓

QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN954 OUT78

✓

QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN914 OUT2

✓

QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN996 OUT289

Instruction Following - Bullet Points 14 passed3 failed

82.4%

IN16 093 OUT3 127

✓

NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN925 OUT217

✓

NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN933 OUT172

✓

NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN985 OUT72

✓

NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN907 OUT121

✓

NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 011 OUT147

✓

SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN914 OUT110

✓

SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN921 OUT247

×

SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckSubBullets(answer)

Failed

IN996 OUT351

✓

SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN948 OUT420

✓

SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN961 OUT85

✓

SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN986 OUT87

×

SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckSubBullets(answer)

Failed

IN924 OUT366

✓

SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN952 OUT164

✓

SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN925 OUT131

✓

SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN932 OUT161

×

SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSubBullets(answer)

Failed

IN948 OUT202

✓

SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN925 OUT74

Instruction Following - Custom Transformations 5 passed6 failed

45.5%

IN9 976 OUT2 242

×

CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckCharacterReverse(answer)

Failed

IN904 OUT16

✓

CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN909 OUT119

✓

CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN920 OUT103

✓

CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN927 OUT278

×

DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckDateFormatList(answer)

Failed

IN903 OUT20

×

EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckEuropeanCapitals(answer)

Failed

IN916 OUT32

✓

Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN916 OUT422

×

Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckMultiples(answer)

Failed

IN890 OUT15

×

ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckReverseNewline(answer)

Failed

IN889 OUT887

✓

SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN898 OUT339

×

WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckWordReverse(answer)

Failed

IN904 OUT11

Instruction Following - Indentation & Emoji 4 passed5 failed8 skipped

44.4%

IN8 331 OUT2 670

–

Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

×

LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckIndentStairs(answer)

Failed

IN901 OUT177

×

LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckIndentStairs(answer)

Failed

IN893 OUT1 036

×

LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckIndentStairs(answer)

Failed

IN902 OUT398

✓

LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN944 OUT44

✓

LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN920 OUT104

✓

LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN969 OUT563

✓

LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN942 OUT71

×

LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)

Failed

IN912 OUT43

×

LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIndentStairs(answer)

Failed

IN948 OUT234

Instruction Following - Keyword & Conjunction Counts 5 passed13 failed

27.8%

IN17 250 OUT3 184

×

Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckConjunctions(answer, 5)

Failed

IN988 OUT422

✓

Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN984 OUT166

✓

Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN947 OUT74

×

Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckConjunctions(answer, 4)

Failed

IN1 010 OUT160

✓

Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN985 OUT97

×

KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "kaleidoscope", "nebula", "whisper", "labyrinth", "paradox")

Failed

IN1 004 OUT461

×

KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "meridian", "gossamer", "eclipse", "threshold", "cascade")

Failed

IN926 OUT176

×

KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "door", "bread", "blue", "lamp", "river")

Failed

IN1 009 OUT542

×

KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "cloud", "rhythm", "fish", "puzzle", "dawn")

Failed

IN1 033 OUT501

×

KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "quickly", "transform", "abundant", "bridge", "through")

Failed

IN953 OUT7

×

Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckNumbersCount(answer, 2)

Failed

IN895 OUT16

✓

Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN982 OUT79

×

Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNumbersCount(answer, 13)

Failed

IN953 OUT136

✓

Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN948 OUT35

×

Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckNumbersCount(answer, 5)

Failed

IN916 OUT14

×

NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNumbersCount(answer, 15)

Failed

IN903 OUT68

×

NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckConjunctions(answer, 4)

Failed

IN900 OUT79

×

NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckNumbersCount(answer, 36)

Failed

IN914 OUT151

Instruction Following - Keyword In Nth Sentence 3 passed6 failed

33.3%

IN8 376 OUT5 632

×

KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncludeKeyword(answer, "malabathrum", 27)

Failed

IN1 020 OUT2 903

✓

KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN916 OUT592

✓

KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN901 OUT77

×

KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncludeKeyword(answer, "heterolith", 15)

Failed

IN895 OUT165

✓

KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN938 OUT403

×

KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncludeKeyword(answer, "trichosis", 22)

Failed

IN962 OUT893

×

KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckWordCountRange(answer, 89, 99)

Failed

IN914 OUT117

×

KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckWordCountRange(answer, 81, 93)

Failed

IN920 OUT123

×

KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncludeKeyword(answer, "betask", 18)

Failed

IN910 OUT359

Instruction Following - Lists, Lines & Options 13 passed6 failed

68.4%

IN17 900 OUT1 761

×

List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckSpecialBullets(answer, "SEPARATOR")

Failed

IN964 OUT42

✓

List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN904 OUT136

✓

List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN918 OUT282

✓

List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN938 OUT21

✓

List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN905 OUT191

×

ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN1 012 OUT89

×

ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")

Failed

IN944 OUT8

×

ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN916 OUT155

✓

NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN918 OUT12

✓

NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN958 OUT10

×

NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckNewLineWords(answer)

Failed

IN929 OUT493

✓

NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN939 OUT299

✓

NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN988 OUT7

✓

Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN910 OUT2

✓

Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN934 OUT3

✓

Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN1 019 OUT3

✓

Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN924 OUT2

✓

Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN962 OUT3

×

OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckNewLineWords(answer)

Failed

IN918 OUT3

Instruction Following - Names, Pronouns & Punctuation 11 passed5 failed

68.8%

IN15 708 OUT8 013

✓

PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 051 OUT130

✓

PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 054 OUT589

×

PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Assertion expression evaluated to false: CheckPersonNames(answer, 3)

Failed

IN1 050 OUT9

✓

PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 017 OUT290

✓

PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 067 OUT584

✓

Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN894 OUT153

✓

Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN909 OUT1 023

✓

Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN973 OUT586

✓

Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN975 OUT28

✓

Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN903 OUT738

×

Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN1 004 OUT2 599

×

Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN1 010 OUT29

×

Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN963 OUT70

✓

Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN951 OUT657

✓

Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN925 OUT116

×

Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN962 OUT412

Instruction Following - Palindromes & Letter Patterns 7 passed10 failed5 skipped

41.2%

IN16 132 OUT4 025

✓

NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN933 OUT8

×

NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN906 OUT14

✓

NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN905 OUT64

×

NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN996 OUT173

×

NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN940 OUT221

–

OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

×

Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPalindromes(answer)

Failed

IN929 OUT37

×

Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPalindromes(answer)

Failed

IN951 OUT281

✓

Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN923 OUT111

✓

Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN920 OUT1 494

✓

Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN939 OUT917

✓

Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 043 OUT207

✓

Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 119 OUT349

×

PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckPrimeLengths(answer)

Failed

IN891 OUT7

×

PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckPrimeLengths(answer)

Failed

IN923 OUT13

×

PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckPrimeLengths(answer)

Failed

IN942 OUT116

×

PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckPrimeLengths(answer)

Failed

IN888 OUT10

×

PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckPrimeLengths(answer)

Failed

IN984 OUT3

Instruction Following - Prompt Repetition 3 passed6 failed

33.3%

IN8 533 OUT601

×

RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatChange(answer, "Write a python program to implement a simple JPEG compression algorithm without any errors.")

Failed

IN932 OUT60

✓

RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN948 OUT32

✓

RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 003 OUT87

✓

RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 027 OUT12

×

RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSimple(answer)

Failed

IN894 OUT17

×

RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)

Failed

IN921 OUT120

×

RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)

Failed

IN932 OUT214

×

RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7)

Failed

IN957 OUT5

×

RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "how to find what value in a quadratic function lands on a certain point. ", 10, 12)

Failed

IN919 OUT54

Instruction Following - Ratios & Stop Words 8 passed12 failed

40.0%

IN19 211 OUT5 210

×

RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN924 OUT1 789

×

RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN945 OUT5

×

RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN907 OUT30

✓

RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN950 OUT28

×

RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN1 031 OUT14

✓

RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN983 OUT11

×

RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN960 OUT110

×

RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN1 001 OUT304

×

RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN984 OUT107

×

RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN979 OUT198

×

RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN911 OUT30

×

RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN952 OUT1 029

×

RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN1 032 OUT89

✓

RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN951 OUT7

✓

RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN898 OUT36

×

RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckStopWords(answer, 20)

Failed

IN928 OUT3

✓

RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN937 OUT64

✓

RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN985 OUT15

✓

RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 004 OUT1 294

✓

RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN949 OUT47

Instruction Following - Sentence Progression 2 passed11 failed

15.4%

IN12 255 OUT1 446

✓

AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 004 OUT27

✓

AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN922 OUT2

×

AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN952 OUT282

×

AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN940 OUT203

×

AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN932 OUT49

×

SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)

Failed

IN945 OUT123

×

SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)

Failed

IN961 OUT275

×

SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)

Failed

IN901 OUT54

×

SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)

Failed

IN936 OUT40

×

SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 1)

Failed

IN905 OUT80

×

SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)

Failed

IN940 OUT161

×

SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)

Failed

IN923 OUT51

×

SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)

Failed

IN994 OUT99

Instruction Following - Sentence Words & Overlap 5 failed9 skipped

0.0%

IN4 746 OUT101

–

RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

×

RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN922 OUT16

×

RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN935 OUT15

×

RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN936 OUT17

×

RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN994 OUT33

×

RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN959 OUT20

Instruction Following - Templates & Whitespace 12 passed6 failed

66.7%

IN16 549 OUT4 553

✓

NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN890 OUT2

✓

NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN891 OUT7

×

NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoWhitespace(response.Text)

Failed

IN888 OUT35

×

NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoWhitespace(response.Text)

Failed

IN927 OUT87

✓

OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN905 OUT192

✓

OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN913 OUT305

✓

OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN986 OUT95

✓

OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN923 OUT188

✓

Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN931 OUT34

×

Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckThesis(answer)

Failed

IN940 OUT31

✓

Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN915 OUT822

✓

Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN927 OUT822

✓

Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN901 OUT31

×

ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckWordCountRange(answer, 231, 239)

Failed

IN916 OUT286

✓

TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN928 OUT701

×

TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckTitleCase(answer)

Failed

IN914 OUT102

×

TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckTitleCase(answer)

Failed

IN954 OUT752

✓

TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN900 OUT61

Instruction Following - Vowels & Word Repeats 3 passed12 failed

20.0%

IN14 068 OUT4 478

×

LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)

Failed

IN972 OUT2 519

×

LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)

Failed

IN909 OUT466

✓

LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN904 OUT44

✓

LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 020 OUT39

✓

LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN902 OUT34

×

ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench