Ai4Users - Qwen 3.6 27B NVFP4 Thinking

Benchmark items
Instruction Following - Alphabet & Consonants 3 passed5 failed11 broken	15.8%	IN18 102 OUT513 164
!	AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN952 OUT18 160
×	AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer)	Failed	IN990 OUT8 998
✓	AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN926 OUT7 740
✓	AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN947 OUT8 012
✓	AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN942 OUT4 605
×	AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckAlphabetLoop(answer)	Failed	IN997 OUT14 020
×	ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN979 OUT20 909
×	ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN933 OUT40 803
!	ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN946 OUT28 442
!	ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN919 OUT13 588
!	ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN993 OUT10 031
!	ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN918 OUT62 144
!	ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN969 OUT62 144
×	ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckConsonantClusters(answer)	Failed	IN955 OUT18 764
!	ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN933 OUT14 261
!	ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN917 OUT10 709
!	ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN1 006 OUT62 144
!	ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN918 OUT62 144
!	ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN962 OUT45 546
Instruction Following - Brackets & Quotes 12 passed6 failed1 broken1 skipped	63.2%	IN17 971 OUT53 172
✓	Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN951 OUT1 966
✓	Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN926 OUT2 238
✓	Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN909 OUT2 730
✓	Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 000 OUT4 151
✓	Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN898 OUT1 643
×	Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedParentheses(answer)	Failed	IN1 006 OUT1 485
✓	Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN944 OUT1 928
×	Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN932 OUT1 518
✓	Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN907 OUT2 573
×	Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN915 OUT2 231
×	Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN922 OUT3 003
×	Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN922 OUT2 999
×	Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNestedQuotes(answer)	Failed	IN1 004 OUT2 561
✓	Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN983 OUT4 524
–	QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
✓	QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN952 OUT2 574
!	QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN942 OUT8 773
✓	QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN952 OUT2 350
✓	QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN912 OUT897
✓	QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN994 OUT3 028
Instruction Following - Bullet Points 15 passed1 failed1 broken	88.2%	IN16 059 OUT61 020
✓	NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN923 OUT2 325
✓	NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN931 OUT1 609
✓	NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN983 OUT2 995
✓	NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN905 OUT1 450
✓	NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 009 OUT1 525
✓	SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN912 OUT1 931
✓	SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN919 OUT1 827
✓	SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN994 OUT2 407
✓	SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN946 OUT2 366
✓	SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN959 OUT2 150
✓	SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN984 OUT1 454
✓	SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN922 OUT1 534
×	SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSubBullets(answer)	Failed	IN950 OUT9 018
✓	SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN923 OUT2 335
!	SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN930 OUT12 857
✓	SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN946 OUT7 521
✓	SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN923 OUT5 716
Instruction Following - Custom Transformations 8 passed3 failed	72.7%	IN9 954 OUT28 609
×	CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharacterReverse(answer)	Failed	IN902 OUT577
✓	CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN907 OUT390
×	CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCsvQuotes(answer)	Failed	IN918 OUT276
✓	CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN925 OUT3 228
✓	DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN901 OUT5 712
×	EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckEuropeanCapitals(answer)	Failed	IN914 OUT8 864
✓	Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN914 OUT1 967
✓	Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN888 OUT198
✓	ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN887 OUT3 375
✓	SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN896 OUT3 600
✓	WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN902 OUT422
Instruction Following - Indentation & Emoji 8 passed1 failed8 skipped	88.9%	IN8 313 OUT27 611
–	Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
–	EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.	Skipped	IN0 OUT0
✓	LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN899 OUT1 694
✓	LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN891 OUT833
✓	LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN900 OUT1 903
✓	LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN942 OUT1 632
✓	LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN918 OUT1 970
✓	LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN967 OUT3 208
✓	LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN940 OUT2 034
✓	LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN910 OUT4 365
×	LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)	Failed	IN946 OUT9 972
Instruction Following - Keyword & Conjunction Counts 17 passed1 broken	94.4%	IN17 214 OUT55 356
✓	Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN986 OUT4 317
✓	Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN982 OUT3 618
✓	Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN945 OUT2 739
✓	Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 008 OUT2 250
✓	Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN983 OUT2 061
✓	KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 002 OUT4 974
✓	KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN924 OUT1 924
✓	KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 007 OUT5 101
!	KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN1 031 OUT5 183
✓	KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN951 OUT4 462
✓	Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN893 OUT869
✓	Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN980 OUT1 739
✓	Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN951 OUT3 819
✓	Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN946 OUT1 127
✓	Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN914 OUT1 803
✓	NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN901 OUT2 456
✓	NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN898 OUT3 369
✓	NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN912 OUT3 545
Instruction Following - Keyword In Nth Sentence 6 passed2 failed1 broken	66.7%	IN8 358 OUT45 667
!	KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN1 018 OUT4 802
✓	KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN914 OUT3 744
✓	KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN899 OUT1 860
✓	KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN893 OUT3 468
✓	KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN936 OUT4 481
×	KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncludeKeyword(answer, "trichosis", 22)	Failed	IN960 OUT14 959
✓	KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN912 OUT2 870
✓	KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN918 OUT2 509
×	KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckWordCountRange(answer, 246, 254)	Failed	IN908 OUT6 974
Instruction Following - Lists, Lines & Options 19 passed	100.0%	IN17 862 OUT95 958
✓	List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN962 OUT2 023
✓	List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN902 OUT1 160
✓	List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN916 OUT2 816
✓	List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN936 OUT1 789
✓	List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN903 OUT1 590
✓	ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 010 OUT5 906
✓	ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN942 OUT3 472
✓	ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN914 OUT4 479
✓	NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN916 OUT1 975
✓	NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN956 OUT55 262
✓	NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN927 OUT4 022
✓	NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN937 OUT4 196
✓	NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN986 OUT481
✓	Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN908 OUT2 494
✓	Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN932 OUT292
✓	Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN1 017 OUT2 555
✓	Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN922 OUT369
✓	Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN960 OUT92
✓	OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN916 OUT985
Instruction Following - Names, Pronouns & Punctuation 11 passed3 failed2 broken	68.8%	IN15 676 OUT117 256
✓	PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 049 OUT1 617
✓	PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 052 OUT2 959
✓	PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN1 048 OUT1 356
✓	PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 015 OUT1 743
✓	PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 065 OUT2 677
✓	Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN892 OUT3 145
!	Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN907 OUT62 144
✓	Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN971 OUT4 393
✓	Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN973 OUT1 833
✓	Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN901 OUT3 267
×	Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN1 002 OUT5 032
×	Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN1 008 OUT3 786
×	Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckPunctuationCover(answer)	Failed	IN961 OUT5 354
✓	Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN949 OUT4 254
!	Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN923 OUT7 648
✓	Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN960 OUT6 048
Instruction Following - Palindromes & Letter Patterns 10 passed2 failed5 broken5 skipped	58.8%	IN16 098 OUT304 437
✓	NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN931 OUT1 757
✓	NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN904 OUT1 801
✓	NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN903 OUT6 854
×	NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN994 OUT5 613
×	NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)	Failed	IN938 OUT5 674
–	OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
–	OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.	Skipped	IN0 OUT0
✓	Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN927 OUT1 787
✓	Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN949 OUT3 617
✓	Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN921 OUT2 196
✓	Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN918 OUT3 153
✓	Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN937 OUT11 939
✓	Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 041 OUT8 949
✓	Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 117 OUT4 654
!	PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN889 OUT62 144
!	PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN921 OUT23 599
!	PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN940 OUT36 412
!	PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN886 OUT62 144
!	PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN982 OUT62 144
Instruction Following - Prompt Repetition 5 passed4 failed	55.6%	IN8 515 OUT11 464
✓	RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN930 OUT949
✓	RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN946 OUT816
✓	RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 001 OUT1 826
✓	RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 025 OUT772
×	RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSimple(answer)	Failed	IN892 OUT868
×	RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)	Failed	IN919 OUT859
×	RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)	Failed	IN930 OUT2 478
×	RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7)	Failed	IN955 OUT2 336
✓	RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN917 OUT560
Instruction Following - Ratios & Stop Words 16 passed4 failed	80.0%	IN19 171 OUT100 649
×	RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN922 OUT2 191
✓	RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN943 OUT1 115
✓	RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN905 OUT1 747
✓	RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN948 OUT1 566
✓	RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 029 OUT1 980
×	RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentBalance(answer)	Failed	IN981 OUT25 478
✓	RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN958 OUT28 287
✓	RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN999 OUT3 361
×	RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN982 OUT1 874
✓	RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN977 OUT2 429
✓	RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN909 OUT720
×	RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckSentType(answer)	Failed	IN950 OUT3 660
✓	RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 030 OUT2 278
✓	RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN949 OUT1 255
✓	RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN896 OUT3 493
✓	RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN926 OUT3 482
✓	RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN935 OUT2 570
✓	RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN983 OUT3 139
✓	RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 002 OUT5 453
✓	RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN947 OUT4 571
Instruction Following - Sentence Progression 3 passed9 failed1 broken	23.1%	IN12 229 OUT109 627
×	AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN1 002 OUT4 938
×	AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN920 OUT7 771
×	AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN950 OUT10 494
×	AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN938 OUT5 983
×	AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)	Failed	IN930 OUT30 004
✓	SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN943 OUT8 600
!	SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN959 OUT12 677
✓	SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN899 OUT4 983
✓	SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN934 OUT2 388
×	SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 1)	Failed	IN903 OUT4 318
×	SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)	Failed	IN938 OUT5 801
×	SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)	Failed	IN921 OUT5 685
×	SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)	Failed	IN992 OUT5 985
Instruction Following - Sentence Words & Overlap 5 failed9 skipped	0.0%	IN4 736 OUT49 982
–	RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
–	RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.	Skipped	IN0 OUT0
×	RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN920 OUT11 981
×	RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN933 OUT10 833
×	RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN934 OUT11 921
×	RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN992 OUT5 237
×	RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckSentenceWords(answer)	Failed	IN957 OUT10 010
Instruction Following - Templates & Whitespace 12 passed5 failed1 broken	66.7%	IN16 513 OUT51 198
×	NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text)	Failed	IN888 OUT1 096
×	NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text)	Failed	IN889 OUT859
×	NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text)	Failed	IN886 OUT1 022
×	NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckNoWhitespace(response.Text)	Failed	IN925 OUT2 027
✓	OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN903 OUT1 204
✓	OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN911 OUT1 002
✓	OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN984 OUT2 206
✓	OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN921 OUT655
✓	Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN929 OUT2 039
✓	Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN938 OUT1 818
✓	Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN913 OUT2 087
✓	Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN925 OUT3 162
✓	Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN899 OUT2 344
✓	ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN914 OUT6 568
✓	TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN926 OUT5 084
!	TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN912 OUT10 091
×	TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckTitleCase(answer)	Failed	IN952 OUT4 915
✓	TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN898 OUT3 019
Instruction Following - Vowels & Word Repeats 5 passed3 failed7 broken	33.3%	IN14 038 OUT326 910
!	LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN970 OUT15 576
×	LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)	Failed	IN907 OUT8 482
✓	LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN902 OUT2 307
✓	LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 018 OUT2 746
✓	LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN900 OUT2 647
✓	ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN900 OUT7 105
×	ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN898 OUT4 059
!	ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN932 OUT62 144
×	ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)	Failed	IN895 OUT4 115
!	ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN961 OUT13 003
!	ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN1 007 OUT62 144
!	ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN891 OUT62 144
!	VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN946 OUT62 144
✓	VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN998 OUT9 174
!	VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN913 OUT9 120
Instruction Following - Word Counts 16 passed2 failed1 broken	84.2%	IN17 936 OUT135 663
✓	Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN919 OUT3 277
!	Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN930 OUT8 846
×	Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 17)	Failed	IN909 OUT12 957
×	Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckJapanese(answer, 5)	Failed	IN899 OUT31 416
✓	Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN923 OUT7 103
✓	UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 102 OUT4 304
✓	UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN972 OUT2 471
✓	UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN888 OUT26 720
✓	UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN897 OUT4 161
✓	UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN1 049 OUT2 327
✓	UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN893 OUT3 516
✓	UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN891 OUT3 217
✓	UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN991 OUT3 023
✓	WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN1 004 OUT4 259
✓	WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN905 OUT4 371
✓	WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN969 OUT2 558
✓	WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN936 OUT3 249
✓	WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench	Passed	IN903 OUT2 379
✓	WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN956 OUT5 509
Instruction Following - Word Position & Order 11 passed3 failed4 broken6 skipped	61.1%	IN17 088 OUT171 532
!	KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN922 OUT5 635
×	KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "beneath", 23, 34)	Failed	IN927 OUT833
✓	KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN990 OUT3 594
✓	KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench	Passed	IN972 OUT4 608
!	LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN913 OUT6 870
✓	LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench	Passed	IN911 OUT5 463
!	LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN918 OUT62 144
!	LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: response.Completed Values: response.Completed = false	Broken	IN1 016 OUT5 321
×	LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench Assertion expression evaluated to false: CheckLastWordFirstNext(answer)	Failed	IN968 OUT17 040
✓	ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN908 OUT37 904
✓	ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN945 OUT2 722
×	ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench Assertion expression evaluated to false: CheckParagraphLastFirst(answer)	Failed	IN981 OUT4 395
✓	ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN913 OUT3 787
✓	ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN999 OUT3 252
✓	ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN922 OUT2 569
–	StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
–	StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.	Skipped	IN0 OUT0
✓	WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN992 OUT2 610
✓	WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN977 OUT682
✓	WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench	Passed	IN914 OUT2 103

Instruction Following - Alphabet & Consonants 3 passed5 failed11 broken

15.8%

IN18 102 OUT513 164

!

AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN952 OUT18 160

×

AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckAlphabetLoop(answer)

Failed

IN990 OUT8 998

✓

AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN926 OUT7 740

✓

AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN947 OUT8 012

✓

AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN942 OUT4 605

×

AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckAlphabetLoop(answer)

Failed

IN997 OUT14 020

×

ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN979 OUT20 909

×

ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN933 OUT40 803

!

ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN946 OUT28 442

!

ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN919 OUT13 588

!

ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN993 OUT10 031

!

ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN918 OUT62 144

!

ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN969 OUT62 144

×

ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckConsonantClusters(answer)

Failed

IN955 OUT18 764

!

ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN933 OUT14 261

!

ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN917 OUT10 709

!

ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN1 006 OUT62 144

!

ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN918 OUT62 144

!

ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN962 OUT45 546

Instruction Following - Brackets & Quotes 12 passed6 failed1 broken1 skipped

63.2%

IN17 971 OUT53 172

✓

Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN951 OUT1 966

✓

Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN926 OUT2 238

✓

Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN909 OUT2 730

✓

Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 000 OUT4 151

✓

Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN898 OUT1 643

×

Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedParentheses(answer)

Failed

IN1 006 OUT1 485

✓

Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN944 OUT1 928

×

Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN932 OUT1 518

✓

Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN907 OUT2 573

×

Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN915 OUT2 231

×

Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN922 OUT3 003

×

Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN922 OUT2 999

×

Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNestedQuotes(answer)

Failed

IN1 004 OUT2 561

✓

Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN983 OUT4 524

–

QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

✓

QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN952 OUT2 574

!

QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN942 OUT8 773

✓

QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN952 OUT2 350

✓

QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN912 OUT897

✓

QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN994 OUT3 028

Instruction Following - Bullet Points 15 passed1 failed1 broken

88.2%

IN16 059 OUT61 020

✓

NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN923 OUT2 325

✓

NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN931 OUT1 609

✓

NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN983 OUT2 995

✓

NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN905 OUT1 450

✓

NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 009 OUT1 525

✓

SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN912 OUT1 931

✓

SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN919 OUT1 827

✓

SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN994 OUT2 407

✓

SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN946 OUT2 366

✓

SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN959 OUT2 150

✓

SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN984 OUT1 454

✓

SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN922 OUT1 534

×

SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSubBullets(answer)

Failed

IN950 OUT9 018

✓

SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN923 OUT2 335

!

SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN930 OUT12 857

✓

SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN946 OUT7 521

✓

SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN923 OUT5 716

Instruction Following - Custom Transformations 8 passed3 failed

72.7%

IN9 954 OUT28 609

×

CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckCharacterReverse(answer)

Failed

IN902 OUT577

✓

CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN907 OUT390

×

CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckCsvQuotes(answer)

Failed

IN918 OUT276

✓

CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN925 OUT3 228

✓

DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN901 OUT5 712

×

EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckEuropeanCapitals(answer)

Failed

IN914 OUT8 864

✓

Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN914 OUT1 967

✓

Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN888 OUT198

✓

ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN887 OUT3 375

✓

SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN896 OUT3 600

✓

WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN902 OUT422

Instruction Following - Indentation & Emoji 8 passed1 failed8 skipped

88.9%

IN8 313 OUT27 611

–

Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

–

EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.

Skipped

IN0 OUT0

✓

LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN899 OUT1 694

✓

LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN891 OUT833

✓

LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN900 OUT1 903

✓

LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN942 OUT1 632

✓

LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN918 OUT1 970

✓

LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN967 OUT3 208

✓

LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN940 OUT2 034

✓

LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN910 OUT4 365

×

LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)

Failed

IN946 OUT9 972

Instruction Following - Keyword & Conjunction Counts 17 passed1 broken

94.4%

IN17 214 OUT55 356

✓

Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN986 OUT4 317

✓

Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN982 OUT3 618

✓

Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN945 OUT2 739

✓

Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 008 OUT2 250

✓

Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN983 OUT2 061

✓

KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 002 OUT4 974

✓

KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN924 OUT1 924

✓

KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 007 OUT5 101

!

KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN1 031 OUT5 183

✓

KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN951 OUT4 462

✓

Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN893 OUT869

✓

Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN980 OUT1 739

✓

Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN951 OUT3 819

✓

Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN946 OUT1 127

✓

Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN914 OUT1 803

✓

NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN901 OUT2 456

✓

NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN898 OUT3 369

✓

NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN912 OUT3 545

Instruction Following - Keyword In Nth Sentence 6 passed2 failed1 broken

66.7%

IN8 358 OUT45 667

!

KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN1 018 OUT4 802

✓

KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN914 OUT3 744

✓

KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN899 OUT1 860

✓

KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN893 OUT3 468

✓

KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN936 OUT4 481

×

KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncludeKeyword(answer, "trichosis", 22)

Failed

IN960 OUT14 959

✓

KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN912 OUT2 870

✓

KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN918 OUT2 509

×

KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckWordCountRange(answer, 246, 254)

Failed

IN908 OUT6 974

Instruction Following - Lists, Lines & Options 19 passed

100.0%

IN17 862 OUT95 958

✓

List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN962 OUT2 023

✓

List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN902 OUT1 160

✓

List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN916 OUT2 816

✓

List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN936 OUT1 789

✓

List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN903 OUT1 590

✓

ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 010 OUT5 906

✓

ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN942 OUT3 472

✓

ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN914 OUT4 479

✓

NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN916 OUT1 975

✓

NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN956 OUT55 262

✓

NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN927 OUT4 022

✓

NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN937 OUT4 196

✓

NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN986 OUT481

✓

Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN908 OUT2 494

✓

Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN932 OUT292

✓

Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN1 017 OUT2 555

✓

Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN922 OUT369

✓

Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN960 OUT92

✓

OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN916 OUT985

Instruction Following - Names, Pronouns & Punctuation 11 passed3 failed2 broken

68.8%

IN15 676 OUT117 256

✓

PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 049 OUT1 617

✓

PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 052 OUT2 959

✓

PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN1 048 OUT1 356

✓

PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 015 OUT1 743

✓

PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 065 OUT2 677

✓

Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN892 OUT3 145

!

Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN907 OUT62 144

✓

Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN971 OUT4 393

✓

Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN973 OUT1 833

✓

Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN901 OUT3 267

×

Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN1 002 OUT5 032

×

Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN1 008 OUT3 786

×

Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckPunctuationCover(answer)

Failed

IN961 OUT5 354

✓

Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN949 OUT4 254

!

Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN923 OUT7 648

✓

Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN960 OUT6 048

Instruction Following - Palindromes & Letter Patterns 10 passed2 failed5 broken5 skipped

58.8%

IN16 098 OUT304 437

✓

NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN931 OUT1 757

✓

NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN904 OUT1 801

✓

NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN903 OUT6 854

×

NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN994 OUT5 613

×

NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)

Failed

IN938 OUT5 674

–

OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

–

OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.

Skipped

IN0 OUT0

✓

Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN927 OUT1 787

✓

Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN949 OUT3 617

✓

Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN921 OUT2 196

✓

Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN918 OUT3 153

✓

Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN937 OUT11 939

✓

Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 041 OUT8 949

✓

Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 117 OUT4 654

!

PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN889 OUT62 144

!

PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN921 OUT23 599

!

PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN940 OUT36 412

!

PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN886 OUT62 144

!

PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN982 OUT62 144

Instruction Following - Prompt Repetition 5 passed4 failed

55.6%

IN8 515 OUT11 464

✓

RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN930 OUT949

✓

RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN946 OUT816

✓

RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 001 OUT1 826

✓

RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 025 OUT772

×

RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSimple(answer)

Failed

IN892 OUT868

×

RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)

Failed

IN919 OUT859

×

RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)

Failed

IN930 OUT2 478

×

RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7)

Failed

IN955 OUT2 336

✓

RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN917 OUT560

Instruction Following - Ratios & Stop Words 16 passed4 failed

80.0%

IN19 171 OUT100 649

×

RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN922 OUT2 191

✓

RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN943 OUT1 115

✓

RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN905 OUT1 747

✓

RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN948 OUT1 566

✓

RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 029 OUT1 980

×

RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentBalance(answer)

Failed

IN981 OUT25 478

✓

RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN958 OUT28 287

✓

RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN999 OUT3 361

×

RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN982 OUT1 874

✓

RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN977 OUT2 429

✓

RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN909 OUT720

×

RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckSentType(answer)

Failed

IN950 OUT3 660

✓

RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 030 OUT2 278

✓

RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN949 OUT1 255

✓

RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN896 OUT3 493

✓

RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN926 OUT3 482

✓

RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN935 OUT2 570

✓

RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN983 OUT3 139

✓

RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 002 OUT5 453

✓

RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN947 OUT4 571

Instruction Following - Sentence Progression 3 passed9 failed1 broken

23.1%

IN12 229 OUT109 627

×

AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN1 002 OUT4 938

×

AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN920 OUT7 771

×

AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN950 OUT10 494

×

AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN938 OUT5 983

×

AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)

Failed

IN930 OUT30 004

✓

SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN943 OUT8 600

!

SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN959 OUT12 677

✓

SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN899 OUT4 983

✓

SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN934 OUT2 388

×

SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 1)

Failed

IN903 OUT4 318

×

SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)

Failed

IN938 OUT5 801

×

SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)

Failed

IN921 OUT5 685

×

SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)

Failed

IN992 OUT5 985

Instruction Following - Sentence Words & Overlap 5 failed9 skipped

0.0%

IN4 736 OUT49 982

–

RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

–

RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.

Skipped

IN0 OUT0

×

RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN920 OUT11 981

×

RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN933 OUT10 833

×

RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN934 OUT11 921

×

RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN992 OUT5 237

×

RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckSentenceWords(answer)

Failed

IN957 OUT10 010

Instruction Following - Templates & Whitespace 12 passed5 failed1 broken

66.7%

IN16 513 OUT51 198

×

NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoWhitespace(response.Text)

Failed

IN888 OUT1 096

×

NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoWhitespace(response.Text)

Failed

IN889 OUT859

×

NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoWhitespace(response.Text)

Failed

IN886 OUT1 022

×

NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckNoWhitespace(response.Text)

Failed

IN925 OUT2 027

✓

OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN903 OUT1 204

✓

OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN911 OUT1 002

✓

OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN984 OUT2 206

✓

OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN921 OUT655

✓

Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN929 OUT2 039

✓

Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN938 OUT1 818

✓

Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN913 OUT2 087

✓

Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN925 OUT3 162

✓

Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN899 OUT2 344

✓

ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN914 OUT6 568

✓

TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN926 OUT5 084

!

TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN912 OUT10 091

×

TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckTitleCase(answer)

Failed

IN952 OUT4 915

✓

TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN898 OUT3 019

Instruction Following - Vowels & Word Repeats 5 passed3 failed7 broken

33.3%

IN14 038 OUT326 910

!

LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN970 OUT15 576

×

LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)

Failed

IN907 OUT8 482

✓

LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN902 OUT2 307

✓

LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 018 OUT2 746

✓

LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN900 OUT2 647

✓

ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN900 OUT7 105

×

ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)

Failed

IN898 OUT4 059

!

ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN932 OUT62 144

×

ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)

Failed

IN895 OUT4 115

!

ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN961 OUT13 003

!

ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN1 007 OUT62 144

!

ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN891 OUT62 144

!

VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN946 OUT62 144

✓

VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN998 OUT9 174

!

VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN913 OUT9 120

Instruction Following - Word Counts 16 passed2 failed1 broken

84.2%

IN17 936 OUT135 663

✓

Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN919 OUT3 277

!

Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN930 OUT8 846

×

Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckJapanese(answer, 17)

Failed

IN909 OUT12 957

×

Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckJapanese(answer, 5)

Failed

IN899 OUT31 416

✓

Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN923 OUT7 103

✓

UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 102 OUT4 304

✓

UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN972 OUT2 471

✓

UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN888 OUT26 720

✓

UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN897 OUT4 161

✓

UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN1 049 OUT2 327

✓

UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN893 OUT3 516

✓

UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN891 OUT3 217

✓

UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN991 OUT3 023

✓

WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN1 004 OUT4 259

✓

WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN905 OUT4 371

✓

WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN969 OUT2 558

✓

WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN936 OUT3 249

✓

WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Passed

IN903 OUT2 379

✓

WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN956 OUT5 509

Instruction Following - Word Position & Order 11 passed3 failed4 broken6 skipped

61.1%

IN17 088 OUT171 532

!

KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN922 OUT5 635

×

KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "beneath", 23, 34)

Failed

IN927 OUT833

✓

KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN990 OUT3 594

✓

KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench

Passed

IN972 OUT4 608

!

LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN913 OUT6 870

✓

LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Passed

IN911 OUT5 463

!

LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN918 OUT62 144

!

LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: response.Completed Values: response.Completed = false

Broken

IN1 016 OUT5 321

×

LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench

Assertion expression evaluated to false: CheckLastWordFirstNext(answer)

Failed

IN968 OUT17 040

✓

ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN908 OUT37 904

✓

ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN945 OUT2 722

×

ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Assertion expression evaluated to false: CheckParagraphLastFirst(answer)

Failed

IN981 OUT4 395

✓

ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN913 OUT3 787

✓

ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN999 OUT3 252

✓

ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN922 OUT2 569

–

StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.

Skipped

IN0 OUT0

–

StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.

Skipped

IN0 OUT0

–

StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.

Skipped

IN0 OUT0

–

StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.

Skipped

IN0 OUT0

–

StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.

Skipped

IN0 OUT0

–

StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench

Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.

Skipped

IN0 OUT0

✓

WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN992 OUT2 610

✓

WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN977 OUT682

✓

WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench

Passed

IN914 OUT2 103

Qwen 3.6 27B NVFP4 Thinking | Benchmark #56

Instruction Following Benchmark based on IFBench

Results by difficulty

Results by category