Ai4Users.comLocal AI & LLM community
Sign in to contributeSign in

Qwen 3.6 27B FP8 Thinking | Benchmark #6

Described in:Qwen 3.8 27B vs Qwen 3.6 27B: benchmark results analysisQwen 3.6 35B vs Qwen 3.6 27B: benchmark results analysis

Benchmark summary

Instruction Following Benchmark based on IFBench

ModelQwen 3.6 27B QuantizationFP8 ModeThinking

IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.

Score 170/271 62.7% passed
170 passed80 failed21 broken29 skipped300 total 271 AI queries 0 tool calls TOKENS IN225 823 TOKENS OUT1 225 926

Results by difficulty

4 groups
Level Easy
15 / 16 93.8%
passed
16 AI queries 0 tool calls IN15 185 OUT27 004
Level Medium
77 / 99 77.8%
passed
99 AI queries 0 tool calls IN87 037 OUT327 740
Level Hard
68 / 102 66.7%
passed
102 AI queries 0 tool calls IN89 661 OUT455 398
Level VeryHard
10 / 54 18.5%
passed
54 AI queries 0 tool calls IN33 940 OUT415 784

Results by category

1 group
Category InstructionFollowing
62.7% 170/271 passed
271 AI queries 0 tool calls TOKENS IN225 823 TOKENS OUT1 225 926

Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.

Benchmark items
Script / test Status Tokens
Instruction Following - Alphabet & Consonants 1 passed13 failed5 broken
5.3%
 IN11 493 OUT178 182
×
AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
The script exceeded the estimated memory limit of 67108864 bytes (allocated about 67111028 bytes).
 Failed IN952 OUT36 051
×
AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
!
AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN926 OUT9 328
×
AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN942 OUT9 674
×
AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN997 OUT6 518
×
ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN979 OUT8 564
×
ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Failed IN0 OUT0
×
ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN946 OUT15 165
×
ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN993 OUT15 259
×
ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Failed IN0 OUT0
×
ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
!
ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN955 OUT9 109
×
ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Failed IN0 OUT0
!
ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN917 OUT17 203
!
ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN1 006 OUT16 643
✓
ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN918 OUT18 413
!
ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN962 OUT16 255
Instruction Following - Brackets & Quotes 16 passed3 failed1 skipped
84.2%
 IN17 049 OUT49 139
✓
Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN951 OUT2 714
✓
Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN926 OUT2 475
✓
Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN909 OUT3 950
✓
Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 000 OUT2 455
✓
Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN898 OUT2 017
✓
Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 006 OUT2 132
✓
Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN944 OUT2 255
✓
Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN932 OUT3 869
✓
Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN907 OUT3 128
×
Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN915 OUT2 113
×
Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw InvalidOperationException: An error occurred while sending the request.
 Failed IN0 OUT0
✓
Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN922 OUT2 321
✓
Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 004 OUT2 897
×
Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN983 OUT3 334
–
QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
✓
QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN952 OUT3 298
✓
QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN942 OUT4 199
✓
QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN952 OUT2 332
✓
QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN912 OUT1 443
✓
QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN994 OUT2 207
Instruction Following - Bullet Points 8 passed9 failed
47.1%
 IN7 586 OUT20 291
×
NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT2 071
×
NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN919 OUT1 861
×
SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT2 270
✓
SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN959 OUT3 125
✓
SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN984 OUT1 485
✓
SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN922 OUT1 477
✓
SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN950 OUT3 726
×
SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN923 OUT4 276
Instruction Following - Custom Transformations 6 passed5 failed
54.5%
 IN9 040 OUT27 098
×
CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckCharacterReverse(answer)
 Failed IN902 OUT691
✓
CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN907 OUT676
×
CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckCsvQuotes(answer)
 Failed IN918 OUT430
✓
CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN925 OUT3 269
✓
DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN901 OUT5 724
×
EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckEuropeanCapitals(answer)
 Failed IN914 OUT6 071
×
Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN888 OUT254
×
ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckReverseNewline(answer)
 Failed IN887 OUT3 095
✓
SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN896 OUT3 251
✓
WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT3 637
Instruction Following - Indentation & Emoji 7 passed2 failed8 skipped
77.8%
 IN8 313 OUT27 989
–
Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
✓
LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT2 030
✓
LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN891 OUT1 260
✓
LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN900 OUT1 787
✓
LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN942 OUT2 225
✓
LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN918 OUT2 010
✓
LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN967 OUT2 444
✓
LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN940 OUT3 377
×
LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)
 Failed IN910 OUT3 576
×
LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)
 Failed IN946 OUT9 280
Instruction Following - Keyword & Conjunction Counts 18 passed
100.0%
 IN17 214 OUT52 797
✓
Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN986 OUT4 091
✓
Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN982 OUT3 566
✓
Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN945 OUT2 837
✓
Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 008 OUT2 028
✓
Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT1 527
✓
KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 002 OUT3 566
✓
KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN924 OUT2 703
✓
KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 007 OUT4 080
✓
KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 031 OUT4 471
✓
KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN951 OUT3 752
✓
Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN893 OUT1 303
✓
Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN980 OUT1 311
✓
Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN951 OUT4 750
✓
Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT1 439
✓
Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN914 OUT2 062
✓
NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN901 OUT2 617
✓
NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN898 OUT3 069
✓
NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN912 OUT3 625
Instruction Following - Keyword In Nth Sentence 7 passed2 broken
77.8%
 IN8 358 OUT46 729
!
KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN1 018 OUT8 059
!
KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN914 OUT3 470
✓
KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT1 749
✓
KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN893 OUT2 299
✓
KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN936 OUT3 449
✓
KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN960 OUT12 047
✓
KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN912 OUT3 730
✓
KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT2 799
✓
KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN908 OUT9 127
Instruction Following - Lists, Lines & Options 14 passed4 failed1 broken
73.7%
 IN17 862 OUT101 391
×
List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "SEPARATOR")
 Failed IN962 OUT2 236
✓
List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN902 OUT1 066
×
List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")
 Failed IN916 OUT2 240
✓
List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN936 OUT1 901
✓
List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT1 147
×
ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN1 010 OUT7 899
✓
ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN942 OUT2 185
✓
ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN914 OUT5 391
✓
NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT3 113
!
NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN956 OUT62 144
✓
NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN927 OUT4 108
✓
NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN937 OUT2 418
✓
NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN986 OUT997
✓
Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN908 OUT1 176
×
Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Assertion expression evaluated to false: CheckOptions(answer, "I know or I don't know")
 Failed IN932 OUT970
✓
Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN1 017 OUT1 775
✓
Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN922 OUT111
✓
Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN960 OUT139
✓
OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT375
Instruction Following - Names, Pronouns & Punctuation 12 passed3 failed1 broken
75.0%
 IN15 676 OUT76 905
✓
PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 049 OUT2 110
✓
PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 052 OUT2 548
✓
PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN1 048 OUT1 612
✓
PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 015 OUT1 560
✓
PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 065 OUT3 525
✓
Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN892 OUT5 380
✓
Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN907 OUT3 986
✓
Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN971 OUT2 285
✓
Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN973 OUT1 781
×
Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckPronouns(answer, 13)
 Failed IN901 OUT7 829
×
Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPunctuationCover(answer)
 Failed IN1 002 OUT13 044
✓
Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 008 OUT3 594
!
Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN961 OUT10 936
✓
Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN949 OUT4 288
×
Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPunctuationCover(answer)
 Failed IN923 OUT6 160
✓
Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN960 OUT6 267
Instruction Following - Palindromes & Letter Patterns 9 passed7 failed1 broken5 skipped
52.9%
 IN11 432 OUT58 853
✓
NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN931 OUT2 159
✓
NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN904 OUT2 891
✓
NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN903 OUT8 226
×
NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN994 OUT5 111
×
NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN938 OUT6 529
–
OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
✓
Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN927 OUT2 523
✓
Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN949 OUT3 584
✓
Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN921 OUT3 018
✓
Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT3 064
×
Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 041 OUT3 795
✓
Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 117 OUT4 499
!
PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN889 OUT13 454
×
PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
Instruction Following - Prompt Repetition 4 passed5 failed
44.4%
 IN8 515 OUT16 019
✓
RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN930 OUT409
✓
RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT1 299
×
RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatChange(answer, "The random, uniform generation of derangements is very simple. Since Dn ≈ n!/e, the obvious procedure of generating a random permutation and check if it is a derangement or not, and generating a new permutation in the negative case, is straight-forward and guarantees a linear time complexity, at least on the average; actually, as we will see, the average complexity is µ1 ≈ e(n − 1).")
 Failed IN1 001 OUT2 422
✓
RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 025 OUT2 987
×
RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSimple(answer)
 Failed IN892 OUT819
×
RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)
 Failed IN919 OUT1 176
×
RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)
 Failed IN930 OUT3 591
×
RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "The walls are solid but the stones are cracked and cold. The windows reflect the dead landscape before you with small plants growing on the walls. \n\nAdd a sentence to make this description of a house \"appealing\". It can be humurous or sarcastic. ", 0, 7)
 Failed IN955 OUT1 818
✓
RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN917 OUT1 498
Instruction Following - Ratios & Stop Words 16 passed1 failed3 broken
80.0%
 IN18 190 OUT64 076
!
RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN922 OUT9 504
✓
RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN943 OUT1 894
✓
RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN905 OUT1 946
✓
RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN948 OUT1 786
✓
RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 029 OUT2 028
×
RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
!
RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN958 OUT3 365
✓
RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN999 OUT5 413
✓
RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN982 OUT2 412
✓
RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN977 OUT2 180
✓
RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN909 OUT556
✓
RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN950 OUT5 485
✓
RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 030 OUT2 389
✓
RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN949 OUT3 053
✓
RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN896 OUT2 972
✓
RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN926 OUT3 767
✓
RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN935 OUT3 852
✓
RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT2 267
!
RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN1 002 OUT5 691
✓
RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN947 OUT3 516
Instruction Following - Sentence Progression 4 passed7 failed2 broken
30.8%
 IN12 229 OUT134 423
✓
AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 002 OUT5 332
✓
AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN920 OUT2 960
×
AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN950 OUT7 461
×
AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN938 OUT6 914
×
AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN930 OUT11 297
×
SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)
 Failed IN943 OUT6 131
!
SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN959 OUT62 144
!
SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN899 OUT4 604
✓
SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN934 OUT3 631
✓
SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN903 OUT4 048
×
SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)
 Failed IN938 OUT8 002
×
SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)
 Failed IN921 OUT6 222
×
SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)
 Failed IN992 OUT5 677
Instruction Following - Sentence Words & Overlap 1 passed4 failed9 skipped
20.0%
 IN2 824 OUT24 569
–
RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
×
RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentenceWords(answer)
 Failed IN933 OUT4 107
✓
RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN934 OUT13 043
×
RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentenceWords(answer)
 Failed IN957 OUT7 419
Instruction Following - Templates & Whitespace 11 passed6 failed1 broken
61.1%
 IN16 513 OUT52 248
×
NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN888 OUT528
×
NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN889 OUT1 454
×
NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN886 OUT6 283
×
NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN925 OUT2 906
✓
OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT907
✓
OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN911 OUT1 816
✓
OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN984 OUT2 743
✓
OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN921 OUT1 652
✓
Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN929 OUT2 111
✓
Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN938 OUT2 213
✓
Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN913 OUT2 629
✓
Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN925 OUT3 622
✓
Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT2 183
×
ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckWordCountRange(answer, 231, 239)
 Failed IN914 OUT5 640
✓
TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN926 OUT4 984
×
TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN912 OUT3 113
!
TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN952 OUT5 141
✓
TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN898 OUT2 323
Instruction Following - Vowels & Word Repeats 5 passed8 failed2 broken
33.3%
 IN10 334 OUT165 540
!
LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN970 OUT9 939
✓
LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN907 OUT24 717
✓
LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT2 781
✓
LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 018 OUT2 510
✓
LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN900 OUT2 645
×
ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN900 OUT3 852
×
ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
×
ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN895 OUT3 074
×
ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
!
ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN1 007 OUT13 860
×
ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN891 OUT16 058
×
VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN946 OUT58 353
✓
VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN998 OUT27 751
×
VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
Instruction Following - Word Counts 16 passed3 failed
84.2%
 IN16 107 OUT57 368
✓
Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN919 OUT4 773
×
Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN909 OUT4 485
×
Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
External function 'AskAi' threw TimeoutException: The operation has timed out.
 Failed IN0 OUT0
✓
Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN923 OUT4 172
✓
UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 102 OUT5 295
✓
UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN972 OUT1 505
✓
UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN888 OUT3 733
✓
UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN897 OUT4 897
✓
UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 049 OUT2 846
✓
UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN893 OUT3 862
✓
UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN891 OUT2 818
✓
UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN991 OUT2 066
✓
WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 004 OUT3 281
✓
WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN905 OUT1 974
✓
WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN969 OUT2 709
✓
WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN936 OUT3 106
✓
WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT1 808
×
WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")
 Failed IN956 OUT4 038
Instruction Following - Word Position & Order 15 passed3 broken6 skipped
83.3%
 IN17 088 OUT72 309
✓
KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN922 OUT3 973
!
KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN927 OUT3 703
✓
KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN990 OUT4 466
✓
KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN972 OUT4 007
✓
LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN913 OUT3 850
✓
LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN911 OUT5 826
✓
LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT5 553
✓
LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 016 OUT3 745
✓
LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN968 OUT11 525
✓
ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN908 OUT2 137
✓
ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN945 OUT2 046
✓
ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN981 OUT2 981
!
ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN913 OUT4 269
!
ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN999 OUT3 901
✓
ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN922 OUT2 160
–
StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
✓
WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN992 OUT3 832
✓
WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN977 OUT2 323
✓
WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN914 OUT2 012