Ai4Users.comLocal AI & LLM community
Sign in to contributeSign in

Qwen 3.8 27B FP8 Think: medium | Benchmark #34

Described in:Qwen 3.8 27B vs Qwen 3.6 27B: benchmark results analysis

Benchmark summary

Instruction Following Benchmark based on IFBench

ModelQwen 3.8 27B QuantizationFP8 ModeThink: medium

IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.

Score 185/271 68.3% passed
185 passed85 failed1 broken29 skipped300 total 271 AI queries 0 tool calls TOKENS IN255 833 TOKENS OUT1 438 869

Results by difficulty

4 groups
Level Easy
15 / 16 93.8%
passed
16 AI queries 0 tool calls IN15 185 OUT13 614
Level Medium
80 / 99 80.8%
passed
99 AI queries 0 tool calls IN93 641 OUT191 736
Level Hard
71 / 102 69.6%
passed
102 AI queries 0 tool calls IN96 132 OUT396 808
Level VeryHard
19 / 54 35.2%
passed
54 AI queries 0 tool calls IN50 875 OUT836 711

Results by category

1 group
Category InstructionFollowing
68.3% 185/271 passed
271 AI queries 0 tool calls TOKENS IN255 833 TOKENS OUT1 438 869

Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.

Benchmark items
Script / test Status Tokens
Instruction Following - Alphabet & Consonants 3 passed16 failed
15.8%
 IN18 102 OUT412 186
×
AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN952 OUT12 117
✓
AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN990 OUT5 737
×
AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN926 OUT6 982
×
AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN947 OUT5 579
✓
AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN942 OUT3 917
×
AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN997 OUT7 801
×
ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN979 OUT38 999
×
ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN933 OUT20 838
✓
ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN946 OUT26 800
×
ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN919 OUT19 720
×
ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN993 OUT27 613
×
ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN918 OUT25 300
×
ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN969 OUT29 852
×
ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN955 OUT21 377
×
ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN933 OUT39 107
×
ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN917 OUT27 612
×
ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN1 006 OUT38 974
×
ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN918 OUT18 077
×
ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN962 OUT35 784
Instruction Following - Brackets & Quotes 11 passed8 failed1 skipped
57.9%
 IN17 971 OUT54 268
✓
Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN951 OUT640
×
Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedParentheses(answer)
 Failed IN926 OUT692
✓
Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN909 OUT1 161
✓
Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 000 OUT852
✓
Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN898 OUT1 131
✓
Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 006 OUT506
✓
Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN944 OUT1 016
×
Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN932 OUT2 482
×
Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN907 OUT3 423
×
Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN915 OUT1 040
×
Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN922 OUT897
×
Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN922 OUT965
×
Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN1 004 OUT1 460
×
Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN983 OUT31 698
–
QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
✓
QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN952 OUT1 957
✓
QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN942 OUT2 339
✓
QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN952 OUT745
✓
QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN912 OUT681
✓
QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN994 OUT583
Instruction Following - Bullet Points 13 passed4 failed
76.5%
 IN16 059 OUT50 072
×
NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSomeBullets(answer)
 Failed IN923 OUT1 131
✓
NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN931 OUT833
✓
NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT613
✓
NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN905 OUT432
✓
NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 009 OUT663
×
SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN912 OUT737
×
SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN919 OUT535
×
SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN994 OUT836
✓
SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT555
✓
SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN959 OUT417
✓
SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN984 OUT201
✓
SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN922 OUT432
✓
SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN950 OUT5 894
✓
SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN923 OUT19 488
✓
SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN930 OUT3 773
✓
SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN946 OUT9 165
✓
SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN923 OUT4 367
Instruction Following - Custom Transformations 9 passed2 failed
81.8%
 IN9 954 OUT19 700
✓
CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT4 158
✓
CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN907 OUT176
✓
CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT312
✓
CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN925 OUT725
✓
DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN901 OUT6 054
×
EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckEuropeanCapitals(answer)
 Failed IN914 OUT2 193
×
Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckMcq(answer)
 Failed IN914 OUT803
✓
Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN888 OUT126
✓
ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN887 OUT1 422
✓
SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN896 OUT3 263
✓
WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT468
Instruction Following - Indentation & Emoji 7 passed2 failed8 skipped
77.8%
 IN8 313 OUT15 434
–
Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
✓
LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT523
✓
LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN891 OUT239
✓
LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN900 OUT1 320
✓
LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN942 OUT351
✓
LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN918 OUT3 108
✓
LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN967 OUT853
×
LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckIndentStairs(answer)
 Failed IN940 OUT1 355
✓
LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN910 OUT3 264
×
LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIndentStairs(answer)
 Failed IN946 OUT4 421
Instruction Following - Keyword & Conjunction Counts 18 passed
100.0%
 IN17 214 OUT37 150
✓
Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN986 OUT1 805
✓
Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN982 OUT1 071
✓
Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN945 OUT1 115
✓
Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 008 OUT1 992
✓
Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT951
✓
KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 002 OUT2 792
✓
KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN924 OUT1 920
✓
KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 007 OUT2 605
✓
KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 031 OUT5 836
✓
KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN951 OUT2 964
✓
Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN893 OUT517
✓
Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN980 OUT475
✓
Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN951 OUT3 078
✓
Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT813
✓
Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN914 OUT1 748
✓
NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN901 OUT2 387
✓
NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN898 OUT1 784
✓
NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN912 OUT3 297
Instruction Following - Keyword In Nth Sentence 9 passed
100.0%
 IN8 358 OUT75 160
✓
KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 018 OUT28 312
✓
KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN914 OUT2 863
✓
KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT804
✓
KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN893 OUT1 343
✓
KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN936 OUT3 261
✓
KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN960 OUT12 207
✓
KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN912 OUT2 533
✓
KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT2 019
✓
KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN908 OUT21 818
Instruction Following - Lists, Lines & Options 14 passed5 failed
73.7%
 IN17 862 OUT26 943
✓
List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN962 OUT909
×
List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")
 Failed IN902 OUT335
✓
List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT1 228
✓
List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN936 OUT580
✓
List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT492
×
ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN1 010 OUT9 157
×
ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")
 Failed IN942 OUT1 010
✓
ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN914 OUT3 023
✓
NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT3 736
✓
NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN956 OUT2 031
×
NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckNewLineWords(answer)
 Failed IN927 OUT186
✓
NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN937 OUT3 130
✓
NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN986 OUT153
✓
Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN908 OUT159
✓
Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN932 OUT125
×
Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Assertion expression evaluated to false: CheckOptions(answer, "a), b), c), d)")
 Failed IN1 017 OUT310
✓
Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN922 OUT103
✓
Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN960 OUT56
✓
OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT220
Instruction Following - Names, Pronouns & Punctuation 15 passed1 failed
93.8%
 IN15 676 OUT50 184
✓
PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 049 OUT513
✓
PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 052 OUT1 700
✓
PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN1 048 OUT911
✓
PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 015 OUT720
✓
PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 065 OUT2 034
✓
Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN892 OUT1 078
✓
Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN907 OUT4 748
✓
Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN971 OUT1 564
✓
Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN973 OUT645
✓
Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN901 OUT3 503
✓
Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 002 OUT15 689
✓
Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 008 OUT4 470
✓
Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN961 OUT3 588
✓
Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN949 OUT3 475
×
Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPunctuationCover(answer)
 Failed IN923 OUT868
✓
Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN960 OUT4 678
Instruction Following - Palindromes & Letter Patterns 10 passed6 failed1 broken5 skipped
58.8%
 IN16 098 OUT182 751
✓
NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN931 OUT850
✓
NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN904 OUT574
✓
NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN903 OUT40 323
✓
NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN994 OUT7 240
×
NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN938 OUT12 775
–
OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
×
Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPalindromes(answer)
 Failed IN927 OUT3 044
✓
Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN949 OUT1 805
✓
Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN921 OUT2 961
✓
Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT3 838
✓
Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN937 OUT1 828
×
Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPalindromes(answer)
 Failed IN1 041 OUT3 218
×
Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPalindromes(answer)
 Failed IN1 117 OUT3 698
×
PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN889 OUT13 323
×
PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN921 OUT17 283
!
PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN940 OUT62 144
✓
PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN886 OUT6 993
✓
PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN982 OUT854
Instruction Following - Prompt Repetition 5 passed4 failed
55.6%
 IN8 515 OUT5 418
✓
RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN930 OUT187
✓
RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT251
✓
RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 001 OUT586
×
RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSimple(answer)
 Failed IN1 025 OUT1 633
×
RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSimple(answer)
 Failed IN892 OUT264
×
RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)
 Failed IN919 OUT434
×
RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)
 Failed IN930 OUT1 502
✓
RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN955 OUT192
✓
RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN917 OUT369
Instruction Following - Ratios & Stop Words 15 passed5 failed
75.0%
 IN19 171 OUT84 092
×
RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN922 OUT6 381
✓
RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN943 OUT495
✓
RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN905 OUT848
✓
RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN948 OUT586
✓
RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 029 OUT1 168
×
RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN981 OUT21 883
×
RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN958 OUT19 372
✓
RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN999 OUT2 546
×
RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentType(answer)
 Failed IN982 OUT1 101
✓
RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN977 OUT2 569
✓
RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN909 OUT481
✓
RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN950 OUT3 470
×
RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentType(answer)
 Failed IN1 030 OUT2 564
✓
RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN949 OUT3 001
✓
RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN896 OUT849
✓
RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN926 OUT2 207
✓
RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN935 OUT2 435
✓
RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT5 702
✓
RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 002 OUT2 463
✓
RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN947 OUT3 971
Instruction Following - Sentence Progression 7 passed6 failed
53.8%
 IN12 229 OUT48 375
✓
AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 002 OUT1 216
✓
AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN920 OUT1 544
×
AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN950 OUT3 946
×
AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN938 OUT7 950
×
AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN930 OUT4 416
✓
SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN943 OUT8 922
✓
SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN959 OUT3 374
✓
SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN899 OUT2 768
✓
SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN934 OUT2 720
✓
SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN903 OUT2 079
×
SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)
 Failed IN938 OUT3 624
×
SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)
 Failed IN921 OUT3 052
×
SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)
 Failed IN992 OUT2 764
Instruction Following - Sentence Words & Overlap 4 passed1 failed9 skipped
80.0%
 IN4 736 OUT23 361
–
RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
×
RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentenceWords(answer)
 Failed IN920 OUT2 613
✓
RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN933 OUT3 791
✓
RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN934 OUT7 469
✓
RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN992 OUT5 781
✓
RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN957 OUT3 707
Instruction Following - Templates & Whitespace 10 passed8 failed
55.6%
 IN16 513 OUT43 066
×
NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN888 OUT189
×
NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN889 OUT452
×
NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN886 OUT283
×
NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN925 OUT2 632
✓
OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT427
✓
OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN911 OUT787
✓
OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN984 OUT1 583
✓
OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN921 OUT798
✓
Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN929 OUT1 050
✓
Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN938 OUT1 222
✓
Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN913 OUT1 474
✓
Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN925 OUT2 532
✓
Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT1 004
×
ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckWordCountRange(answer, 231, 239)
 Failed IN914 OUT5 179
×
TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN926 OUT10 604
×
TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN912 OUT6 407
×
TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN952 OUT4 865
✓
TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN898 OUT1 578
Instruction Following - Vowels & Word Repeats 6 passed9 failed
40.0%
 IN14 038 OUT201 082
✓
LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN970 OUT7 384
✓
LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN907 OUT2 856
✓
LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT1 040
✓
LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 018 OUT864
✓
LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN900 OUT1 300
×
ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN900 OUT5 820
×
ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN898 OUT11 711
✓
ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN932 OUT22 984
×
ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN895 OUT10 933
×
ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN961 OUT23 948
×
ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN1 007 OUT25 557
×
ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN891 OUT14 583
×
VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN946 OUT22 840
×
VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN998 OUT24 637
×
VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN913 OUT24 625
Instruction Following - Word Counts 16 passed3 failed
84.2%
 IN17 936 OUT63 534
✓
Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN919 OUT587
×
Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckJapanese(answer, 29)
 Failed IN930 OUT19 358
✓
Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN909 OUT8 198
×
Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckJapanese(answer, 5)
 Failed IN899 OUT8 767
✓
Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN923 OUT6 448
✓
UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 102 OUT2 249
✓
UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN972 OUT1 396
✓
UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN888 OUT1 094
✓
UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN897 OUT823
✓
UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 049 OUT1 571
✓
UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN893 OUT1 675
✓
UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN891 OUT780
✓
UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN991 OUT1 191
✓
WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 004 OUT1 724
✓
WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN905 OUT1 825
×
WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckWordCountRange(answer, 24, 28)
 Failed IN969 OUT1 258
✓
WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN936 OUT1 384
✓
WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT612
✓
WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN956 OUT2 594
Instruction Following - Word Position & Order 13 passed5 failed6 skipped
72.2%
 IN17 088 OUT46 093
✓
KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN922 OUT6 885
✓
KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN927 OUT5 185
✓
KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN990 OUT2 767
✓
KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN972 OUT3 197
✓
LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN913 OUT1 470
✓
LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN911 OUT1 414
✓
LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT940
×
LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckLastWordFirstNext(answer)
 Failed IN1 016 OUT3 311
×
LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckLastWordFirstNext(answer)
 Failed IN968 OUT3 660
✓
ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN908 OUT580
✓
ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN945 OUT2 723
×
ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN981 OUT1 318
×
ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN913 OUT7 265
×
ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN999 OUT1 383
✓
ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN922 OUT774
–
StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
✓
WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN992 OUT2 556
✓
WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN977 OUT341
✓
WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN914 OUT324