Ai4Users.comLocal AI & LLM community
Sign in to contributeSign in

Qwen 3.8 27B NVFP4 Think: medium | Benchmark #58

Described in:Qwen 3.8 27B vs Qwen 3.6 27B: benchmark results analysis

Benchmark summary

Instruction Following Benchmark based on IFBench

ModelQwen 3.8 27B QuantizationNVFP4 ModeThink: medium

IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.

Score 173/271 63.8% passed
173 passed96 failed2 broken29 skipped300 total 271 AI queries 0 tool calls TOKENS IN255 833 TOKENS OUT1 273 038

Results by difficulty

4 groups
Level Easy
15 / 16 93.8%
passed
16 AI queries 0 tool calls IN15 185 OUT12 317
Level Medium
77 / 99 77.8%
passed
99 AI queries 0 tool calls IN93 641 OUT212 391
Level Hard
68 / 102 66.7%
passed
102 AI queries 0 tool calls IN96 132 OUT317 081
Level VeryHard
13 / 54 24.1%
passed
54 AI queries 0 tool calls IN50 875 OUT731 249

Results by category

1 group
Category InstructionFollowing
63.8% 173/271 passed
271 AI queries 0 tool calls TOKENS IN255 833 TOKENS OUT1 273 038

Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.

Benchmark items
Script / test Status Tokens
Instruction Following - Alphabet & Consonants 1 passed17 failed1 broken
5.3%
 IN18 102 OUT364 937
×
AlphabetLoop200IFBench/200 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN952 OUT11 126
×
AlphabetLoop201IFBench/201 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN990 OUT12 962
✓
AlphabetLoop202IFBench/202 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN926 OUT2 824
×
AlphabetLoop203IFBench/203 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN947 OUT5 826
×
AlphabetLoop204IFBench/204 (words:alphabet): each word starts with the next alphabet letter, looping A after Z.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN942 OUT4 332
×
AlphabetLoop205IFBench/205 (words:alphabet, format:parentheses): alphabet loop AND >=5-deep nested brackets.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckAlphabetLoop(answer)
 Failed IN997 OUT10 895
×
ConsonantCluster206IFBench/206 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN979 OUT24 268
!
ConsonantCluster207IFBench/207 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN933 OUT62 144
×
ConsonantCluster208IFBench/208 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN946 OUT6 771
×
ConsonantCluster209IFBench/209 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN919 OUT44 095
×
ConsonantCluster210IFBench/210 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN993 OUT27 928
×
ConsonantCluster211IFBench/211 (words:consonants): every word has a consonant cluster (2+ adjacent consonants).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN918 OUT13 898
×
ConsonantCluster212IFBench/212 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted — its reference_text is absent from normalized.tsv and the check needs nltk.word_tokenize+ngrams within +/-2%, which is not faithfully reproducible.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN969 OUT24 801
×
ConsonantCluster213IFBench/213 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN955 OUT10 626
×
ConsonantCluster214IFBench/214 (words:consonants + ratio:overlap): consonant cluster in every word. Secondary ratio:overlap NOT asserted (non-reproducible NLTK trigram overlap; see IFBench/212).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN933 OUT21 142
×
ConsonantCluster215IFBench/215 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted — it needs syllapy syllable counts, which are not reproducibly portable (documented SkipTest reason for that family).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN917 OUT38 704
×
ConsonantCluster216IFBench/216 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN1 006 OUT20 482
×
ConsonantCluster217IFBench/217 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN918 OUT8 844
×
ConsonantCluster218IFBench/218 (words:consonants + words:odd_even_syllables): consonant cluster in every word. Secondary words:odd_even_syllables NOT asserted (syllapy; not reproducible — see IFBench/215).1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckConsonantClusters(answer)
 Failed IN962 OUT13 269
Instruction Following - Brackets & Quotes 14 passed5 failed1 skipped
73.7%
 IN17 971 OUT25 589
✓
Parentheses106IFBench/106 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN951 OUT808
✓
Parentheses107IFBench/107 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN926 OUT1 030
✓
Parentheses108IFBench/108 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN909 OUT1 051
✓
Parentheses109IFBench/109 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 000 OUT1 878
✓
Parentheses110IFBench/110 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN898 OUT2 451
✓
Parentheses111IFBench/111 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 006 OUT298
×
Parentheses112IFBench/112 (format:parentheses): nest brackets/braces at least 5 levels deep.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedParentheses(answer)
 Failed IN944 OUT669
×
Quotes118IFBench/118 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN932 OUT1 619
✓
Quotes119IFBench/119 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN907 OUT1 025
✓
Quotes120IFBench/120 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN915 OUT572
×
Quotes121IFBench/121 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN922 OUT641
✓
Quotes122IFBench/122 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN922 OUT673
×
Quotes123IFBench/123 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN1 004 OUT1 335
×
Quotes124IFBench/124 (format:quotes): quotes nested >=3 levels deep, alternating double/single.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNestedQuotes(answer)
 Failed IN983 OUT4 554
–
QuotesEmoji125IFBench/125 (format:quotes, format:emoji): SKIPPED — the format:emoji constraint relies on the Python emoji library's is_emoji (EMOJI_DATA table), not faithfully reproducible via Unicode codepoint ranges; SC# char iteration also splits astral emoji into UTF-16 surrogate pairs, so per-character emoji detection is infeasible. Skipped rather than partially verifying only quotes.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
✓
QuoteUnquote113IFBench/113 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN952 OUT3 088
✓
QuoteUnquote114IFBench/114 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN942 OUT1 989
✓
QuoteUnquote115IFBench/115 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN952 OUT726
✓
QuoteUnquote116IFBench/116 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN912 OUT560
✓
QuoteUnquote117IFBench/117 (format:quote_unquote): every quoted phrase followed by an unquoted explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN994 OUT622
Instruction Following - Bullet Points 12 passed5 failed
70.6%
 IN16 059 OUT36 902
×
NoBulletsBullets095IFBench/95 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSomeBullets(answer)
 Failed IN923 OUT1 307
✓
NoBulletsBullets096IFBench/96 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN931 OUT611
✓
NoBulletsBullets097IFBench/97 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT664
✓
NoBulletsBullets098IFBench/98 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN905 OUT357
✓
NoBulletsBullets099IFBench/99 (format:no_bullets_bullets): >=2 sentences ending in '.' then >=2 '*' bullets.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 009 OUT532
✓
SubBullets126IFBench/126 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN912 OUT454
×
SubBullets127IFBench/127 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN919 OUT403
×
SubBullets128IFBench/128 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN994 OUT702
×
SubBullets129IFBench/129 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN946 OUT805
×
SubBullets130IFBench/130 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSubBullets(answer)
 Failed IN959 OUT662
✓
SubBullets131IFBench/131 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN984 OUT130
✓
SubBullets132IFBench/132 (format:sub-bullets): '*' bullets, each with a '-' sub-bullet.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN922 OUT612
✓
SubBulletsStopWords133IFBench/133 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 27%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN950 OUT8 944
✓
SubBulletsStopWords134IFBench/134 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 37%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN923 OUT1 963
✓
SubBulletsStopWords135IFBench/135 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 30%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN930 OUT12 808
✓
SubBulletsStopWords136IFBench/136 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 9%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN946 OUT3 993
✓
SubBulletsStopWords137IFBench/137 (format:sub-bullets, ratio:stop_words): sub-bullets AND stopwords <= 14%.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN923 OUT1 955
Instruction Following - Custom Transformations 8 passed3 failed
72.7%
 IN9 954 OUT13 985
✓
CharacterReverse273IFBench/273 (custom:character_reverse): answer reversed per letter (contains 'elgae dlab').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT4 649
✓
CsvCity274IFBench/274 (custom:csv_city): valid 7-row CSV with exact header.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN907 OUT168
✓
CsvQuotes275IFBench/275 (custom:csv_quotes): 3 tab-delimited rows, every field double-quoted.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT272
✓
CsvSpecial276IFBench/276 (custom:csv_special_character): 14 rows + one quoted field with a special character.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN925 OUT784
✓
DateFormatList277IFBench/277 (custom:date_format_list): Napoleon battle start dates as YYYY-MM-DD, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN901 OUT1 522
×
EuropeanCapitals278IFBench/278 (custom:european_capitals_sort): 27 capitals sorted by latitude, exact order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckEuropeanCapitals(answer)
 Failed IN914 OUT1 499
×
Mcq279IFBench/279 (custom:mcq_count_length): 4 MCQs, 5 options each, progressively longer, no explanation.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckMcq(answer)
 Failed IN914 OUT1 195
✓
Multiples280IFBench/280 (custom:multiples): print only the multiples of 7 from 10 to 50.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN888 OUT144
×
ReverseNewline281IFBench/281 (custom:reverse_newline): African countries reverse-alphabetical, one per line.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckReverseNewline(answer)
 Failed IN887 OUT2 569
✓
SentenceAlphabet282IFBench/282 (custom:sentence_alphabet): 26-sentence story, sentences start a..z in order.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN896 OUT928
✓
WordReverse283IFBench/283 (custom:word_reverse): answer reversed per word (contains 'bald eagle').1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT255
Instruction Following - Indentation & Emoji 7 passed2 failed8 skipped
77.8%
 IN8 313 OUT16 622
–
Emoji065IFBench/65 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji066IFBench/66 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji067IFBench/67 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji068IFBench/68 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji069IFBench/69 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
Emoji070IFBench/70 (format:emoji): emoji at the end of every sentence. SKIPPED (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
EmojiKeyword071IFBench/71 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
–
EmojiKeyword072IFBench/72 (format:emoji, sentence:keyword): SKIPPED — primary format:emoji constraint unverifiable (emoji library).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Skipped: Not migrated to SC#: format:emoji needs the Python emoji library (EMOJI_DATA/ZWJ) - not available in SC#.
 Skipped IN0 OUT0
✓
LineIndent073IFBench/73 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT542
✓
LineIndent074IFBench/74 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN891 OUT295
×
LineIndent075IFBench/75 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckIndentStairs(answer)
 Failed IN900 OUT735
✓
LineIndent076IFBench/76 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN942 OUT227
✓
LineIndent077IFBench/77 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN918 OUT1 684
✓
LineIndent078IFBench/78 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN967 OUT756
✓
LineIndent079IFBench/79 (format:line_indent): create stairs by incrementally indenting each new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN940 OUT2 460
✓
LineIndentSentenceWords080IFBench/80 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN910 OUT3 292
×
LineIndentSentenceWords081IFBench/81 (format:line_indent, ratio:sentence_words): indentation stairs AND three equal-length, all-distinct-word sentences.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckCharCountUniqueWords(answer)
 Failed IN946 OUT6 631
Instruction Following - Keyword & Conjunction Counts 17 passed1 failed
94.4%
 IN17 214 OUT40 224
✓
Conjunctions005IFBench/5 (count:conjunctions): at least 5 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN986 OUT1 699
✓
Conjunctions006IFBench/6 (count:conjunctions): at least 2 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN982 OUT1 407
✓
Conjunctions007IFBench/7 (count:conjunctions): at least 3 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN945 OUT1 104
✓
Conjunctions008IFBench/8 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 008 OUT1 407
✓
Conjunctions009IFBench/9 (count:conjunctions): at least 4 different coordinating conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT1 216
✓
KeywordsMultiple000IFBench/0 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 002 OUT2 756
✓
KeywordsMultiple001IFBench/1 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN924 OUT2 168
✓
KeywordsMultiple002IFBench/2 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 007 OUT3 998
✓
KeywordsMultiple003IFBench/3 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 031 OUT6 931
×
KeywordsMultiple004IFBench/4 (count:keywords_multiple): exact keyword counts 1/2/3/5/7.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckKeywordsMultiple(answer, "quickly", "transform", "abundant", "bridge", "through")
 Failed IN951 OUT3 897
✓
Numbers017IFBench/17 (count:numbers): exactly 2 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN893 OUT433
✓
Numbers018IFBench/18 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN980 OUT804
✓
Numbers019IFBench/19 (count:numbers): exactly 13 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN951 OUT3 141
✓
Numbers020IFBench/20 (count:numbers): exactly 3 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT1 063
✓
Numbers021IFBench/21 (count:numbers): exactly 5 numbers in the response.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN914 OUT754
✓
NumbersConjunctions022IFBench/22 (count:numbers, count:conjunctions): exactly 15 numbers AND >=5 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN901 OUT1 414
✓
NumbersConjunctions023IFBench/23 (count:numbers, count:conjunctions): exactly 7 numbers AND >=4 unique conjunctions.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN898 OUT1 141
✓
NumbersUnique024IFBench/24 (count:numbers, count:unique_word_count): exactly 36 numbers AND >=36 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN912 OUT4 891
Instruction Following - Keyword In Nth Sentence 4 passed5 failed
44.4%
 IN8 358 OUT50 428
×
KeywordInSentence191IFBench/191 (sentence:keyword): keyword "malabathrum" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncludeKeyword(answer, "malabathrum", 27)
 Failed IN1 018 OUT13 061
×
KeywordInSentence192IFBench/192 (sentence:keyword): keyword "migratory" must appear in the 28th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncludeKeyword(answer, "migratory", 28)
 Failed IN914 OUT2 952
✓
KeywordInSentence193IFBench/193 (sentence:keyword): keyword "angora" must appear in the 3rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT414
✓
KeywordInSentence194IFBench/194 (sentence:keyword): keyword "heterolith" must appear in the 15th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN893 OUT1 128
×
KeywordInSentence195IFBench/195 (sentence:keyword): keyword "fleetness" must appear in the 27th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncludeKeyword(answer, "fleetness", 27)
 Failed IN936 OUT3 340
×
KeywordInSentence196IFBench/196 (sentence:keyword, count:word_count_range): keyword "trichosis" in the 22nd sentence AND 443-478 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncludeKeyword(answer, "trichosis", 22)
 Failed IN960 OUT19 389
✓
KeywordInSentence197IFBench/197 (sentence:keyword, count:word_count_range): keyword "iritic" in the 5th sentence AND 89-99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN912 OUT2 325
✓
KeywordInSentence198IFBench/198 (sentence:keyword, count:word_count_range): keyword "cactiform" in the 4th sentence AND 81-93 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT2 414
×
KeywordInSentence199IFBench/199 (sentence:keyword, count:word_count_range): keyword "betask" in the 18th sentence AND 246-254 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckWordCountRange(answer, 246, 254)
 Failed IN908 OUT5 405
Instruction Following - Lists, Lines & Options 16 passed3 failed
84.2%
 IN17 862 OUT18 944
✓
List082IFBench/82 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN962 OUT974
✓
List083IFBench/83 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN902 OUT341
✓
List084IFBench/84 (format:list): list items using !?!? instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT1 321
✓
List085IFBench/85 (format:list): list items using SEPARATOR instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN936 OUT491
✓
List086IFBench/86 (format:list): list items using - instead of bullet points.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT523
×
ListNoConsecutive087IFBench/87 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN1 010 OUT4 647
×
ListNoConsecutive088IFBench/88 (format:list, words:no_consecutive): !?!? separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")
 Failed IN942 OUT924
✓
ListNoConsecutive089IFBench/89 (format:list, words:no_consecutive): - separators AND no two consecutive words sharing a first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN914 OUT1 507
✓
NewLine090IFBench/90 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT1 727
✓
NewLine091IFBench/91 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN956 OUT2 537
✓
NewLine092IFBench/92 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN927 OUT1 051
✓
NewLine093IFBench/93 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN937 OUT1 511
✓
NewLine094IFBench/94 (format:newline): write each word on a new line.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN986 OUT266
✓
Options100IFBench/100 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN908 OUT139
✓
Options101IFBench/101 (format:options): answer with exactly "I know" or "I don't know".1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN932 OUT229
×
Options102IFBench/102 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Assertion expression evaluated to false: CheckOptions(answer, "a), b), c), d)")
 Failed IN1 017 OUT397
✓
Options103IFBench/103 (format:options): answer with exactly one of yes/no/maybe.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN922 OUT118
✓
Options104IFBench/104 (format:options): answer with exactly one of a)/b)/c)/d) (strict).1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN960 OUT54
✓
OptionsNewLine105IFBench/105 (format:options + format:newline): treść = jedna z opcji ORAZ każde słowo w osobnej linii.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN916 OUT187
Instruction Following - Names, Pronouns & Punctuation 14 passed2 failed
87.5%
 IN15 676 OUT34 759
✓
PersonNames025IFBench/25 (count:person_names): mention at least 10 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 049 OUT618
✓
PersonNames026IFBench/26 (count:person_names): mention at least 18 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 052 OUT1 518
✓
PersonNames027IFBench/27 (count:person_names): mention at least 3 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN1 048 OUT587
✓
PersonNames028IFBench/28 (count:person_names): mention at least 14 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 015 OUT613
✓
PersonNames029IFBench/29 (count:person_names): mention at least 23 different names from the fixed list.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 065 OUT2 988
✓
Pronouns030IFBench/30 (count:pronouns): response should include at least 22 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN892 OUT1 219
✓
Pronouns031IFBench/31 (count:pronouns): response should include at least 39 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN907 OUT4 168
✓
Pronouns032IFBench/32 (count:pronouns): response should include at least 7 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN971 OUT2 492
✓
Pronouns033IFBench/33 (count:pronouns): response should include at least 5 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN973 OUT397
✓
Pronouns034IFBench/34 (count:pronouns): response should include at least 13 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN901 OUT2 860
×
Punctuation035IFBench/35 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPunctuationCover(answer)
 Failed IN1 002 OUT8 098
✓
Punctuation036IFBench/36 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 008 OUT3 361
✓
Punctuation037IFBench/37 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN961 OUT1 286
✓
Punctuation038IFBench/38 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN949 OUT2 001
×
Punctuation039IFBench/39 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPunctuationCover(answer)
 Failed IN923 OUT777
✓
Punctuation040IFBench/40 (count:punctuation): use every standard punctuation mark incl. the interrobang.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN960 OUT1 776
Instruction Following - Palindromes & Letter Patterns 9 passed8 failed5 skipped
52.9%
 IN16 098 OUT129 770
✓
NoConsecutive224IFBench/224 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN931 OUT484
✓
NoConsecutive225IFBench/225 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN904 OUT537
✓
NoConsecutive226IFBench/226 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN903 OUT16 324
×
NoConsecutive227IFBench/227 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN994 OUT5 025
×
NoConsecutive228IFBench/228 (words:no_consecutive): no two consecutive words share the same first letter.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoConsecutiveFirstLetter(answer)
 Failed IN938 OUT8 716
–
OddEvenSyllables229IFBench/229 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables230IFBench/230 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables231IFBench/231 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables232IFBench/232 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
–
OddEvenSyllables233IFBench/233 (words:odd_even_syllables): SkipTest — syllapy syllable counting not reproducible.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: words:odd_even_syllables needs syllable counting (syllapy) - not available in SC#.
 Skipped IN0 OUT0
✓
Palindrome234IFBench/234 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN927 OUT1 252
✓
Palindrome235IFBench/235 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN949 OUT1 695
✓
Palindrome236IFBench/236 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN921 OUT1 525
✓
Palindrome237IFBench/237 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN918 OUT4 543
✓
Palindrome238IFBench/238 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN937 OUT3 057
×
Palindrome239IFBench/239 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckPalindromes(answer)
 Failed IN1 041 OUT3 135
✓
Palindrome240IFBench/240 (words:palindrome): include >=10 single-word palindromes, each >=5 characters.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 117 OUT5 111
×
PrimeLengths247IFBench/247 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN889 OUT8 637
×
PrimeLengths248IFBench/248 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN921 OUT12 416
×
PrimeLengths249IFBench/249 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN940 OUT37 329
×
PrimeLengths250IFBench/250 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN886 OUT19 059
×
PrimeLengths251IFBench/251 (words:prime_lengths): every word length must be a prime number.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckPrimeLengths(answer)
 Failed IN982 OUT925
Instruction Following - Prompt Repetition 6 passed3 failed
66.7%
 IN8 515 OUT6 476
✓
RepeatChange060IFBench/60 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN930 OUT201
✓
RepeatChange061IFBench/61 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN946 OUT264
✓
RepeatChange062IFBench/62 (repeat:repeat_change): repeat the request changing only the first word.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 001 OUT510
✓
RepeatSimple063IFBench/63 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 025 OUT217
×
RepeatSimple064IFBench/64 (repeat:repeat_simple): output only the fixed sentence, ignore the task.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSimple(answer)
 Failed IN892 OUT68
×
RepeatSpan284IFBench/284 (repeat:repeat_span): copy the inclusive word span [3, 6].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "explain the context, code and theory behind the paper Identifying Candidate Spaces for Advert Implantation.", 3, 6)
 Failed IN919 OUT1 444
×
RepeatSpan285IFBench/285 (repeat:repeat_span): copy the inclusive word span [5, 15].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckRepeatSpan(answer, "Write a descriptive, fictional, imaginative screenplay of a fish being pulled out of the water by a human and made to stare at the solar eclipse.", 5, 15)
 Failed IN930 OUT2 455
✓
RepeatSpan286IFBench/286 (repeat:repeat_span): copy the inclusive word span [0, 7].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN955 OUT344
✓
RepeatSpan287IFBench/287 (repeat:repeat_span): copy the inclusive word span [10, 12].1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN917 OUT973
Instruction Following - Ratios & Stop Words 13 passed6 failed1 broken
65.0%
 IN19 171 OUT100 159
×
RatioSentBalance153IFBench/153 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN922 OUT2 651
✓
RatioSentBalance154IFBench/154 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN943 OUT406
✓
RatioSentBalance155IFBench/155 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN905 OUT778
×
RatioSentBalance156IFBench/156 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN948 OUT781
✓
RatioSentBalance157IFBench/157 (ratio:sentence_balance): balanced declarative/interrogative/exclamatory counts.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 029 OUT1 237
×
RatioSentBalance158IFBench/158 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN981 OUT32 403
×
RatioSentBalance159IFBench/159 (ratio:sentence_balance, words:consonants): balanced sentence types AND every word has a consonant cluster.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentBalance(answer)
 Failed IN958 OUT15 225
✓
RatioSentType160IFBench/160 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN999 OUT1 781
×
RatioSentType161IFBench/161 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentType(answer)
 Failed IN982 OUT986
✓
RatioSentType162IFBench/162 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN977 OUT1 409
✓
RatioSentType163IFBench/163 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN909 OUT397
✓
RatioSentType164IFBench/164 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN950 OUT4 123
×
RatioSentType165IFBench/165 (ratio:sentence_type): 2:1 declarative-to-interrogative ratio.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckSentType(answer)
 Failed IN1 030 OUT2 041
✓
RatioStopWords171IFBench/171 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN949 OUT1 908
✓
RatioStopWords172IFBench/172 (ratio:stop_words): stop words <= 23% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN896 OUT928
✓
RatioStopWords173IFBench/173 (ratio:stop_words): stop words <= 20% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN926 OUT21 105
!
RatioStopWords174IFBench/174 (ratio:stop_words): stop words <= 47% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: response.Completed Values: response.Completed = false
 Broken IN935 OUT5 723
✓
RatioStopWords175IFBench/175 (ratio:stop_words): stop words <= 39% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN983 OUT1 933
✓
RatioStopWords176IFBench/176 (ratio:stop_words): stop words <= 31% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 002 OUT2 216
✓
RatioStopWords177IFBench/177 (ratio:stop_words): stop words <= 53% of total words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN947 OUT2 128
Instruction Following - Sentence Progression 7 passed6 failed
53.8%
 IN12 229 OUT52 150
✓
AlliterationIncrement178IFBench/178 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 002 OUT1 844
✓
AlliterationIncrement179IFBench/179 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN920 OUT1 710
×
AlliterationIncrement180IFBench/180 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN950 OUT10 259
×
AlliterationIncrement181IFBench/181 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingAlliteration(answer)
 Failed IN938 OUT2 935
✓
AlliterationIncrement182IFBench/182 (sentence:alliteration_increment): each sentence a longer alliterative run than the prior.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN930 OUT3 534
✓
SentenceIncrement183IFBench/183 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN943 OUT8 664
×
SentenceIncrement184IFBench/184 (sentence:increment): each sentence exactly 3 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)
 Failed IN959 OUT5 744
✓
SentenceIncrement185IFBench/185 (sentence:increment): each sentence exactly 5 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN899 OUT2 058
✓
SentenceIncrement186IFBench/186 (sentence:increment): each sentence exactly 4 more words than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN934 OUT4 122
✓
SentenceIncrement187IFBench/187 (sentence:increment): each sentence exactly 1 more word than the previous one.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN903 OUT2 058
×
SentenceIncrement188IFBench/188 (sentence:increment, format:thesis): sentences +3 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 3)
 Failed IN938 OUT3 374
×
SentenceIncrement189IFBench/189 (sentence:increment, format:thesis): sentences +4 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 4)
 Failed IN921 OUT3 194
×
SentenceIncrement190IFBench/190 (sentence:increment, format:thesis): sentences +5 words each AND each section opens with an italic (HTML) thesis.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckIncrementingWordCount(answer, 5)
 Failed IN992 OUT2 654
Instruction Following - Sentence Words & Overlap 4 passed1 failed9 skipped
80.0%
 IN4 736 OUT20 951
–
RatioOverlap144IFBench/144 (ratio:overlap): trigram overlap 72% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap145IFBench/145 (ratio:overlap): trigram overlap 93% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap146IFBench/146 (ratio:overlap): trigram overlap 74% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap147IFBench/147 (ratio:overlap): trigram overlap 59% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap148IFBench/148 (ratio:overlap): trigram overlap 6% (±2%) with reference text.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap149IFBench/149 (ratio:overlap, sentence:keyword): trigram overlap 13% (±2%); also keyword 'giraffe' in 7th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap150IFBench/150 (ratio:overlap, sentence:keyword): trigram overlap 24% (±2%); also keyword 'metricism' in 3rd sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap151IFBench/151 (ratio:overlap, sentence:keyword): trigram overlap 23% (±2%); also keyword 'horary' in 30th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
–
RatioOverlap152IFBench/152 (ratio:overlap, sentence:keyword): trigram overlap 11% (±2%); also keyword 'tinstuff' in 6th sentence (secondary constraint unported).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Skipped: Not migrated to SC#: ratio:overlap needs nltk.word_tokenize trigrams within a +/-2% band - no faithful SC#/.NET equivalent.
 Skipped IN0 OUT0
×
RatioSentWords166IFBench/166 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckSentenceWords(answer)
 Failed IN920 OUT2 808
✓
RatioSentWords167IFBench/167 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN933 OUT3 176
✓
RatioSentWords168IFBench/168 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN934 OUT6 116
✓
RatioSentWords169IFBench/169 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN992 OUT3 086
✓
RatioSentWords170IFBench/170 (ratio:sentence_words): exactly three sentences, equal char count, all different words.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN957 OUT5 765
Instruction Following - Templates & Whitespace 9 passed9 failed
50.0%
 IN16 513 OUT34 275
×
NoWhitespace296IFBench/296 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN888 OUT128
×
NoWhitespace297IFBench/297 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN889 OUT445
×
NoWhitespace298IFBench/298 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN886 OUT299
×
NoWhitespace299IFBench/299 (format:no_whitespace): the output must contain no whitespace.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckNoWhitespace(response.Text)
 Failed IN925 OUT1 344
✓
OutputTemplate292IFBench/292 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT426
✓
OutputTemplate293IFBench/293 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN911 OUT481
✓
OutputTemplate294IFBench/294 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN984 OUT2 087
✓
OutputTemplate295IFBench/295 (format:output_template): use the exact My Answer/My Conclusion/Future Outlook template.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN921 OUT593
✓
Thesis138IFBench/138 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN929 OUT932
✓
Thesis139IFBench/139 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN938 OUT1 201
✓
Thesis140IFBench/140 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN913 OUT1 075
✓
Thesis141IFBench/141 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN925 OUT2 824
✓
Thesis142IFBench/142 (format:thesis): each section begins with an italic (HTML) thesis statement.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN899 OUT829
×
ThesisWordCount143IFBench/143 (format:thesis, count:word_count_range): italic-thesis sections AND 231-239 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckWordCountRange(answer, 231, 239)
 Failed IN914 OUT8 490
×
TitleCase288IFBench/288 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN926 OUT4 736
×
TitleCase289IFBench/289 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN912 OUT1 983
×
TitleCase290IFBench/290 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN952 OUT5 533
×
TitleCase291IFBench/291 (format:title_case): write the entire response in title case.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckTitleCase(answer)
 Failed IN898 OUT869
Instruction Following - Vowels & Word Repeats 6 passed9 failed
40.0%
 IN14 038 OUT192 329
×
LimitedWordRepeat252IFBench/252 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)
 Failed IN970 OUT2 048
×
LimitedWordRepeat253IFBench/253 (words:repeats, small_n=4): no word repeated more than 4 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckLimitedWordRepeat(answer, 4)
 Failed IN907 OUT21 239
✓
LimitedWordRepeat254IFBench/254 (words:repeats, small_n=2): no word repeated more than 2 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN902 OUT1 320
✓
LimitedWordRepeat255IFBench/255 (words:repeats, small_n=5): no word repeated more than 5 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 018 OUT2 433
✓
LimitedWordRepeat256IFBench/256 (words:repeats, small_n=3): no word repeated more than 3 times.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN900 OUT1 073
✓
ThreeVowelParagraph263IFBench/263 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN900 OUT13 185
×
ThreeVowelParagraph264IFBench/264 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN898 OUT9 054
×
ThreeVowelParagraph265IFBench/265 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN932 OUT23 599
×
ThreeVowelParagraph266IFBench/266 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN895 OUT10 206
✓
ThreeVowelParagraph267IFBench/267 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN961 OUT24 256
×
ThreeVowelParagraph268IFBench/268 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN1 007 OUT13 491
×
ThreeVowelParagraph269IFBench/269 (words:vowel): single paragraph using at most three distinct vowels.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN891 OUT12 167
×
VowelPronouns270IFBench/270 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=12 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN946 OUT23 534
✓
VowelPronouns271IFBench/271 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=4 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN998 OUT14 287
×
VowelPronouns272IFBench/272 (words:vowel + count:pronouns): single 3-vowel paragraph AND >=27 pronouns.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckThreeVowelParagraph(answer)
 Failed IN913 OUT20 437
Instruction Following - Word Counts 17 passed2 failed
89.5%
 IN17 936 OUT77 825
✓
Japanese047IFBench/47 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN919 OUT1 612
✓
Japanese048IFBench/48 (count:words_japanese): every 29th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN930 OUT26 369
✓
Japanese049IFBench/49 (count:words_japanese): every 17th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN909 OUT8 756
×
Japanese050IFBench/50 (count:words_japanese): every 5th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckJapanese(answer, 5)
 Failed IN899 OUT7 031
✓
Japanese051IFBench/51 (count:words_japanese): every 25th word of the response must be in Japanese.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN923 OUT3 737
✓
UniqueWords052IFBench/52 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 102 OUT1 507
✓
UniqueWords053IFBench/53 (count:unique_word_count): use at least 55 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN972 OUT954
✓
UniqueWords054IFBench/54 (count:unique_word_count): use at least 85 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN888 OUT1 384
✓
UniqueWords055IFBench/55 (count:unique_word_count): use at least 135 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN897 OUT11 539
✓
UniqueWords056IFBench/56 (count:unique_word_count): use at least 65 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN1 049 OUT1 631
✓
UniqueWords057IFBench/57 (count:unique_word_count): use at least 160 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN893 OUT1 529
✓
UniqueWords058IFBench/58 (count:unique_word_count): use at least 128 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN891 OUT1 029
✓
UniqueWords059IFBench/59 (count:unique_word_count): use at least 170 unique words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN991 OUT932
✓
WordCountRange041IFBench/41 (count:word_count_range): response must contain between 71 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN1 004 OUT2 110
✓
WordCountRange042IFBench/42 (count:word_count_range): response must contain between 57 and 61 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN905 OUT1 963
✓
WordCountRange043IFBench/43 (count:word_count_range): response must contain between 24 and 28 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN969 OUT1 013
✓
WordCountRange044IFBench/44 (count:word_count_range): response must contain between 77 and 99 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN936 OUT2 376
✓
WordCountRange045IFBench/45 (count:word_count_range): response must contain between 30 and 73 words.1 AI query 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
 Passed IN903 OUT259
×
WordCountRangeList046IFBench/46 (count:word_count_range, format:list): 90-100 words AND >=2 "!?!?" separators.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckSpecialBullets(answer, "!?!?")
 Failed IN956 OUT2 094
Instruction Following - Word Position & Order 9 passed9 failed6 skipped
50.0%
 IN17 088 OUT56 713
✓
KeywordSpecificPosition010IFBench/10 (words:keywords_specific_position): keyword "giggle" as the 33rd word of the 22nd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN922 OUT3 550
✓
KeywordSpecificPosition011IFBench/11 (words:keywords_specific_position): keyword "beneath" as the 34th word of the 23rd sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN927 OUT3 019
×
KeywordSpecificPosition012IFBench/12 (words:keywords_specific_position): keyword "spotty" as the 35th word of the 24th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
Assertion expression evaluated to false: CheckKeywordSpecificPosition(answer, "spotty", 24, 35)
 Failed IN990 OUT1 039
✓
KeywordSpecificPosition013IFBench/13 (words:keywords_specific_position): keyword "slowly" as the 36th word of the 25th sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:VeryHardsource:IFBench
 Passed IN972 OUT3 640
×
LastWordFirstNext219IFBench/219 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckLastWordFirstNext(answer)
 Failed IN913 OUT1 256
✓
LastWordFirstNext220IFBench/220 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
 Passed IN911 OUT1 879
×
LastWordFirstNext221IFBench/221 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckLastWordFirstNext(answer)
 Failed IN918 OUT1 076
×
LastWordFirstNext222IFBench/222 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckLastWordFirstNext(answer)
 Failed IN1 016 OUT3 887
×
LastWordFirstNext223IFBench/223 (words:last_first): the last word of each sentence must start the next sentence.1 AI query 0 tool callscategory:InstructionFollowinglevel:Hardsource:IFBench
Assertion expression evaluated to false: CheckLastWordFirstNext(answer)
 Failed IN968 OUT6 009
×
ParagraphLastFirst241IFBench/241 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN908 OUT1 098
✓
ParagraphLastFirst242IFBench/242 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN945 OUT1 915
×
ParagraphLastFirst243IFBench/243 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN981 OUT1 202
×
ParagraphLastFirst244IFBench/244 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN913 OUT17 058
×
ParagraphLastFirst245IFBench/245 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
Assertion expression evaluated to false: CheckParagraphLastFirst(answer)
 Failed IN999 OUT2 177
✓
ParagraphLastFirst246IFBench/246 (words:paragraph_last_first): each paragraph must end with the word it started with.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN922 OUT5 155
–
StartVerb257IFBench/257 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb258IFBench/258 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb259IFBench/259 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb260IFBench/260 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb261IFBench/261 (words:start_verb): response must start with a verb.0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
–
StartVerb262IFBench/262 (words:start_verb, count:person_names): response must start with a verb (primary, unreproducible).0 AI queries 0 tool callscategory:InstructionFollowinglevel:Easysource:IFBench
Skipped: Not migrated to SC#: words:start_verb needs an NLTK POS tagger (nltk.pos_tag) - not available in SC#.
 Skipped IN0 OUT0
✓
WordsPosition014IFBench/14 (words:words_position): 2nd and 2nd-to-last words must be "vibrant".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN992 OUT1 712
✓
WordsPosition015IFBench/15 (words:words_position): 2nd and 2nd-to-last words must be "whisper".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN977 OUT616
✓
WordsPosition016IFBench/16 (words:words_position): 2nd and 2nd-to-last words must be "despite".1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:IFBench
 Passed IN914 OUT425