Benchmark summary
HumanEval adapted to SC#, adding an extra challenge: models must generate correct code in a custom sandboxed language while following its documentation and constraints.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Code Generation - Algorithms & Combinatorics
2 passed14 failed |
12.5% |
IN140 575 OUT53 062 | |
| × | BalancedParenthesesModel checks whether a string of parentheses is correctly balanced. Provenance: HumanEval/61 (entry point correct_bracketing). Level: Easy — single left-to-right pass tracking a balance that must never go negative and must end at zero.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: pair: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:83: Unexpected character '`'.
1:90: Unexpected character '#'.
5:24: Unexpected character '#'.
5:55: Unexpected character '#'.
5:104: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 738 OUT205 |
| × | CountBucketLoweringsModel counts total bucket lowerings needed to empty rows of wells. Provenance: HumanEval/115 (entry point max_fill). Level: Medium — per-row water sum, ceiling division by capacity, then a total; the ceiling with integer arithmetic is the trap.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: capacity-one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:44: Unexpected character '#'.
9:9: Unexpected character '#'.
9:16: Char literals must contain exactly one character and close with '''.
9:175: Char literals must contain exactly one character and close with '''.
13:63: Unexpected character '✓'.
14:42: Unexpected character '✓'.
15:42: Unexpected character '✓'.
19:2: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 783 OUT502 |
| × | EvaluateAlgebraExpressionModel writes an SC# function evaluating a small arithmetic expression with Python operator precedence. Provenance: HumanEval/160 (entry point do_algebra). Level: Hard — a mini expression evaluator over five operators across three precedence tiers, with the right-associativity of ** as the key trap and floor-division to get right.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: power-times-plus: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 890 OUT4 124 |
| × | FindValueCoordinatesInRaggedMatrixModel returns the coordinates of every cell equal to x in a ragged matrix, in a fixed order. Provenance: HumanEval/87 (entry point get_row). SC# represents each coordinate as a two-item List<int> [row, column]. Level: Medium — collect matches across variable-length rows, then a dual-key ordering (rows ascending, columns descending) that is easy to get backwards.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-multiple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 783 OUT366 |
| ✓ | GreatestFrequentIntegerModel returns the greatest positive integer whose frequency in the list is at least its own value, or -1 if none. Provenance: HumanEval/69 (entry point search). Level: Medium — count occurrences and combine two conditions (value>0 and frequency>=value) while tracking the maximum qualifying value.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 750 OUT722 |
| × | HasZeroSumTripleModel decides whether three elements at distinct positions sum to zero. Provenance: HumanEval/40 (entry point triples_sum_to_zero). Level: Medium — brute-force over index triples, with the trap that distinctness is by position, not by value (repeated equal values at different indices count).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-false: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:60: Unexpected character '`'.
1:75: Unexpected character '`'.
1:85: Unexpected character '`'.
1:99: Unexpected character '`'.
1:104: Unexpected character '`'.
3:2: Char literals must contain exactly one character and close with '''.
3:120: Unexpected character '³'.
5:176: Unexpected character '#'.
7:52: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 744 OUT315 |
| × | LargestKValuesSortedAscendingModel returns the k largest values sorted ascending. Provenance: HumanEval/120 (entry point maximum). Level: Medium — sort then take the top k, with duplicates preserved and the k = 0 edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: all-elements: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 747 OUT726 |
| × | LexicographicMinGridPathModel writes an SC# function returning the values along the lexicographically smallest length-k grid path. Provenance: HumanEval/129 (entry point minPath). Level: VeryHard — the naive search space is huge; a correct answer needs the non-obvious insight that the optimum bounces between the global minimum 1 and its smallest edge-neighbor. Unique answer, so exact list comparison is deterministic.1 AI query
0 tool callscategory:CodeGenerationlevel:VeryHardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: three-by-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 831 OUT3 260 |
| × | MinimumSubarraySumModel finds the minimum sum over all non-empty contiguous subarrays. Provenance: HumanEval/114 (entry point minSubArraySum). Level: Hard — a Kadane-style running-minimum with state, plus the all-positive trap (the answer is the smallest element, never 0). Uses long because inputs exceed int range.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: positive-baseline: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
1:185: Unexpected character '#'.
39:4: Char literals must contain exactly one character and close with '''.
57:101: Integer literal '9223372036854775808' is out of range for type long.
58:112: Integer literal '9223372036854775809' is out of range for type long.
63:455: Char literals must contain exactly one character and close with '''.
67:98: Unexpected character '#'.
69:32: Unexpected character '#'.
69:63: Char literals must contain exactly one character and close with '''.
147:16: Unexpected character '#'.
149:99: Char literals must contain exactly one character and close with '''.
151:59: Unexpected character '#'.
153:16: Unexpected character '#'.
153:288: Unexpected character '#'.
153:306: Unexpected character '#'.
153:587: Char literals must contain exactly one character and close with '''.
159:8: Unexpected character '#'.
159:16: Unexpected character '`'.
159:30: Unexpected character '`'.
161:443: Char literals must contain exactly one character and close with '''.
165:102: Char literals must contain exactly one character and close with '''.
171:486: Char literals must contain exactly one character and close with '''.
173:550: Char literals must contain exactly one character and close with '''.
181:27: Unexpected character '`'.
181:45: Unexpected character '`'.
183:16: Unexpected character '`'.
183:56: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 754 OUT6 358 |
| ✓ | MostFrequentLettersModel returns the most frequent letter(s) and their count, keeping every tie. Provenance: HumanEval/111 (entry point histogram). Level: Medium — tally counts, find the maximum, then collect all keys reaching it (ties included).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 767 OUT986 |
| × | OrderByDigitScoreModel writes an SC# function that stably sorts integers by a signed digit-sum score. Provenance: HumanEval/145 (entry point order_by_points). Level: Hard — combines a subtle scoring rule (leading digit of a negative number counts negative), a stable sort whose tie-break by original position is mandatory for determinism, and empty/negative/tie edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:171: Unexpected character '#'.
1:225: Unexpected character '#'.
1:708: Unexpected character '#'.
1:1180: Char literals must contain exactly one character and close with '''.
3:13: Unexpected character '#'.
3:233: Char literals must contain exactly one character and close with '''.
9:86: Unexpected character '#'.
35:9: Unexpected character '#'.
35:333: Unexpected character '#'.
35:406: Unexpected character '#'.
35:434: Unexpected character '#'.
39:113: Unexpected character '#'.
45:65: Char literals must contain exactly one character and close with '''.
47:75: Unexpected character '#'.
49:4: Char literals must contain exactly one character and close with '''.
65:87: Unexpected character '#'.
71:18: Unexpected character '`'.
71:53: Unexpected character '`'.
71:64: Unexpected character '`'.
71:68: Unexpected character '`'.
73:18: Unexpected character '`'.
73:37: Unexpected character '`'.
75:18: Unexpected character '`'.
75:33: Unexpected character '`'.
85:11: Unexpected character '`'.
85:35: Unexpected character '`'.
87:18: Unexpected character '`'.
87:65: Unexpected character '`'.
89:18: Unexpected character '`'.
89:33: Unexpected character '`'.
106:4: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 824 OUT2 565 |
| × | PlanetsBetweenModel writes an SC# function listing the planets between two named planets, ordered from the Sun. Provenance: HumanEval/148 (entry point bf). SC# returns a List<string> instead of a Python tuple. Level: Medium — map names to orbital positions, emit the strictly-between range sun-outward regardless of argument order, and handle invalid-name, equal-name and adjacent (empty) edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: outer: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:23: Unexpected character '#'.
1:130: Unexpected character '#'.
1:224: Unexpected character '#'.
1:501: Char literals must contain exactly one character and close with '''.
6:17: Unexpected character '#'.
7:282: Unexpected character '#'.
7:475: Unexpected character '`'.
7:490: Unexpected character '`'.
7:508: Unexpected character '`'.
7:519: Unexpected character '`'.
7:549: Unexpected character '`'.
7:588: Unexpected character '`'.
7:593: Unexpected character '`'.
7:620: Unexpected character '`'.
10:3: Unexpected character '#'.
12:48: Char literals must contain exactly one character and close with '''.
13:4: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 794 OUT1 043 |
| × | PolynomialRealRootModel finds one real root of a polynomial given its coefficients. Provenance: HumanEval/32 (entry point find_zero). Level: Hard — requires a numeric root-finding method (e.g. bracket a sign change, then bisect); no closed form, and correctness is a tolerance on the residual rather than an exact value.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: linear: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found 'for'.
Values:
run.Compiled = false |
Failed | IN8 768 OUT29 528 |
| × | SumEvenElementsAtOddIndicesModel sums the even values that sit at odd zero-based indices. Provenance: HumanEval/85 (entry point add). Level: Easy — one stride-2 loop with a parity filter and an accumulator.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: single-odd-index: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:69: Unexpected character '`'.
1:76: Unexpected character '#'.
5:30: Unexpected character '→'.
7:2: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 759 OUT224 |
| × | SwapCaseAndShiftVowelsModel swaps letter case and shifts vowels forward by two. Provenance: HumanEval/93 (entry point encode). Level: Medium — per-character case swap composed with a vowel-only alphabet shift, keeping the swapped case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: upper: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 787 OUT1 545 |
| × | ValidateMmDdYyyyDateModel validates a strict mm-dd-yyyy date string. Provenance: HumanEval/124 (entry point valid_date). Level: Hard — several interacting rules (fixed format, digit counts, dash positions, month range, per-month day range, no leap year) where any single miss flips the answer.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: valid-march: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:66: Unexpected character '`'.
1:73: Unexpected character '#'.
10:2: Char literals must contain exactly one character and close with '''.
20:9: Unexpected character '#'.
20:16: Char literals must contain exactly one character and close with '''.
20:128: Unexpected character '`'.
20:136: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 856 OUT593 |
Code Generation - Arithmetic & Predicates
2 passed12 failed |
14.3% |
IN122 591 OUT8 614 | |
| × | AbsoluteScoreDifferencesModel returns the element-wise absolute difference between two equal-length integer lists. Provenance: HumanEval/152 (entry point compare). Level: Easy — a single element-wise map (absolute difference) over paired lists, no edge cases beyond equal length.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 735 OUT197 |
| × | GpaToLetterGradesModel maps a list of numeric GPAs to letter grades using a fixed threshold table. Provenance: HumanEval/81 (entry point numerical_letter_grade). Level: Medium — 13 ordered thresholds with strict "greater-than" boundaries plus the exact 4.0 == A+ special case; boundary values are easy to misclassify.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-mixed: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
1:137: Unexpected character '#'.
1:1056: Unexpected character '#'.
16:48: Unexpected character '#'.
19:56: Unexpected character '#'.
19:63: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 855 OUT1 066 |
| ✓ | IsListMonotonicModel decides whether a list is monotonically non-decreasing or non-increasing. Provenance: HumanEval/57 (entry point monotonic). Level: Easy — single pass tracking two direction flags; equal adjacent values allowed.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 725 OUT490 |
| × | LargerOfTwoNumericStringsModel compares two numeric strings (decimal separator '.' or ',') and returns the larger one unchanged, or empty if equal. Provenance: HumanEval/137 (entry point compare_one). SC# uses string inputs; equal values return an empty string instead of Python None. Level: Medium — normalize separators, parse and compare numerically, then return the original string (preserving commas) or empty on a tie.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-int-strings: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 756 OUT708 |
| × | MeanAbsoluteDeviationModel computes the mean absolute deviation of a list of doubles around their mean. Provenance: HumanEval/4 (entry point mean_absolute_deviation).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:104: Unexpected character '#'.
1:181: Char literals must contain exactly one character and close with '''.
14:20: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 737 OUT426 |
| × | MedianOfIntegerListModel returns the median of a list of integers, averaging the two middle values for even length. Provenance: HumanEval/47 (entry point median). Level: Medium — sort then branch on parity, returning a possibly fractional average.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-odd: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 725 OUT811 |
| × | NearestNegativeAndPositiveModel finds the negative closest to zero and the smallest positive, with presence flags. Provenance: HumanEval/136 (entry point largest_smallest_integers). SC# returns a struct instead of Python's tuple with None. Level: Medium — two independent scans plus presence flags, excluding zero and handling both-absent cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-positive-only: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 828 OUT556 |
| × | PolynomialDerivativeCoefficientsModel returns the coefficients of a polynomial's derivative in the same coefficient form. Provenance: HumanEval/62 (entry point derivative). Level: Easy — single loop multiplying each coefficient by its index; constant gives an empty list.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 746 OUT353 |
| × | SortEvenLengthStringsModel keeps only even-length strings and returns them sorted by length then alphabetically, without mutating the input. Provenance: HumanEval/149 (entry point sorted_list_sum). Level: Medium — filter by parity plus a two-key (length, then lexicographic) ordering; several tie cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:18: Unexpected character '`'.
1:66: Unexpected character '`'.
1:73: Unexpected character '#'.
7:7: Char literals must contain exactly one character and close with '''.
14:40: Char literals must contain exactly one character and close with '''.
16:39: Unexpected character '`'.
16:41: Unexpected character '`'.
16:44: Unexpected character '`'.
16:47: Unexpected character '`'.
16:50: Unexpected character '`'.
16:52: Unexpected character '`'.
16:55: Unexpected character '`'.
16:58: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 735 OUT337 |
| × | SumOddValuesAtEvenIndexesModel sums the odd values located at even zero-based indexes. Provenance: HumanEval/121 (entry point solution). Level: Easy — a single strided loop with an odd-value guard; returns 0 when nothing matches.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-twelve: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 741 OUT389 |
| × | SumOfSquaredCeilingsModel rounds each number up with ceiling, squares it, and sums the results. Provenance: HumanEval/133 (entry point sum_squares). Level: Easy — map ceiling then square then accumulate; handles negatives (ceiling of -2.4 is -2) and the double-to-int result.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one-two-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found 'in'.
Values:
run.Compiled = false |
Failed | IN8 782 OUT536 |
| × | SumSquaresOfPositiveOddIntegersModel sums the squares of every value that is a positive odd integer, ignoring negatives, evens and non-integers. Provenance: HumanEval/151 (entry point double_the_difference). Level: Medium — parity plus an integrality guard over doubles (the non-integer filter is the trap), then sum of squares.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:245: Unexpected character '#'.
3:15: Unexpected character '#'.
3:336: Unexpected character '#'.
3:1171: Char literals must contain exactly one character and close with '''.
34:8: Unexpected character '#'.
38:67: Unexpected character '#'.
40:4: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 730 OUT1 585 |
| ✓ | SumWithIndexBasedPowersModel transforms each list entry by its index (square when index is a multiple of 3, cube when a multiple of 4 but not 3, unchanged otherwise) and sums the results. Provenance: HumanEval/142 (entry point sum_squares). Level: Medium — three-way index-based branch whose precedence (multiple-of-3 wins over multiple-of-4) is a genuine trap; empty-list edge case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 768 OUT809 |
| × | WordWithMostUniqueCharsModel returns the word with the most distinct characters, breaking ties by lexicographically smallest. Provenance: HumanEval/158 (entry point find_max). Level: Medium — count distinct characters per word, take the argmax, and break ties by lexicographic order (~3 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-string: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:18: Unexpected character '`'.
1:46: Unexpected character '`'.
1:53: Unexpected character '#'.
8:50: Char literals must contain exactly one character and close with '''.
16:17: Unexpected character '`'.
16:19: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 728 OUT351 |
Code Generation - Easy Fundamentals
1 passed9 failed |
10.0% |
IN87 166 OUT10 037 | |
| × | AddTwoIntegersModel implements integer addition. Provenance: HumanEval/53 (entry point Add).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: zero-left: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:50: Unexpected character '`'.
1:72: Unexpected character '`'.
1:87: Unexpected character '`'.
1:93: Unexpected character '`'.
1:100: Unexpected character '#'.
1:127: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 697 OUT81 |
| × | BalanceEverGoesNegativeModel detects whether a running balance ever goes negative. Provenance: HumanEval/3 (entry point BelowZero).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:53: Unexpected character '`'.
1:68: Unexpected character '`'.
1:89: Unexpected character '`'.
3:24: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 720 OUT117 |
| × | BinaryStringXorModel implements binary XOR on two equal-length bit strings. Provenance: HumanEval/11 (entry point StringXor).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 745 OUT495 |
| × | FilterStringsBySubstringModel filters strings by substring membership. Provenance: HumanEval/7 (entry point FilterBySubstring).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:101: Unexpected character '`'.
3:18: Unexpected character '#'.
4:8: Char literals must contain exactly one character and close with '''.
5:13: Unexpected character '`'.
5:21: Unexpected character '`'.
5:24: Unexpected character '`'.
5:27: Unexpected character '`'.
5:30: Unexpected character '`'.
5:39: Unexpected character '`'.
5:42: Unexpected character '`'.
5:66: Unexpected character '`'.
7:7: Unexpected character '`'.
7:11: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 710 OUT186 |
| ✓ | FractionalPartModel returns the fractional part of a positive number. Provenance: HumanEval/2 (entry point TruncateNumber).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 714 OUT149 |
| × | HasCloseElementsModel detects whether any two elements are closer than a threshold. Provenance: HumanEval/0 (entry point HasCloseElements).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-true: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 716 OUT8 464 |
| × | InsertDelimiterModel inserts a delimiter between consecutive elements. Provenance: HumanEval/5 (entry point Intersperse).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 724 OUT165 |
| × | StringLengthModel implements string length. Provenance: HumanEval/23 (entry point GetStringLength).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:8: Expected '(' to declare a function or '=' to declare a global variable after 'int Get', found end of file. Global variables must have an initializer.
Values:
run.Compiled = false |
Failed | IN8 695 OUT56 |
| × | SumAndProductModel returns the sum and product of a list of integers. Provenance: HumanEval/8 (entry point SumProduct).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:55: Unexpected character '#'.
1:182: Unexpected character '#'.
4:17: Unexpected character '`'.
4:34: Unexpected character '`'.
4:41: Unexpected character '`'.
4:49: Unexpected character '`'.
4:55: Unexpected character '`'.
4:67: Unexpected character '`'.
5:18: Unexpected character '`'.
5:36: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 741 OUT186 |
| × | SumFromOneToNModel implements the sum 1..n inclusive. Provenance: HumanEval/60 (entry point SumToN).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:50: Unexpected character '`'.
1:68: Unexpected character '`'.
7:81: Char literals must contain exactly one character and close with '''.
9:22: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 704 OUT138 |
Code Generation - Lists & Sequences
1 passed9 failed |
10.0% |
IN87 296 OUT4 775 | |
| × | AllValuesBelowThresholdModel checks that every value is strictly below a threshold. Provenance: HumanEval/52 (entry point BelowThreshold).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-true: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:58: Unexpected character '`'.
1:73: Unexpected character '`'.
1:83: Unexpected character '`'.
1:92: Unexpected character '`'.
1:96: Unexpected character '`'.
1:119: Unexpected character '`'.
1:124: Unexpected character '`'.
1:179: Unexpected character '`'.
1:185: Unexpected character '`'.
3:18: Unexpected character '#'.
5:7: Unexpected character '`'.
5:15: Unexpected character '`'.
6:25: Unexpected character '`'.
6:33: Unexpected character '`'.
6:36: Unexpected character '`'.
6:39: Unexpected character '`'.
6:42: Unexpected character '`'.
6:49: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 720 OUT188 |
| × | CanMakeFirstListAllEvenModel decides whether unlimited swaps between two lists can make the first list all even. Provenance: HumanEval/110 (entry point exchange).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-yes: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 744 OUT1 107 |
| × | CarrotsEatenAndRemainingModel computes total carrots eaten and carrots left after a meal, capping consumption at the remaining stock. Provenance: HumanEval/159 (entry point eat).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-example-1: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 773 OUT214 |
| × | CountEvenAndOddDigitsModel counts the even and odd digits of a number (using its absolute value). Provenance: HumanEval/155 (entry point even_odd_count).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-7: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 743 OUT892 |
| × | IncrementEachElementModel returns a new list with every element incremented by one. Provenance: HumanEval/42 (entry point IncrList).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 720 OUT194 |
| × | KeepPositiveNumbersModel keeps only positive integers, preserving order. Provenance: HumanEval/30 (entry point GetPositive).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-mixed: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 718 OUT156 |
| × | MaximumListElementModel returns the maximum element of a non-empty list. Provenance: HumanEval/35 (entry point MaxElement).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found 'in'.
Values:
run.Compiled = false |
Failed | IN8 703 OUT186 |
| ✓ | PairSummingToZeroModel detects two distinct positions summing to zero. Provenance: HumanEval/43 (entry point PairsSumToZero).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 725 OUT660 |
| × | RollingMaximumModel implements a rolling (running) maximum over a list. Provenance: HumanEval/9 (entry point rolling_max).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 722 OUT428 |
| × | SortedCommonElementsModel returns the ascending-sorted list of values common to two integer lists, without duplicates. Provenance: HumanEval/58 (entry point common).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-many: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 728 OUT750 |
Code Generation - Number Theory
6 passed8 failed |
42.9% |
IN122 541 OUT13 221 | |
| × | BuildStonePileModel builds a pile of n levels: the first level has n stones and each next level has 2 more. Provenance: HumanEval/100 (entry point make_a_pile). Level: Easy — a single accumulation loop (n + 2*i); no real edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 768 OUT308 |
| ✓ | CountOddBoundaryDigitsModel counts values greater than 10 whose first and last decimal digits are both odd. Provenance: HumanEval/146 (entry point specialFilter). Level: Medium — per element: a magnitude guard, extracting the leading digit (its own loop), extracting the last digit, and an odd-digit test (~3-4 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 745 OUT699 |
| ✓ | CountPositiveSignedDigitSumsModel counts the integers whose signed digit sum is greater than 0, where a negative number's most significant digit carries the minus sign (-123 -> -1 + 2 + 3 = 4). Provenance: HumanEval/108 (entry point count_nums). Level: Medium — extract the most significant digit, apply the sign only to it, sum the digits, and count positives, with the negative and zero edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 792 OUT1 608 |
| ✓ | CountSevensInMultiplesOf11Or13Model counts occurrences of the digit 7 across the multiples of 11 or 13 below n. Provenance: HumanEval/36 (entry point fizz_buzz). Level: Medium — iterate a range, filter by a divisibility union without double counting multiples of both, and count a digit.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 724 OUT641 |
| × | DigitSumInBinaryModel sums the decimal digits of an integer and returns that sum in binary. Provenance: HumanEval/84 (entry point solve). Level: Medium — composes two sub-steps (decimal digit sum, then base-2 conversion) with a zero edge case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-1000: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
24:7: Type 'StringBuilder' has no supported method named 'Reverse' with 0 argument(s). Supported methods: Append, AppendLine, Clear, ToString.
Values:
run.Compiled = false |
Failed | IN8 779 OUT1 679 |
| ✓ | EveryThreeConsecutiveDistinctModel checks whether every window of three consecutive characters in a string is made of distinct characters. Provenance: HumanEval/80 (entry point is_happy). Level: Easy — a single sliding window with a length guard and a pairwise-distinct check.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 740 OUT444 |
| × | FactorialOrTriangularByParityModel returns a list of size n where, for 1-based index i, even i holds i! and odd i holds 1 + 2 + ... + i. Provenance: HumanEval/106 (entry point f). Level: Medium — a loop with a parity branch selecting between two sub-computations (factorial vs triangular sum), plus the 1-based-index twist.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
1:319: Char literals must contain exactly one character and close with '''.
3:16: Unexpected character '#'.
3:624: Unexpected character '#'.
7:67: Unexpected character '#'.
13:20: Unexpected character '#'.
15:112: Unexpected character '#'.
17:36: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 769 OUT881 |
| ✓ | FractionProductIsWholeModel decides whether the product of two positive fractions is a whole number. Provenance: HumanEval/144 (entry point simplify). Level: Medium — parse two "num/den" strings into four integers, cross-multiply, then test divisibility (~3 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 727 OUT1 119 |
| ✓ | IsPrimeNumberModel tests whether an integer is a prime number. Provenance: HumanEval/31 (entry point is_prime). Level: Easy — a single well-known trial-division loop with an early exit and one edge (values below 2).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 711 OUT762 |
| × | IsSimplePowerOfNModel decides whether one integer is an exact power of another. Provenance: HumanEval/76 (entry point is_simple_power). Level: Medium — the model must handle the n==1 special case and the exponent-0 case (x==1) that a naive multiply loop misses; the spec no longer hands over the loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-sixteen-two: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 796 OUT1 367 |
| × | NumbersWithAllOddDigitsModel returns, sorted ascending, the values whose decimal digits are all odd. Provenance: HumanEval/104 (entry point unique_digits). Level: Medium — filter by an all-odd-digits predicate (per-value digit decomposition) then sort.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-mixed: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 753 OUT1 165 |
| × | PrimesBelowNModel returns all prime numbers less than n in ascending order (empty when n <= 2). Provenance: HumanEval/96 (entry point count_up_to). Level: Medium — compose a primality test (inner loop) with a collection loop and the n<=2 edge; more than a single simple loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 720 OUT581 |
| × | RangeAverageInBinaryModel computes the average of the inclusive integer range n..m, rounds it to the nearest integer, and returns that value in binary with a "0b" prefix; returns "-1" when n > m. Provenance: HumanEval/103 (entry point rounded_avg). SC# uses a string return type, so the invalid range yields "-1" instead of Python's integer -1. Determinism note: every input uses an even (n + m) so the mean is an exact integer and no rounding tie occurs (the original's tie cases depend on Python banker's rounding, which is not portable to SC# Math.Round). Level: Medium — sum/average a range, then a base-conversion loop to binary, plus the invalid edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 752 OUT1 016 |
| × | SortByBinaryOneBitsModel sorts integers by their binary one-bit count, breaking ties by numeric value. Provenance: HumanEval/116 (entry point sort_array). Level: Medium — combine a bit-count computation with a stable two-key ordering; negatives count bits of the absolute value to match the benchmark.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-positive: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 765 OUT951 |
Code Generation - Numbers & Math
4 passed14 failed1 broken |
21.1% |
IN165 874 OUT27 413 | |
| × | BiggestEvenInRangeModel returns the largest even integer in an inclusive range, or -1 when none exists. Provenance: HumanEval/102 (entry point choose_num).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-middle: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
3:179: Unexpected character '#'.
3:700: Char literals must contain exactly one character and close with '''.
5:8: Unexpected character '#'.
5:123: Unexpected character '#'.
17:47: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 725 OUT681 |
| × | CarRaceCollisionsModel computes the number of head-on car collisions. Provenance: HumanEval/41 (entry point CarRaceCollision).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-two: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:67: Unexpected character '`'.
3:24: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 741 OUT104 |
| × | CountMangoesInBasketModel computes the count of mango fruits given a "N apples and M oranges" string and a fruit total. Provenance: HumanEval/67 (entry point fruit_distribution).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-5-6-19: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:10: Expected '(' to declare a function or '=' to declare a global variable after 'int Fruit', found end of file. Global variables must have an initializer.
Values:
run.Compiled = false |
Failed | IN8 743 OUT455 |
| × | CountNumbersStartingOrEndingWithOneModel counts n-digit positive integers that start or end with the digit 1. Provenance: HumanEval/83 (entry point starts_one_ends).1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
1:96: Unexpected character '#'.
1:2117: Char literals must contain exactly one character and close with '''.
3:8: Unexpected character '#'.
15:20: Unexpected character '#'.
19:254: Char literals must contain exactly one character and close with '''.
27:4: Char literals must contain exactly one character and close with '''.
29:533: Unexpected character '#'.
42:15: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 728 OUT1 723 |
| × | EvenDigitsInRangeModel returns the even single digits lying between two bounds, in ascending order. Provenance: HumanEval/163 (entry point generate_integers).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-2-10: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:7: Expected '>' to close the List type, found end of file.
Values:
run.Compiled = false |
Failed | IN8 763 OUT268 |
| × | GreatestCommonDivisorModel implements the greatest common divisor of two integers. Provenance: HumanEval/13 (entry point greatest_common_divisor). Level: Medium — Euclid's algorithm keeps state across iterations (repeated modulo + swap until the remainder is 0), beyond a single expression or plain accumulator loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-coprime: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:70: Unexpected character '#'.
1:74: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 706 OUT155 |
| ✓ | HasPrimeLengthModel returns true when a string's length is a prime number. Provenance: HumanEval/82 (entry point prime_length).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 708 OUT479 |
| ✓ | IsPerfectCubeModel detects whether an integer is a perfect cube, including negatives and zero. Provenance: HumanEval/77 (entry point iscube).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 714 OUT1 909 |
| × | IsRightAngleTriangleModel determines whether three side lengths form a right-angled triangle. Provenance: HumanEval/157 (entry point right_angle_triangle).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-3-4-5: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:860: Char literals must contain exactly one character and close with '''.
10:8: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 730 OUT830 |
| × | IsSumOfFourPositiveEvensModel decides whether n is expressible as a sum of exactly four positive even numbers. Provenance: HumanEval/138 (entry point is_equal_to_sum_even).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-4: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 713 OUT269 |
| ✓ | LargestProperDivisorModel returns the largest proper divisor. Provenance: HumanEval/24 (entry point largest_divisor).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 704 OUT897 |
| × | ProductOfOddDigitsModel returns the product of the odd decimal digits, or 0 when every digit is even. Provenance: HumanEval/131 (entry point digits).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-5: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:61: Unexpected character '`'.
8:15: Char literals must contain exactly one character and close with '''.
10:20: Char literals must contain exactly one character and close with '''.
12:2: Char literals must contain exactly one character and close with '''.
23:48: Char literals must contain exactly one character and close with '''.
25:64: Unexpected character '→'.
27:24: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 715 OUT386 |
| × | ProductOfUnitDigitsModel returns the product of the unit digits of two integers, treating them as non-negative. Provenance: HumanEval/97 (entry point multiply).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-148-412: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:66: Unexpected character '`'.
1:122: Unexpected character '`'.
1:124: Unexpected character '`'.
1:130: Unexpected character '`'.
1:132: Unexpected character '`'.
4:30: Unexpected character '`'.
4:32: Unexpected character '`'.
5:30: Unexpected character '`'.
5:32: Unexpected character '`'.
10:26: Unexpected character '`'.
10:35: Unexpected character '`'.
10:48: Unexpected character '`'.
10:51: Unexpected character '`'.
13:11: Unexpected character '`'.
13:28: Unexpected character '`'.
13:44: Unexpected character '`'.
13:65: Unexpected character '`'.
15:11: Unexpected character '`'.
15:28: Unexpected character '`'.
17:38: Unexpected character '`'.
17:51: Unexpected character '`'.
17:54: Unexpected character '`'.
17:72: Unexpected character '`'.
17:97: Char literals must contain exactly one character and close with '''.
19:16: Unexpected character '`'.
19:52: Unexpected character '`'.
21:21: Unexpected character '#'.
21:39: Unexpected character '#'.
21:53: Unexpected character '`'.
21:55: Unexpected character '`'.
21:130: Unexpected character '`'.
21:144: Unexpected character '`'.
21:152: Unexpected character '`'.
21:169: Unexpected character '`'.
24:76: Unexpected character '✓'.
25:75: Unexpected character '✓'.
Values:
run.Compiled = false |
Failed | IN8 759 OUT512 |
| ! | RoundToClosestIntegerModel parses a numeric string and rounds it to the closest integer, breaking .5 ties away from zero. Provenance: HumanEval/99 (entry point closest_integer).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Skipped: AI response was not completed |
Broken | IN8 744 OUT15 536 |
| ✓ | SelectValueByPrimalityModel returns x when n is prime and y otherwise. Provenance: HumanEval/150 (entry point x_or_y).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 719 OUT544 |
| × | SumMagnitudesTimesSignProductModel multiplies the sum of magnitudes by the product of the signs (any zero makes the product zero); empty input returns null. Provenance: HumanEval/128 (entry point prod_signs).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-negative: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 749 OUT1 338 |
| × | SumSmallElementsFromPrefixModel sums the values among the first k entries whose absolute value has at most two digits. Provenance: HumanEval/122 (entry point add_elements).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-negatives: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 739 OUT212 |
| × | TriangleAreaFromBaseAndHeightModel returns the area of a triangle from base and height. Provenance: HumanEval/45 (entry point TriangleArea).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-five-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 717 OUT97 |
| × | TriangleAreaRoundedModel computes Heron's triangle area rounded to two decimals, or -1 for degenerate side lengths. Provenance: HumanEval/71 (entry point triangle_area).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-3-4-5: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 757 OUT1 018 |
Code Generation - Parsing & Decoding
2 passed12 failed |
14.3% |
IN122 655 OUT14 393 | |
| ✓ | CircularShiftDigitsModel circularly shifts the decimal digits of an integer to the right, returning the result as a string. Provenance: HumanEval/65 (entry point circular_shift). Level: Medium — convert to digits, branch on shift vs digit count (reverse when it overflows), rotate by the right amount, and preserve leading zeros the rotation creates; several interacting edge cases and an easy direction/off-by-one trap.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 765 OUT1 888 |
| × | ConvertIntegerToSmallBaseModel converts a positive integer to a base below ten and returns its string representation. Provenance: HumanEval/44 (entry point change_base). Level: Medium — repeated division/modulo building the digits in the correct order; a small but genuine algorithm.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-8-base-3: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 736 OUT1 341 |
| × | CountPrimeHexDigitsModel counts how many hexadecimal digits of a string have prime value. Provenance: HumanEval/78 (entry point hex_key). Level: Easy — a single pass counting membership in a fixed six-element set (the set is part of the original spec, not a hint).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-ab: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Unterminated string literal. Strings must be closed with '"' on the same line.
Values:
run.Compiled = false |
Failed | IN8 756 OUT358 |
| × | DecimalToBinaryWithDelimitersModel converts a non-negative integer to its binary digits and wraps the result with "db" markers. Provenance: HumanEval/79 (entry point decimal_to_binary). Level: Easy — a standard repeated-division binary conversion (single loop) with the zero edge case and a fixed prefix/suffix wrap.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-zero: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 766 OUT1 370 |
| × | DecodeCaesarShiftByFiveModel decodes lowercase text that was Caesar-shifted forward by five, by shifting each letter back five with wraparound. Provenance: HumanEval/50 (entry point decode_shift). Level: Easy — single pass of modular character arithmetic with one wraparound edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-hello: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 738 OUT1 149 |
| × | DecodeCyclicGroupsOfThreeModel decodes text whose full three-character groups were rotated left, by rotating each such group right. Provenance: HumanEval/38 (entry point decode_cyclic). Level: Medium — process the string in groups of three, rotate each full group, and leave a short trailing group unchanged.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 753 OUT887 |
| × | HasNestedBracketSubsequenceModel detects whether a bracket string has a subsequence containing a nested bracket pair. Provenance: HumanEval/132 (entry point is_nested). Level: Hard — subsequence nesting is subtly different from balanced brackets; naive stack or balance checks give wrong answers on many cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-nested: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 831 OUT2 490 |
| × | MaxNestingDepthPerGroupModel reports the maximum nesting depth of each parenthesis group. Provenance: HumanEval/6 (entry point parse_nested_parens). Level: Medium — per-group depth tracking with a running maximum, reset at each group boundary; composed loop with state, beyond a single expression.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-multiple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 741 OUT1 146 |
| × | ParseMusicBeatsModel parses ASCII note tokens into beat counts. Provenance: HumanEval/17 (entry point parse_music). Level: Easy — split on spaces and map each token through a fixed three-way legend; no state, single empty-input edge. Below the old "Medium" bucket.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 741 OUT615 |
| × | RotationSubstringCheckModel reports whether the second word or any rotation of it occurs as a contiguous substring of the first. Provenance: HumanEval/154 (entry point cycpattern_check). Level: Medium — the "any rotation" requirement is a real trap; a nested scan over all substrings and all rotations (b+b trick) is needed.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-false: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 723 OUT823 |
| ✓ | SeparateBalancedParenGroupsModel separates a string into its top-level balanced parenthesis groups. Provenance: HumanEval/1 (entry point separate_paren_groups). Level: Medium — a depth-counter state machine that accumulates characters and emits a group each time depth returns to zero, plus the space-ignoring rule; a couple of composed steps beyond a plain loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 783 OUT1 305 |
| × | SplitWordsOrCountLettersModel splits text on spaces, else on commas, else counts a subset of its lowercase letters, returning a discriminated result struct. Provenance: HumanEval/125 (entry point split_words). Level: Medium — three mutually exclusive branches, an empty-part filter, and a parity-based letter count, all packed into a typed result struct that emulates Python's list-or-int return.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-space: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:22: Unexpected character '`'.
1:33: Unexpected character '`'.
3:18: Unexpected character '`'.
3:35: Unexpected character '`'.
3:49: Unexpected character '`'.
3:62: Unexpected character '`'.
3:65: Unexpected character '`'.
3:75: Unexpected character '`'.
3:78: Unexpected character '`'.
3:97: Unexpected character '`'.
5:34: Unexpected character '→'.
6:39: Unexpected character '→'.
7:11: Unexpected character '→'.
9:64: Char literals must contain exactly one character and close with '''.
11:132: Unexpected character '`'.
11:151: Unexpected character '`'.
13:34: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 857 OUT546 |
| × | StringToMd5DigestModel returns the lowercase hexadecimal MD5 digest of text, or null when text is empty. Provenance: HumanEval/162 (entry point string_to_md5). Level: Easy — a single guarded call to the built-in Md5 over the text's UTF-8 bytes, plus the empty-input null guard; the challenge is finding and using the library primitive, not the algorithm.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: empty-null: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 723 OUT264 |
| × | ValidateAngleBracketNestingModel validates that a string of angle brackets is correctly nested. Provenance: HumanEval/56 (entry point correct_bracketing). Level: Medium — running depth counter with two interacting failure modes (never negative, and ends balanced).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-simple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:83: Unexpected character '`'.
5:24: Unexpected character '#'.
5:55: Unexpected character '#'.
5:86: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 742 OUT211 |
Code Generation - Recursion & Sequences
3 passed12 failed |
20.0% |
IN131 369 OUT17 063 | |
| × | BrazilianFactorialModel writes an SC# function returning the Brazilian factorial n! * (n-1)! * ... * 1!. Provenance: HumanEval/139 (entry point special_factorial). Level: Medium — a running-factorial product loop (~3 steps); the main trap is that the result overflows int at n=7, so it must be computed and returned as long.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
1:1066: Unexpected character '#'.
1:1157: Unexpected character '#'.
1:1252: Char literals must contain exactly one character and close with '''.
5:47: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 776 OUT599 |
| × | CountEvenAndOddPalindromesModel counts even and odd integer palindromes from 1 to n. Provenance: HumanEval/107 (entry point even_odd_palindrome). SC# returns the tuple as a two-item List<int> [evenCount, oddCount]. Level: Medium — palindrome detection over a range plus two parity-partitioned counters.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: one-two-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 767 OUT560 |
| ✓ | CountTriplesDivisibleByThreeModel writes an SC# function counting index triples whose mapped values sum to a multiple of 3. Provenance: HumanEval/147 (entry point get_max_triples). Level: Medium — build a[i] = i*i - i + 1 over a 1-based range, then count triples i<j<k with sum divisible by 3; a straightforward triple loop suffices for the tested sizes (up to n=100).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 764 OUT2 382 |
| × | Fib4SequenceModel computes the four-step Fibonacci-like sequence iteratively. Provenance: HumanEval/46 (entry point fib4). Level: Medium — maintain a sliding window of four previous terms with four specific base cases; recursion is disallowed for efficiency.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: zero: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:86: Unexpected character '#'.
9:38: Char literals must contain exactly one character and close with '''.
27:28: Unexpected character '#'.
30:16: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 771 OUT493 |
| ✓ | FibFibSequenceModel computes the three-step Fibonacci-like (tribonacci) sequence iteratively. Provenance: HumanEval/63 (entry point fibfib). Level: Medium — maintain a sliding window of three previous terms with three base cases; compute efficiently rather than by naive recursion.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 759 OUT707 |
| × | IntervalIntersectionLengthIsPrimeModel reports whether the length of two intervals' intersection is prime. Provenance: HumanEval/127 (entry point intersection). SC# represents each interval as a two-item List<int> [start, end]. Level: Medium — compute the overlap, apply the (end - start) length convention, then test primality.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: touching: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:7: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 809 OUT2 218 |
| ✓ | LargestPrimeFactorModel returns the largest prime factor of n. Provenance: HumanEval/59 (entry point largest_prime_factor). Level: Medium — trial division with state, dividing each factor out fully before advancing, and handling the final remaining prime.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 707 OUT742 |
| × | NthFibonacciModel returns the nth Fibonacci number. Provenance: HumanEval/55 (entry point fib). Level: Easy — the classic two-variable accumulator loop with base cases 0 and 1.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: zero: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:58: Unexpected character '#'.
3:3: Unexpected character '`'.
3:18: Unexpected character '`'.
8:2: Char literals must contain exactly one character and close with '''.
10:20: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 723 OUT209 |
| × | NthPrimeFibonacciModel returns the nth Fibonacci number that is also prime, 1-based. Provenance: HumanEval/39 (entry point prime_fib). Level: Medium — compose iterative Fibonacci generation with a primality test and a running count until the nth prime Fibonacci is reached (all tested values stay within int).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: first: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:63: Unexpected character '`'.
15:9: Unexpected character '#'.
15:16: Char literals must contain exactly one character and close with '''.
16:6: Unexpected character '`'.
16:23: Unexpected character '`'.
17:12: Unexpected character '`'.
17:28: Unexpected character '`'.
24:105: Char literals must contain exactly one character and close with '''.
26:28: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 733 OUT713 |
| × | OddNumbersInCollatzSequenceModel returns the odd values of a Collatz sequence, sorted ascending. Provenance: HumanEval/123 (entry point get_odd_collatz). Level: Medium — generate the state-driven sequence to 1, filter odds (including 1), then sort.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: fourteen: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 775 OUT575 |
| × | PowerOfTwoModuloModel computes 2 to the power n modulo p. Provenance: HumanEval/49 (entry point modp). Level: Medium — the trap is numeric overflow: with n up to ~1100, 2^n must be reduced modulo p each step rather than computed then reduced.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:62: Unexpected character '`'.
13:9: Unexpected character '→'.
14:54: Char literals must contain exactly one character and close with '''.
16:24: Unexpected character '#'.
16:124: Char literals must contain exactly one character and close with '''.
16:324: Char literals must contain exactly one character and close with '''.
24:24: Unexpected character '#'.
24:32: Unexpected character '`'.
24:37: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 743 OUT583 |
| × | PrimeFactorizationModel implements integer prime factorization, listing factors from smallest to largest with each factor repeated as many times as it divides the input. Provenance: HumanEval/25 (entry point factorize). Level: Medium — trial division with running state (divide the factor out and re-test it), emitting repeats in a defined ascending order; more than a single expression or plain accumulator loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: prime-two: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 760 OUT676 |
| × | ProductOfThreePrimesModel decides whether a number is the product of exactly three prime numbers (with multiplicity). Provenance: HumanEval/75 (entry point is_multiply_prime). Level: Medium — factor the number and check that the total count of prime factors, counting repeats, equals exactly three.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: single-prime: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 749 OUT764 |
| × | SumDigitsOfLargestPrimeModel returns the digit sum of the largest prime in a list. Provenance: HumanEval/94 (entry point skjkasdkd). Level: Medium — primality test, tracking the running maximum prime, then summing its digits.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-181: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:23: Unexpected character '#'.
1:391: Unexpected character '#'.
1:693: Unexpected character '±'.
3:20: Unexpected character '#'.
7:172: Unexpected character '#'.
7:451: Char literals must contain exactly one character and close with '''.
41:9: Unexpected character '#'.
41:318: Unexpected character '`'.
41:327: Unexpected character '`'.
41:353: Unexpected character '`'.
41:362: Unexpected character '`'.
41:401: Unexpected character '`'.
41:410: Unexpected character '`'.
41:485: Unexpected character '#'.
41:511: Unexpected character '#'.
48:41: Char literals must contain exactly one character and close with '''.
54:4: Char literals must contain exactly one character and close with '''.
100:8: Unexpected character '#'.
104:149: Char literals must contain exactly one character and close with '''.
114:164: Unexpected character '#'.
114:171: Char literals must contain exactly one character and close with '''.
120:16: Unexpected character '#'.
120:24: Unexpected character '`'.
120:32: Unexpected character '`'.
120:43: Unexpected character '`'.
120:49: Unexpected character '`'.
120:77: Char literals must contain exactly one character and close with '''.
122:63: Unexpected character '`'.
122:95: Unexpected character '`'.
126:20: Unexpected character '#'.
128:16: Unexpected character '`'.
128:20: Unexpected character '`'.
128:38: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 737 OUT3 934 |
| × | TribonacciLikeSequenceModel writes an SC# function returning the first n+1 values of a Tribonacci-like sequence. Provenance: HumanEval/130 (entry point tri). Level: Hard — the odd-index rule references tri(n+1), a forward reference that would loop under naive recursion; the solver must realize the referenced term is even-indexed and directly computable, then build the list iteratively.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: zero: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 796 OUT1 908 |
Code Generation - Sequences & Sorting
4 passed11 failed |
26.7% |
IN131 132 OUT10 612 | |
| ✓ | CanSortByRightShiftsModel decides whether a list can be sorted ascending using only whole-array right shifts (rotations). Provenance: HumanEval/109 (entry point move_one_ball). Level: Medium — needs the insight that a rotation-sortable list has at most one circular descent; a couple of edge cases (empty, already sorted).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 769 OUT1 609 |
| × | FindClosestPairModel finds the two closest values in a list and returns them in ascending order. Provenance: HumanEval/20 (entry point find_closest_elements). Level: Medium — an O(n^2) scan over all pairs tracking the minimal distance, plus ordering the winning pair.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-39-40: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:63: Unexpected character '`'.
1:78: Unexpected character '`'.
1:91: Unexpected character '`'.
7:77: Char literals must contain exactly one character and close with '''.
9:28: Unexpected character '#'.
10:17: Unexpected character '`'.
10:30: Unexpected character '`'.
11:13: Unexpected character '`'.
11:25: Unexpected character '`'.
14:12: Unexpected character '`'.
14:25: Unexpected character '`'.
16:19: Char literals must contain exactly one character and close with '''.
17:3: Unexpected character '`'.
17:16: Unexpected character '`'.
17:18: Unexpected character '✓'.
18:3: Unexpected character '`'.
18:11: Unexpected character '`'.
18:13: Unexpected character '✓'.
19:3: Unexpected character '`'.
19:14: Unexpected character '`'.
19:16: Unexpected character '✓'.
20:3: Unexpected character '`'.
20:11: Unexpected character '`'.
20:24: Unexpected character '`'.
20:28: Unexpected character '`'.
20:35: Unexpected character '✓'.
21:3: Unexpected character '`'.
21:15: Unexpected character '`'.
21:17: Unexpected character '✓'.
22:3: Unexpected character '`'.
22:14: Unexpected character '`'.
22:16: Unexpected character '✓'.
24:2: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 731 OUT397 |
| × | IsSortedWithLimitedDuplicatesModel checks that a list is non-decreasing and no value occurs more than twice. Provenance: HumanEval/126 (entry point is_sorted). Level: Medium — two coupled checks (order plus a per-value occurrence count) with empty/single-element edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-single: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 752 OUT860 |
| × | LargestDecreasingIndexModel returns the largest index whose element is smaller than its predecessor, or -1 if none. Provenance: HumanEval/135 (entry point can_arrange). Level: Easy — a single pass keeping the last decreasing index; -1 for empty and single-element lists.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-middle: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found 'in'.
Values:
run.Compiled = false |
Failed | IN8 742 OUT379 |
| ✓ | MinChangesToPalindromeModel finds the minimum number of element changes needed to make a list a palindrome. Provenance: HumanEval/73 (entry point smallest_change). Level: Medium — the spec no longer names the pair-counting trick, so the model must derive that only mismatched symmetric pairs cost a change (the middle element of odd-length lists is free) and index them without off-by-one.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 732 OUT337 |
| ✓ | PluckSmallestEvenNodeModel returns the smallest even value together with its first index as [value, index], or an empty list when there is none. Provenance: HumanEval/68 (entry point pluck). Level: Medium — track a running minimum plus its index, break ties toward the earliest index, and handle empty / no-even inputs; a few interacting edge cases beyond a plain scan.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 748 OUT681 |
| × | RemoveNonUniqueElementsModel removes every value that appears more than once, keeping only values that occur exactly once. Provenance: HumanEval/26 (entry point remove_duplicates).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 738 OUT724 |
| × | RescaleToUnitIntervalModel linearly rescales a list of doubles so the smallest becomes 0 and the largest becomes 1. Provenance: HumanEval/21 (entry point rescale_to_unit).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-up: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:38: Unexpected character '#'.
1:70: Unexpected character '#'.
1:160: Unexpected character '#'.
13:101: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 746 OUT677 |
| × | SecondSmallestDistinctValueModel returns the second smallest DISTINCT element of a list, or null when there is none. Provenance: HumanEval/90 (entry point next_smallest; canonical dedups via set before taking index 1). Level: Medium — dedup + sort + take second, with the null cases (empty, all-equal) and the distinct-value subtlety that flips [1,1,1,1,0] from 1-vs-something.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-sorted: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:74: Unexpected character '`'.
7:2: Char literals must contain exactly one character and close with '''.
12:24: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 725 OUT251 |
| × | SortArrayByEndpointParityModel sorts a copy of a non-negative integer list ascending or descending depending on the parity of the sum of its first and last elements, without mutating the input. Provenance: HumanEval/88 (entry point sort_array). Level: Medium — compose a copy, an endpoint-parity decision that selects the sort direction, and a sort, plus the empty/single edge cases; more than a plain sort.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 759 OUT212 |
| × | SortedUniqueIntegersModel returns the distinct elements of a list sorted ascending. Provenance: HumanEval/34 (entry point unique). Level: Easy — deduplicate then sort; two straightforward operations, one empty-list edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 712 OUT711 |
| ✓ | SortNumberWordsModel sorts number words from zero to nine into ascending numeric order. Provenance: HumanEval/19 (entry point sort_numbers).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 750 OUT1 521 |
| × | SortValuesAtEvenIndicesModel sorts the values at even indices, leaving odd indices in place. Provenance: HumanEval/37 (entry point sort_even). Level: Medium — collect even-index values, sort them, then scatter them back; index bookkeeping across two passes.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-small: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 739 OUT574 |
| × | SortValuesAtIndicesDivisibleByThreeModel sorts the values that sit at indices divisible by three, leaving the other positions in place. Provenance: HumanEval/33 (entry point sort_third). Level: Medium — collect every third value, sort them, then scatter them back by index; a few steps with index bookkeeping.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-small: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 740 OUT880 |
| × | StrangeSortAlternatingMinMaxModel reorders a list by alternately taking the smallest and largest of the remaining values. Provenance: HumanEval/70 (entry point strange_sort_list). Level: Medium — recognise the sort-then-two-pointer pattern, keep state while alternating min/max, and preserve duplicates and odd-length handling.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-even-count: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected '<' after 'List', found end of file.
Values:
run.Compiled = false |
Failed | IN8 749 OUT799 |
Code Generation - String Algorithms
9 passed7 failed |
56.3% |
IN140 066 OUT17 331 | |
| ✓ | CountSentencesStartingWithIModel counts sentences that start with the standalone word "I". Sentences are delimited by '.', '?' or '!'. Provenance: HumanEval/91 (entry point is_bored). The exact "starts with word I" rule (I vs Is/It, standalone I at sentence end) is spelled out to make the count deterministic. Level: Hard — scan with sentence-boundary state, skip leading spaces, detect a standalone leading "I" while rejecting distractors like Is/It and mid-sentence I, across several subtle edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 790 OUT2 382 |
| × | DeleteCharsThenCheckPalindromeModel deletes characters that appear in a removal set and reports whether the remainder is a palindrome. Provenance: HumanEval/112 (entry point reverse_delete). SC# returns a result struct instead of Python's tuple. Level: Easy — filter characters, then a straightforward two-pointer palindrome check; the empty string counts as a palindrome.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-abcde: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 761 OUT822 |
| ✓ | DictionaryKeysUniformCaseModel returns true when every key of a non-empty dictionary is lowercase, or every key is uppercase; false for empty, mixed, or titlecase keys. Provenance: HumanEval/95 (entry point check_dict_case). SC# Dictionary<string,string> keys are always strings, so Python's non-string-key case is intentionally not represented. Level: Medium — reduce over the keys checking a uniform-casing property, with the empty, mixed and titlecase edge cases and the "cased letters required" subtlety.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 735 OUT1 802 |
| ✓ | DigitNamesDescendingModel sorts the integers, walks them in descending order, keeps only values 1..9 and replaces each with its English name in title case. Provenance: HumanEval/105 (entry point by_length). Level: Medium — chain sort + descending traversal + range filter + name mapping, with the out-of-range and empty edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 762 OUT919 |
| ✓ | FlipCaseOrReverseNoLettersModel swaps the case of every letter, or reverses the string when it contains no letters. Provenance: HumanEval/161 (entry point solve). Level: Medium — one pass swaps each letter's case while tracking whether any letter was seen, then a second branch reverses the whole string when none was; two distinct output modes plus the empty/no-letter edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 746 OUT1 213 |
| ✓ | IntegerToLowercaseRomanModel converts an integer in 1..1000 to its lowercase Roman numeral, using subtractive forms. Provenance: HumanEval/156 (entry point int_to_mini_roman). Level: Medium — greedy subtraction against the full value/symbol table including all subtractive pairs; one wrong table entry fails a case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 755 OUT697 |
| ✓ | KeepWordsWithPrimeLengthModel keeps only the words whose length is prime, preserving order, and rejoins them. Provenance: HumanEval/143 (entry point words_in_sentence). Level: Medium — split, a per-word primality sub-check, order-preserving filter, and rejoin (~3 composed steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 743 OUT997 |
| ✓ | NormalizeSpaceRunsModel rewrites runs of spaces: one or two become underscores, three or more become a single hyphen. Provenance: HumanEval/140 (entry point fix_spaces). Level: Medium — run-length replacement with a threshold, applied at the start, middle and end of the string.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 793 OUT1 439 |
| × | RemoveVowelsFromTextModel returns the input string with every vowel removed. Provenance: HumanEval/51 (entry point remove_vowels). Level: Easy — single pass filtering characters against a small vowel set.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 729 OUT489 |
| × | SelectWordsByConsonantCountModel selects the words containing exactly n consonants, preserving order. Provenance: HumanEval/117 (entry point select_words). Level: Easy — tokenize on spaces and count consonants per word; a couple of edge cases (empty string, repeated spaces).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-little: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:23: Unexpected character '#'.
1:148: Unexpected character '#'.
1:965: Unexpected character '#'.
22:201: Char literals must contain exactly one character and close with '''.
23:9: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 769 OUT1 141 |
| ✓ | ShortestPalindromeExtensionModel extends a string into the shortest palindrome that begins with it. Provenance: HumanEval/10 (entry point make_palindrome). Level: Medium — requires the insight of finding the longest palindromic suffix and appending the reversed remaining prefix, with edge cases (empty, single char, already a palindrome).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 754 OUT1 719 |
| ✓ | SortCharactersWithinWordsModel sorts the characters within each space-separated word ascending by ASCII value, keeping word order and every space. Provenance: HumanEval/86 (entry point anti_shuffle). Level: Medium — tokenise on spaces while preserving blanks (including consecutive and trailing spaces), sort each word's characters, and rejoin without dropping or collapsing whitespace.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 745 OUT780 |
| × | SplitWordsByCommaOrSpaceModel splits a string into words separated by commas or spaces, dropping empty tokens. Provenance: HumanEval/101 (entry point words_string). Level: Easy — a single-pass tokenizer over two separators with the empty/repeated-separator edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-hi: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
3:15: Unexpected character '#'.
3:348: Unexpected character '#'.
5:519: Char literals must contain exactly one character and close with '''.
5:904: Char literals must contain exactly one character and close with '''.
5:916: Char literals must contain exactly one character and close with '''.
9:19: Unexpected character '#'.
9:152: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 733 OUT794 |
| × | StrongestClassExtensionModel picks the extension whose strength (uppercase letters minus lowercase letters) is greatest, breaking ties by first occurrence, and formats ClassName.Extension. Provenance: HumanEval/153 (entry point Strongest_Extension). Level: Medium — compute a per-extension score, take the argmax with a first-wins tie rule, then format (~3 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-watashi: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 753 OUT781 |
| × | SumUppercaseAsciiCodesModel sums the ASCII codes of the uppercase English letters in a string, ignoring all other characters. Provenance: HumanEval/66 (entry point digitSum). Level: Easy — a single pass with one character-range condition and an accumulator.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:158: Unexpected character '#'.
1:212: Unexpected character '#'.
1:834: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 729 OUT330 |
| × | ValidateFileNameModel validates a file name against a set of naming rules. Provenance: HumanEval/141 (entry point file_name_check). Level: Medium — five independent rules (digit count, exactly one dot, non-empty stem, leading latin letter, allowed extension) with several distinct edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-valid: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:4: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 769 OUT1 026 |
Code Generation - Strings & Text
1 passed18 failed |
5.3% |
IN165 946 OUT9 741 | |
| × | AllPrefixesModel returns all non-empty prefixes from shortest to longest. Provenance: HumanEval/14 (entry point all_prefixes).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:81: Unexpected character '`'.
8:9: Unexpected character '#'.
9:3: Unexpected character '`'.
9:33: Unexpected character '`'.
13:41: Unexpected character '#'.
16:7: Unexpected character '`'.
16:20: Unexpected character '`'.
17:7: Unexpected character '`'.
17:28: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 712 OUT233 |
| × | CanConcatenateToBalancedModel checks whether two parenthesis strings can be concatenated in some order to form a balanced string. Provenance: HumanEval/119 (entry point match_parens).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-yes: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:22: Unexpected character '`'.
1:34: Unexpected character '`'.
1:50: Unexpected character '#'.
4:10: Unexpected character '`'.
4:30: Unexpected character '`'.
13:2: Char literals must contain exactly one character and close with '''.
15:19: Char literals must contain exactly one character and close with '''.
16:3: Unexpected character '`'.
16:10: Unexpected character '`'.
16:31: Unexpected character '`'.
16:33: Unexpected character '`'.
17:3: Unexpected character '`'.
17:11: Unexpected character '`'.
17:34: Unexpected character '`'.
17:39: Unexpected character '`'.
18:3: Unexpected character '`'.
18:8: Unexpected character '`'.
18:26: Unexpected character '`'.
18:29: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 759 OUT362 |
| × | ClosestVowelBetweenConsonantsModel finds the rightmost vowel that lies strictly between two consonants. Provenance: HumanEval/118 (entry point get_closest_vowel).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-yogurt: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:22: Expected '{' to start a block, found '-'.
Values:
run.Compiled = false |
Failed | IN8 762 OUT1 849 |
| × | ConcatenateStringsModel concatenates a list of strings in order. Provenance: HumanEval/28 (entry point concatenate).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 711 OUT193 |
| × | CountDistinctCharactersModel counts distinct characters case-insensitively. Provenance: HumanEval/16 (entry point count_distinct_characters).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:6: Expected '(' to declare a function or '=' to declare a global variable after 'int C', found end of file. Global variables must have an initializer.
Values:
run.Compiled = false |
Failed | IN8 702 OUT329 |
| ✓ | CountOverlappingOccurrencesModel counts overlapping substring occurrences. Provenance: HumanEval/18 (entry point how_many_times).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 707 OUT604 |
| × | CountUppercaseVowelsAtEvenIndicesModel counts uppercase vowels located at even indices of a string. Provenance: HumanEval/98 (entry point count_upper).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-aBCdEf: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found 'in'.
Values:
run.Compiled = false |
Failed | IN8 719 OUT404 |
| × | CountVowelsWithTrailingYModel counts vowels, additionally counting a trailing y as a vowel. Provenance: HumanEval/64 (entry point vowels_count).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-abcde: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:131: Unexpected character '#'.
Values:
run.Compiled = false |
Failed | IN8 746 OUT607 |
| × | FilterStringsByPrefixModel filters strings by prefix, preserving order. Provenance: HumanEval/29 (entry point FilterByPrefix).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:95: Unexpected character '`'.
3:18: Unexpected character '#'.
4:13: Unexpected character '`'.
4:37: Unexpected character '`'.
5:17: Unexpected character '`'.
5:33: Unexpected character '`'.
6:32: Unexpected character '`'.
6:51: Unexpected character '`'.
7:3: Unexpected character '`'.
7:11: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 718 OUT195 |
| × | FlipCaseModel flips ASCII letter case. Provenance: HumanEval/27 (entry point flip_case).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:34: Unexpected character '`'.
1:43: Unexpected character '`'.
1:59: Unexpected character '#'.
3:16: Unexpected character '#'.
4:13: Unexpected character '`'.
4:21: Unexpected character '`'.
5:63: Unexpected character '`'.
5:74: Unexpected character '`'.
5:82: Unexpected character '`'.
5:110: Unexpected character '`'.
6:32: Unexpected character '`'.
6:46: Unexpected character '`'.
7:8: Char literals must contain exactly one character and close with '''.
11:53: Unexpected character '`'.
11:81: Unexpected character '`'.
12:53: Unexpected character '`'.
12:81: Unexpected character '`'.
15:2: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 713 OUT389 |
| × | LastCharIsStandaloneLetterModel reports whether the final character is a standalone single-letter word. Provenance: HumanEval/134 (entry point check_if_last_char_is_a_letter).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-apple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
10:26: Unexpected character '→'.
18:152: Unexpected character '`'.
18:163: Unexpected character '`'.
18:271: Unexpected character '`'.
18:282: Unexpected character '`'.
23:15: Char literals must contain exactly one character and close with '''.
24:15: Char literals must contain exactly one character and close with '''.
24:101: Char literals must contain exactly one character and close with '''.
26:84: Char literals must contain exactly one character and close with '''.
28:120: Char literals must contain exactly one character and close with '''.
28:163: Char literals must contain exactly one character and close with '''.
30:81: Char literals must contain exactly one character and close with '''.
32:53: Char literals must contain exactly one character and close with '''.
34:55: Unexpected character '→'.
36:54: Unexpected character '→'.
40:30: Char literals must contain exactly one character and close with '''.
40:75: Char literals must contain exactly one character and close with '''.
40:87: Char literals must contain exactly one character and close with '''.
40:101: Char literals must contain exactly one character and close with '''.
40:113: Char literals must contain exactly one character and close with '''.
42:23: Char literals must contain exactly one character and close with '''.
42:65: Char literals must contain exactly one character and close with '''.
46:363: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 760 OUT883 |
| × | ListWithSmallerCharTotalModel returns whichever list has the smaller total character count, preferring the first on a tie. Provenance: HumanEval/74 (entry point total_match).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty-both: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:54: Unexpected character '`'.
8:17: Unexpected character '#'.
8:41: Unexpected character '`'.
8:49: Unexpected character '`'.
8:52: Unexpected character '`'.
8:55: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 731 OUT222 |
| × | LongestStringModel returns the first longest string, or null for empty input. Provenance: HumanEval/12 (entry point longest).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:43: Unexpected character '`'.
1:81: Unexpected character '`'.
1:88: Unexpected character '#'.
6:24: Unexpected character '#'.
9:73: Unexpected character '`'.
9:75: Unexpected character '`'.
11:2: Char literals must contain exactly one character and close with '''.
11:66: Unexpected character '`'.
11:77: Unexpected character '`'.
11:101: Unexpected character '`'.
11:108: Unexpected character '`'.
11:140: Unexpected character '`'.
11:148: Unexpected character '`'.
13:94: Unexpected character '`'.
13:107: Unexpected character '`'.
13:159: Unexpected character '`'.
13:167: Unexpected character '`'.
15:16: Unexpected character '`'.
15:34: Unexpected character '`'.
15:57: Unexpected character '`'.
15:83: Unexpected character '`'.
15:115: Unexpected character '`'.
15:122: Unexpected character '`'.
15:173: Unexpected character '`'.
15:181: Unexpected character '`'.
15:216: Unexpected character '`'.
15:224: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 718 OUT507 |
| × | OddDigitCountSentencesModel builds a fixed sentence per digit string where every letter 'i' is replaced by the odd-digit count. Provenance: HumanEval/113 (entry point odd_count).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:9: Unexpected character '`'.
3:52: Unexpected character '`'.
12:25: Unexpected character '→'.
12:45: Unexpected character '→'.
21:16: Unexpected character '→'.
21:35: Unexpected character '→'.
22:33: Unexpected character '→'.
23:23: Unexpected character '→'.
24:11: Unexpected character '→'.
24:23: Unexpected character '→'.
29:7: Unexpected character '→'.
29:13: Unexpected character '→'.
30:14: Unexpected character '→'.
30:23: Char literals must contain exactly one character and close with '''.
38:144: Char literals must contain exactly one character and close with '''.
38:186: Char literals must contain exactly one character and close with '''.
38:313: Char literals must contain exactly one character and close with '''.
38:332: Char literals must contain exactly one character and close with '''.
40:10: Char literals must contain exactly one character and close with '''.
40:66: Char literals must contain exactly one character and close with '''.
40:110: Char literals must contain exactly one character and close with '''.
50:48: Unexpected character '#'.
50:104: Char literals must contain exactly one character and close with '''.
50:147: Char literals must contain exactly one character and close with '''.
50:238: Unexpected character '—'.
50:246: Unexpected character '#'.
50:290: Unexpected character '`'.
50:306: Unexpected character '`'.
50:349: Unexpected character '`'.
50:357: Unexpected character '`'.
50:391: Unexpected character '`'.
50:408: Unexpected character '`'.
50:415: Unexpected character '`'.
50:421: Unexpected character '`'.
52:27: Unexpected character '`'.
52:55: Unexpected character '`'.
52:65: Unexpected character '`'.
52:73: Unexpected character '`'.
60:28: Unexpected character '`'.
60:71: Unexpected character '`'.
64:2: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 869 OUT1 199 |
| × | PalindromeWithinWeightModel returns true when a list is a palindrome and its sum fits a weight limit. Provenance: HumanEval/72 (entry point will_it_fly).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-true: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:151: Unexpected character '#'.
7:79: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 729 OUT423 |
| × | RotateAlphabetCipherModel rotates each lowercase letter forward through the alphabet with wraparound. Provenance: HumanEval/89 (entry point encrypt).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-hi: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 750 OUT256 |
| × | SameDistinctCharactersModel checks whether two strings share exactly the same distinct characters. Provenance: HumanEval/54 (entry point SameChars).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-true-long: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:3: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 715 OUT418 |
| × | SpaceDelimitedSequenceModel builds a space-delimited sequence of integers from 0 through n. Provenance: HumanEval/15 (entry point string_sequence).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-zero: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 717 OUT493 |
| × | StringIsPalindromeModel checks whether a string is a palindrome. Provenance: HumanEval/48 (entry point IsPalindrome).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:5: Expected a name, found end of file.
Values:
run.Compiled = false |
Failed | IN8 708 OUT175 |