Benchmark summary
HumanEval adapted to SC#, adding an extra challenge: models must generate correct code in a custom sandboxed language while following its documentation and constraints.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Code Generation - Algorithms & Combinatorics
11 passed5 failed |
68.8% |
IN140 127 OUT19 256 | |
| ✓ | BalancedParenthesesModel checks whether a string of parentheses is correctly balanced. Provenance: HumanEval/61 (entry point correct_bracketing). Level: Easy — single left-to-right pass tracking a balance that must never go negative and must end at zero.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 710 OUT84 |
| ✓ | CountBucketLoweringsModel counts total bucket lowerings needed to empty rows of wells. Provenance: HumanEval/115 (entry point max_fill). Level: Medium — per-row water sum, ceiling division by capacity, then a total; the ceiling with integer arithmetic is the trap.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 755 OUT117 |
| × | EvaluateAlgebraExpressionModel writes an SC# function evaluating a small arithmetic expression with Python operator precedence. Provenance: HumanEval/160 (entry point do_algebra). Level: Hard — a mini expression evaluator over five operators across three precedence tiers, with the right-associativity of ** as the key trap and floor-division to get right.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: power-times-plus: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
96:19: Unknown variable 'operands'.
99:22: Unknown variable 'operators'.
Values:
run.Compiled = false |
Failed | IN8 862 OUT663 |
| × | FindValueCoordinatesInRaggedMatrixModel returns the coordinates of every cell equal to x in a ragged matrix, in a fixed order. Provenance: HumanEval/87 (entry point get_row). SC# represents each coordinate as a two-item List<int> [row, column]. Level: Medium — collect matches across variable-length rows, then a dual-key ordering (rows ascending, columns descending) that is easy to get backwards.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-multiple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
12:43: Expected '(' after the List type. Lists are created with 'new List<T>()' or 'new List<T>(items)', found '{'.
Values:
run.Compiled = false |
Failed | IN8 755 OUT1 597 |
| ✓ | GreatestFrequentIntegerModel returns the greatest positive integer whose frequency in the list is at least its own value, or -1 if none. Provenance: HumanEval/69 (entry point search). Level: Medium — count occurrences and combine two conditions (value>0 and frequency>=value) while tracking the maximum qualifying value.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 722 OUT662 |
| ✓ | HasZeroSumTripleModel decides whether three elements at distinct positions sum to zero. Provenance: HumanEval/40 (entry point triples_sum_to_zero). Level: Medium — brute-force over index triples, with the trap that distinctness is by position, not by value (repeated equal values at different indices count).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 716 OUT151 |
| ✓ | LargestKValuesSortedAscendingModel returns the k largest values sorted ascending. Provenance: HumanEval/120 (entry point maximum). Level: Medium — sort then take the top k, with duplicates preserved and the k = 0 edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 719 OUT663 |
| × | LexicographicMinGridPathModel writes an SC# function returning the values along the lexicographically smallest length-k grid path. Provenance: HumanEval/129 (entry point minPath). Level: VeryHard — the naive search space is huge; a correct answer needs the non-obvious insight that the optimum bounces between the global minimum 1 and its smallest edge-neighbor. Unique answer, so exact list comparison is deterministic.1 AI query
0 tool callscategory:CodeGenerationlevel:VeryHardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: three-by-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
10:8: Expected a variable name, found '['.
Values:
run.Compiled = false |
Failed | IN8 803 OUT5 708 |
| ✓ | MinimumSubarraySumModel finds the minimum sum over all non-empty contiguous subarrays. Provenance: HumanEval/114 (entry point minSubArraySum). Level: Hard — a Kadane-style running-minimum with state, plus the all-positive trap (the answer is the smallest element, never 0). Uses long because inputs exceed int range.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 726 OUT2 172 |
| ✓ | MostFrequentLettersModel returns the most frequent letter(s) and their count, keeping every tie. Provenance: HumanEval/111 (entry point histogram). Level: Medium — tally counts, find the maximum, then collect all keys reaching it (ties included).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 739 OUT1 000 |
| ✓ | OrderByDigitScoreModel writes an SC# function that stably sorts integers by a signed digit-sum score. Provenance: HumanEval/145 (entry point order_by_points). Level: Hard — combines a subtle scoring rule (leading digit of a negative number counts negative), a stable sort whose tie-break by original position is mandatory for determinism, and empty/negative/tie edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 796 OUT1 588 |
| ✓ | PlanetsBetweenModel writes an SC# function listing the planets between two named planets, ordered from the Sun. Provenance: HumanEval/148 (entry point bf). SC# returns a List<string> instead of a Python tuple. Level: Medium — map names to orbital positions, emit the strictly-between range sun-outward regardless of argument order, and handle invalid-name, equal-name and adjacent (empty) edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 766 OUT385 |
| × | PolynomialRealRootModel finds one real root of a polynomial given its coefficients. Provenance: HumanEval/32 (entry point find_zero). Level: Hard — requires a numeric root-finding method (e.g. bracket a sign change, then bisect); no closed form, and correctness is a tolerance on the residual rather than an exact value.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: linear: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
70:9: Unknown variable 'currentValue'. Declare it first, e.g. 'int currentValue = ...;'.
72:32: Unknown variable 'currentValue'.
72:73: Unknown variable 'currentValue'.
80:17: Unknown variable 'midValue'. Declare it first, e.g. 'int midValue = ...;'.
82:21: Unknown variable 'midValue'.
87:68: Unknown variable 'midValue'.
88:68: Unknown variable 'midValue'.
101:21: Unknown variable 'currentValue'.
Values:
run.Compiled = false |
Failed | IN8 740 OUT2 272 |
| ✓ | SumEvenElementsAtOddIndicesModel sums the even values that sit at odd zero-based indices. Provenance: HumanEval/85 (entry point add). Level: Easy — one stride-2 loop with a parity filter and an accumulator.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 731 OUT87 |
| ✓ | SwapCaseAndShiftVowelsModel swaps letter case and shifts vowels forward by two. Provenance: HumanEval/93 (entry point encode). Level: Medium — per-character case swap composed with a vowel-only alphabet shift, keeping the swapped case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 759 OUT1 796 |
| × | ValidateMmDdYyyyDateModel validates a strict mm-dd-yyyy date string. Provenance: HumanEval/124 (entry point valid_date). Level: Hard — several interacting rules (fixed format, digit counts, dash positions, month range, per-month day range, no leap year) where any single miss flips the answer.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: valid-march: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:14: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared.
Values:
run.Compiled = false |
Failed | IN8 828 OUT311 |
Code Generation - Arithmetic & Predicates
12 passed2 failed |
85.7% |
IN122 199 OUT4 790 | |
| ✓ | AbsoluteScoreDifferencesModel returns the element-wise absolute difference between two equal-length integer lists. Provenance: HumanEval/152 (entry point compare). Level: Easy — a single element-wise map (absolute difference) over paired lists, no edge cases beyond equal length.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 707 OUT329 |
| ✓ | GpaToLetterGradesModel maps a list of numeric GPAs to letter grades using a fixed threshold table. Provenance: HumanEval/81 (entry point numerical_letter_grade). Level: Medium — 13 ordered thresholds with strict "greater-than" boundaries plus the exact 4.0 == A+ special case; boundary values are easy to misclassify.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 827 OUT316 |
| ✓ | IsListMonotonicModel decides whether a list is monotonically non-decreasing or non-increasing. Provenance: HumanEval/57 (entry point monotonic). Level: Easy — single pass tracking two direction flags; equal adjacent values allowed.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 697 OUT130 |
| ✓ | LargerOfTwoNumericStringsModel compares two numeric strings (decimal separator '.' or ',') and returns the larger one unchanged, or empty if equal. Provenance: HumanEval/137 (entry point compare_one). SC# uses string inputs; equal values return an empty string instead of Python None. Level: Medium — normalize separators, parse and compare numerically, then return the original string (preserving commas) or empty on a tie.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 728 OUT649 |
| ✓ | MeanAbsoluteDeviationModel computes the mean absolute deviation of a list of doubles around their mean. Provenance: HumanEval/4 (entry point mean_absolute_deviation).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 709 OUT628 |
| ✓ | MedianOfIntegerListModel returns the median of a list of integers, averaging the two middle values for even length. Provenance: HumanEval/47 (entry point median). Level: Medium — sort then branch on parity, returning a possibly fractional average.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 697 OUT127 |
| × | NearestNegativeAndPositiveModel finds the negative closest to zero and the smallest positive, with presence flags. Provenance: HumanEval/136 (entry point largest_smallest_integers). SC# returns a struct instead of Python's tuple with None. Level: Medium — two independent scans plus presence flags, excluding zero and handling both-absent cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-positive-only: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
5:43: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.
Values:
run.Compiled = false |
Failed | IN8 800 OUT322 |
| × | PolynomialDerivativeCoefficientsModel returns the coefficients of a polynomial's derivative in the same coefficient form. Provenance: HumanEval/62 (entry point derivative). Level: Easy — single loop multiplying each coefficient by its index; constant gives an empty list.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '{'.
Values:
run.Compiled = false |
Failed | IN8 718 OUT835 |
| ✓ | SortEvenLengthStringsModel keeps only even-length strings and returns them sorted by length then alphabetically, without mutating the input. Provenance: HumanEval/149 (entry point sorted_list_sum). Level: Medium — filter by parity plus a two-key (length, then lexicographic) ordering; several tie cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 707 OUT901 |
| ✓ | SumOddValuesAtEvenIndexesModel sums the odd values located at even zero-based indexes. Provenance: HumanEval/121 (entry point solution). Level: Easy — a single strided loop with an odd-value guard; returns 0 when nothing matches.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 713 OUT92 |
| ✓ | SumOfSquaredCeilingsModel rounds each number up with ceiling, squares it, and sums the results. Provenance: HumanEval/133 (entry point sum_squares). Level: Easy — map ceiling then square then accumulate; handles negatives (ceiling of -2.4 is -2) and the double-to-int result.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 754 OUT72 |
| ✓ | SumSquaresOfPositiveOddIntegersModel sums the squares of every value that is a positive odd integer, ignoring negatives, evens and non-integers. Provenance: HumanEval/151 (entry point double_the_difference). Level: Medium — parity plus an integrality guard over doubles (the non-integer filter is the trap), then sum of squares.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 702 OUT113 |
| ✓ | SumWithIndexBasedPowersModel transforms each list entry by its index (square when index is a multiple of 3, cube when a multiple of 4 but not 3, unchanged otherwise) and sums the results. Provenance: HumanEval/142 (entry point sum_squares). Level: Medium — three-way index-based branch whose precedence (multiple-of-3 wins over multiple-of-4) is a genuine trap; empty-list edge case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 740 OUT140 |
| ✓ | WordWithMostUniqueCharsModel returns the word with the most distinct characters, breaking ties by lexicographically smallest. Provenance: HumanEval/158 (entry point find_max). Level: Medium — count distinct characters per word, take the argmax, and break ties by lexicographic order (~3 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 700 OUT136 |
Code Generation - Easy Fundamentals
10 passed |
100.0% |
IN86 886 OUT3 114 | |
| ✓ | AddTwoIntegersModel implements integer addition. Provenance: HumanEval/53 (entry point Add).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 669 OUT20 |
| ✓ | BalanceEverGoesNegativeModel detects whether a running balance ever goes negative. Provenance: HumanEval/3 (entry point BelowZero).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 692 OUT61 |
| ✓ | BinaryStringXorModel implements binary XOR on two equal-length bit strings. Provenance: HumanEval/11 (entry point StringXor).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 717 OUT92 |
| ✓ | FilterStringsBySubstringModel filters strings by substring membership. Provenance: HumanEval/7 (entry point FilterBySubstring).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 682 OUT257 |
| ✓ | FractionalPartModel returns the fractional part of a positive number. Provenance: HumanEval/2 (entry point TruncateNumber).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 686 OUT21 |
| ✓ | HasCloseElementsModel detects whether any two elements are closer than a threshold. Provenance: HumanEval/0 (entry point HasCloseElements).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 688 OUT113 |
| ✓ | InsertDelimiterModel inserts a delimiter between consecutive elements. Provenance: HumanEval/5 (entry point Intersperse).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 696 OUT640 |
| ✓ | StringLengthModel implements string length. Provenance: HumanEval/23 (entry point GetStringLength).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 667 OUT745 |
| ✓ | SumAndProductModel returns the sum and product of a list of integers. Provenance: HumanEval/8 (entry point SumProduct).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 713 OUT1 106 |
| ✓ | SumFromOneToNModel implements the sum 1..n inclusive. Provenance: HumanEval/60 (entry point SumToN).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 676 OUT59 |
Code Generation - Lists & Sequences
10 passed |
100.0% |
IN87 016 OUT4 961 | |
| ✓ | AllValuesBelowThresholdModel checks that every value is strictly below a threshold. Provenance: HumanEval/52 (entry point BelowThreshold).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 692 OUT49 |
| ✓ | CanMakeFirstListAllEvenModel decides whether unlimited swaps between two lists can make the first list all even. Provenance: HumanEval/110 (entry point exchange).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 716 OUT1 579 |
| ✓ | CarrotsEatenAndRemainingModel computes total carrots eaten and carrots left after a meal, capping consumption at the remaining stock. Provenance: HumanEval/159 (entry point eat).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 745 OUT855 |
| ✓ | CountEvenAndOddDigitsModel counts the even and odd digits of a number (using its absolute value). Provenance: HumanEval/155 (entry point even_odd_count).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 715 OUT795 |
| ✓ | IncrementEachElementModel returns a new list with every element incremented by one. Provenance: HumanEval/42 (entry point IncrList).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 692 OUT265 |
| ✓ | KeepPositiveNumbersModel keeps only positive integers, preserving order. Provenance: HumanEval/30 (entry point GetPositive).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 690 OUT59 |
| ✓ | MaximumListElementModel returns the maximum element of a non-empty list. Provenance: HumanEval/35 (entry point MaxElement).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 675 OUT498 |
| ✓ | PairSummingToZeroModel detects two distinct positions summing to zero. Provenance: HumanEval/43 (entry point PairsSumToZero).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 697 OUT71 |
| ✓ | RollingMaximumModel implements a rolling (running) maximum over a list. Provenance: HumanEval/9 (entry point rolling_max).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 694 OUT675 |
| ✓ | SortedCommonElementsModel returns the ascending-sorted list of values common to two integer lists, without duplicates. Provenance: HumanEval/58 (entry point common).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 700 OUT115 |
Code Generation - Number Theory
9 passed5 failed |
64.3% |
IN122 149 OUT13 098 | |
| ✓ | BuildStonePileModel builds a pile of n levels: the first level has n stones and each next level has 2 more. Provenance: HumanEval/100 (entry point make_a_pile). Level: Easy — a single accumulation loop (n + 2*i); no real edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 740 OUT81 |
| ✓ | CountOddBoundaryDigitsModel counts values greater than 10 whose first and last decimal digits are both odd. Provenance: HumanEval/146 (entry point specialFilter). Level: Medium — per element: a magnitude guard, extracting the leading digit (its own loop), extracting the last digit, and an odd-digit test (~3-4 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 717 OUT163 |
| ✓ | CountPositiveSignedDigitSumsModel counts the integers whose signed digit sum is greater than 0, where a negative number's most significant digit carries the minus sign (-123 -> -1 + 2 + 3 = 4). Provenance: HumanEval/108 (entry point count_nums). Level: Medium — extract the most significant digit, apply the sign only to it, sum the digits, and count positives, with the negative and zero edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 764 OUT2 196 |
| ✓ | CountSevensInMultiplesOf11Or13Model counts occurrences of the digit 7 across the multiples of 11 or 13 below n. Provenance: HumanEval/36 (entry point fizz_buzz). Level: Medium — iterate a range, filter by a divisibility union without double counting multiples of both, and count a digit.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 696 OUT1 162 |
| × | DigitSumInBinaryModel sums the decimal digits of an integer and returns that sum in binary. Provenance: HumanEval/84 (entry point solve). Level: Medium — composes two sub-steps (decimal digit sum, then base-2 conversion) with a zero edge case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-1000: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
23:7: Type 'StringBuilder' has no supported method named 'Reverse' with 0 argument(s). Supported methods: Append, AppendLine, Clear, ToString.
Values:
run.Compiled = false |
Failed | IN8 751 OUT156 |
| ✓ | EveryThreeConsecutiveDistinctModel checks whether every window of three consecutive characters in a string is made of distinct characters. Provenance: HumanEval/80 (entry point is_happy). Level: Easy — a single sliding window with a length guard and a pairwise-distinct check.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 712 OUT1 014 |
| × | FactorialOrTriangularByParityModel returns a list of size n where, for 1-based index i, even i holds i! and odd i holds 1 + 2 + ... + i. Provenance: HumanEval/106 (entry point f). Level: Medium — a loop with a parity branch selecting between two sub-computations (factorial vs triangular sum), plus the 1-based-index twist.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.ResultJson == expectedJson
Context: baseline-five: expected [1,2,6,24,15], got [1,2,4,8,9]
Values:
run.ResultJson = "[1,2,4,8,9]"
expectedJson = "[1,2,6,24,15]" |
Failed | IN8 741 OUT1 782 |
| ✓ | FractionProductIsWholeModel decides whether the product of two positive fractions is a whole number. Provenance: HumanEval/144 (entry point simplify). Level: Medium — parse two "num/den" strings into four integers, cross-multiply, then test divisibility (~3 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 699 OUT467 |
| ✓ | IsPrimeNumberModel tests whether an integer is a prime number. Provenance: HumanEval/31 (entry point is_prime). Level: Easy — a single well-known trial-division loop with an early exit and one edge (values below 2).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 683 OUT115 |
| ✓ | IsSimplePowerOfNModel decides whether one integer is an exact power of another. Provenance: HumanEval/76 (entry point is_simple_power). Level: Medium — the model must handle the n==1 special case and the exponent-0 case (x==1) that a naive multiply loop misses; the spec no longer hands over the loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 768 OUT89 |
| × | NumbersWithAllOddDigitsModel returns, sorted ascending, the values whose decimal digits are all odd. Provenance: HumanEval/104 (entry point unique_digits). Level: Medium — filter by an all-odd-digits predicate (per-value digit decomposition) then sort.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-mixed: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '{'.
Values:
run.Compiled = false |
Failed | IN8 725 OUT1 240 |
| ✓ | PrimesBelowNModel returns all prime numbers less than n in ascending order (empty when n <= 2). Provenance: HumanEval/96 (entry point count_up_to). Level: Medium — compose a primality test (inner loop) with a collection loop and the n<=2 edge; more than a single simple loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 692 OUT720 |
| × | RangeAverageInBinaryModel computes the average of the inclusive integer range n..m, rounds it to the nearest integer, and returns that value in binary with a "0b" prefix; returns "-1" when n > m. Provenance: HumanEval/103 (entry point rounded_avg). SC# uses a string return type, so the invalid range yields "-1" instead of Python's integer -1. Determinism note: every input uses an even (n + m) so the mean is an exact integer and no rounding tie occurs (the original's tie cases depend on Python banker's rounding, which is not portable to SC# Math.Round). Level: Medium — sum/average a range, then a base-conversion loop to binary, plus the invalid edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-one-five: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
9:18: Unknown variable 'Convert'.
Values:
run.Compiled = false |
Failed | IN8 724 OUT1 541 |
| × | SortByBinaryOneBitsModel sorts integers by their binary one-bit count, breaking ties by numeric value. Provenance: HumanEval/116 (entry point sort_array). Level: Medium — combine a bit-count computation with a stable two-key ordering; negatives count bits of the absolute value to match the benchmark.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-positive: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
4:27: Expected ';' after the variable declaration, found 'Math'.
Values:
run.Compiled = false |
Failed | IN8 737 OUT2 372 |
Code Generation - Numbers & Math
15 passed4 failed |
78.9% |
IN165 342 OUT8 036 | |
| ✓ | BiggestEvenInRangeModel returns the largest even integer in an inclusive range, or -1 when none exists. Provenance: HumanEval/102 (entry point choose_num).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 697 OUT262 |
| ✓ | CarRaceCollisionsModel computes the number of head-on car collisions. Provenance: HumanEval/41 (entry point CarRaceCollision).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 713 OUT19 |
| × | CountMangoesInBasketModel computes the count of mango fruits given a "N apples and M oranges" string and a fruit total. Provenance: HumanEval/67 (entry point fruit_distribution).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-5-6-19: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
7:30: Type 'List<string>' has no fields. Only struct values support '.Length'.
Values:
run.Compiled = false |
Failed | IN8 715 OUT158 |
| ✓ | CountNumbersStartingOrEndingWithOneModel counts n-digit positive integers that start or end with the digit 1. Provenance: HumanEval/83 (entry point starts_one_ends).1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 700 OUT1 348 |
| ✓ | EvenDigitsInRangeModel returns the even single digits lying between two bounds, in ascending order. Provenance: HumanEval/163 (entry point generate_integers).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 735 OUT901 |
| ✓ | GreatestCommonDivisorModel implements the greatest common divisor of two integers. Provenance: HumanEval/13 (entry point greatest_common_divisor). Level: Medium — Euclid's algorithm keeps state across iterations (repeated modulo + swap until the remainder is 0), beyond a single expression or plain accumulator loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 678 OUT729 |
| ✓ | HasPrimeLengthModel returns true when a string's length is a prime number. Provenance: HumanEval/82 (entry point prime_length).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 680 OUT119 |
| × | IsPerfectCubeModel detects whether an integer is a perfect cube, including negatives and zero. Provenance: HumanEval/77 (entry point iscube).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.ResultJson == expectedJson
Context: baseline-negative-one: expected true, got false
Values:
run.ResultJson = "false"
expectedJson = "true" |
Failed | IN8 686 OUT163 |
| × | IsRightAngleTriangleModel determines whether three side lengths form a right-angled triangle. Provenance: HumanEval/157 (entry point right_angle_triangle).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-3-4-5: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
5:32: Expected '(' after the List type. Lists are created with 'new List<T>()' or 'new List<T>(items)', found '{'.
Values:
run.Compiled = false |
Failed | IN8 702 OUT114 |
| × | IsSumOfFourPositiveEvensModel decides whether n is expressible as a sum of exactly four positive even numbers. Provenance: HumanEval/138 (entry point is_equal_to_sum_even).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-4: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
5:1: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 685 OUT1 055 |
| ✓ | LargestProperDivisorModel returns the largest proper divisor. Provenance: HumanEval/24 (entry point largest_divisor).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 676 OUT89 |
| ✓ | ProductOfOddDigitsModel returns the product of the odd decimal digits, or 0 when every digit is even. Provenance: HumanEval/131 (entry point digits).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 687 OUT118 |
| ✓ | ProductOfUnitDigitsModel returns the product of the unit digits of two integers, treating them as non-negative. Provenance: HumanEval/97 (entry point multiply).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 731 OUT1 620 |
| ✓ | RoundToClosestIntegerModel parses a numeric string and rounds it to the closest integer, breaking .5 ties away from zero. Provenance: HumanEval/99 (entry point closest_integer).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 716 OUT233 |
| ✓ | SelectValueByPrimalityModel returns x when n is prime and y otherwise. Provenance: HumanEval/150 (entry point x_or_y).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 691 OUT83 |
| ✓ | SumMagnitudesTimesSignProductModel multiplies the sum of magnitudes by the product of the signs (any zero makes the product zero); empty input returns null. Provenance: HumanEval/128 (entry point prod_signs).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 721 OUT136 |
| ✓ | SumSmallElementsFromPrefixModel sums the values among the first k entries whose absolute value has at most two digits. Provenance: HumanEval/122 (entry point add_elements).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 711 OUT735 |
| ✓ | TriangleAreaFromBaseAndHeightModel returns the area of a triangle from base and height. Provenance: HumanEval/45 (entry point TriangleArea).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 689 OUT26 |
| ✓ | TriangleAreaRoundedModel computes Heron's triangle area rounded to two decimals, or -1 for degenerate side lengths. Provenance: HumanEval/71 (entry point triangle_area).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 729 OUT128 |
Code Generation - Parsing & Decoding
9 passed5 failed |
64.3% |
IN122 263 OUT18 725 | |
| ✓ | CircularShiftDigitsModel circularly shifts the decimal digits of an integer to the right, returning the result as a string. Provenance: HumanEval/65 (entry point circular_shift). Level: Medium — convert to digits, branch on shift vs digit count (reverse when it overflows), rotate by the right amount, and preserve leading zeros the rotation creates; several interacting edge cases and an easy direction/off-by-one trap.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 737 OUT2 010 |
| × | ConvertIntegerToSmallBaseModel converts a positive integer to a base below ten and returns its string representation. Provenance: HumanEval/44 (entry point change_base). Level: Medium — repeated division/modulo building the digits in the correct order; a small but genuine algorithm.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-8-base-3: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
15:20: Type 'StringBuilder' has no fields. Only struct values support '.Length'.
17:23: Type 'StringBuilder' cannot be indexed with [...]. Only List, Dictionary and string values support indexing.
Values:
run.Compiled = false |
Failed | IN8 708 OUT1 380 |
| ✓ | CountPrimeHexDigitsModel counts how many hexadecimal digits of a string have prime value. Provenance: HumanEval/78 (entry point hex_key). Level: Easy — a single pass counting membership in a fixed six-element set (the set is part of the original spec, not a hint).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 728 OUT75 |
| × | DecimalToBinaryWithDelimitersModel converts a non-negative integer to its binary digits and wraps the result with "db" markers. Provenance: HumanEval/79 (entry point decimal_to_binary). Level: Easy — a standard repeated-division binary conversion (single loop) with the zero edge case and a fixed prefix/suffix wrap.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.ResultJson == expectedJson
Context: baseline-103: expected "db1100111db", got "db1111100db"
Values:
run.ResultJson = "\"db1111100db\""
expectedJson = "\"db1100111db\"" |
Failed | IN8 738 OUT138 |
| × | DecodeCaesarShiftByFiveModel decodes lowercase text that was Caesar-shifted forward by five, by shifting each letter back five with wraparound. Provenance: HumanEval/50 (entry point decode_shift). Level: Easy — single pass of modular character arithmetic with one wraparound edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-hello: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:79: Unexpected character '#'.
14:5: Char literals must contain exactly one character and close with '''.
22:2: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 710 OUT218 |
| ✓ | DecodeCyclicGroupsOfThreeModel decodes text whose full three-character groups were rotated left, by rotating each such group right. Provenance: HumanEval/38 (entry point decode_cyclic). Level: Medium — process the string in groups of three, rotate each full group, and leave a short trailing group unchanged.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 725 OUT1 407 |
| ✓ | HasNestedBracketSubsequenceModel detects whether a bracket string has a subsequence containing a nested bracket pair. Provenance: HumanEval/132 (entry point is_nested). Level: Hard — subsequence nesting is subtly different from balanced brackets; naive stack or balance checks give wrong answers on many cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 803 OUT10 021 |
| × | MaxNestingDepthPerGroupModel reports the maximum nesting depth of each parenthesis group. Provenance: HumanEval/6 (entry point parse_nested_parens). Level: Medium — per-group depth tracking with a running maximum, reset at each group boundary; composed loop with state, beyond a single expression.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-multiple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '{'.
Values:
run.Compiled = false |
Failed | IN8 713 OUT980 |
| × | ParseMusicBeatsModel parses ASCII note tokens into beat counts. Provenance: HumanEval/17 (entry point parse_music). Level: Easy — split on spaces and map each token through a fixed three-way legend; no state, single empty-input edge. Below the old "Medium" bucket.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
5:21: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared.
Values:
run.Compiled = false |
Failed | IN8 713 OUT159 |
| ✓ | RotationSubstringCheckModel reports whether the second word or any rotation of it occurs as a contiguous substring of the first. Provenance: HumanEval/154 (entry point cycpattern_check). Level: Medium — the "any rotation" requirement is a real trap; a nested scan over all substrings and all rotations (b+b trick) is needed.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 695 OUT122 |
| ✓ | SeparateBalancedParenGroupsModel separates a string into its top-level balanced parenthesis groups. Provenance: HumanEval/1 (entry point separate_paren_groups). Level: Medium — a depth-counter state machine that accumulates characters and emits a group each time depth returns to zero, plus the space-ignoring rule; a couple of composed steps beyond a plain loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 755 OUT1 363 |
| ✓ | SplitWordsOrCountLettersModel splits text on spaces, else on commas, else counts a subset of its lowercase letters, returning a discriminated result struct. Provenance: HumanEval/125 (entry point split_words). Level: Medium — three mutually exclusive branches, an empty-part filter, and a parity-based letter count, all packed into a typed result struct that emulates Python's list-or-int return.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 829 OUT315 |
| ✓ | StringToMd5DigestModel returns the lowercase hexadecimal MD5 digest of text, or null when text is empty. Provenance: HumanEval/162 (entry point string_to_md5). Level: Easy — a single guarded call to the built-in Md5 over the text's UTF-8 bytes, plus the empty-input null guard; the challenge is finding and using the library primitive, not the algorithm.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 695 OUT449 |
| ✓ | ValidateAngleBracketNestingModel validates that a string of angle brackets is correctly nested. Provenance: HumanEval/56 (entry point correct_bracketing). Level: Medium — running depth counter with two interacting failure modes (never negative, and ends balanced).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 714 OUT88 |
Code Generation - Recursion & Sequences
13 passed2 failed |
86.7% |
IN130 949 OUT19 619 | |
| ✓ | BrazilianFactorialModel writes an SC# function returning the Brazilian factorial n! * (n-1)! * ... * 1!. Provenance: HumanEval/139 (entry point special_factorial). Level: Medium — a running-factorial product loop (~3 steps); the main trap is that the result overflows int at n=7, so it must be computed and returned as long.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 748 OUT2 656 |
| × | CountEvenAndOddPalindromesModel counts even and odd integer palindromes from 1 to n. Provenance: HumanEval/107 (entry point even_odd_palindrome). SC# returns the tuple as a two-item List<int> [evenCount, oddCount]. Level: Medium — palindrome detection over a range plus two parity-partitioned counters.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: one-two-three: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '{'.
Values:
run.Compiled = false |
Failed | IN8 739 OUT988 |
| ✓ | CountTriplesDivisibleByThreeModel writes an SC# function counting index triples whose mapped values sum to a multiple of 3. Provenance: HumanEval/147 (entry point get_max_triples). Level: Medium — build a[i] = i*i - i + 1 over a 1-based range, then count triples i<j<k with sum divisible by 3; a straightforward triple loop suffices for the tested sizes (up to n=100).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 736 OUT3 443 |
| ✓ | Fib4SequenceModel computes the four-step Fibonacci-like sequence iteratively. Provenance: HumanEval/46 (entry point fib4). Level: Medium — maintain a sliding window of four previous terms with four specific base cases; recursion is disallowed for efficiency.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 743 OUT190 |
| ✓ | FibFibSequenceModel computes the three-step Fibonacci-like (tribonacci) sequence iteratively. Provenance: HumanEval/63 (entry point fibfib). Level: Medium — maintain a sliding window of three previous terms with three base cases; compute efficiently rather than by naive recursion.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 731 OUT155 |
| ✓ | IntervalIntersectionLengthIsPrimeModel reports whether the length of two intervals' intersection is prime. Provenance: HumanEval/127 (entry point intersection). SC# represents each interval as a two-item List<int> [start, end]. Level: Medium — compute the overlap, apply the (end - start) length convention, then test primality.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 781 OUT964 |
| ✓ | LargestPrimeFactorModel returns the largest prime factor of n. Provenance: HumanEval/59 (entry point largest_prime_factor). Level: Medium — trial division with state, dividing each factor out fully before advancing, and handling the final remaining prime.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 679 OUT115 |
| ✓ | NthFibonacciModel returns the nth Fibonacci number. Provenance: HumanEval/55 (entry point fib). Level: Easy — the classic two-variable accumulator loop with base cases 0 and 1.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 695 OUT573 |
| ✓ | NthPrimeFibonacciModel returns the nth Fibonacci number that is also prime, 1-based. Provenance: HumanEval/39 (entry point prime_fib). Level: Medium — compose iterative Fibonacci generation with a primality test and a running count until the nth prime Fibonacci is reached (all tested values stay within int).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 705 OUT3 117 |
| ✓ | OddNumbersInCollatzSequenceModel returns the odd values of a Collatz sequence, sorted ascending. Provenance: HumanEval/123 (entry point get_odd_collatz). Level: Medium — generate the state-driven sequence to 1, filter odds (including 1), then sort.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 747 OUT2 397 |
| ✓ | PowerOfTwoModuloModel computes 2 to the power n modulo p. Provenance: HumanEval/49 (entry point modp). Level: Medium — the trap is numeric overflow: with n up to ~1100, 2^n must be reduced modulo p each step rather than computed then reduced.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 715 OUT956 |
| ✓ | PrimeFactorizationModel implements integer prime factorization, listing factors from smallest to largest with each factor repeated as many times as it divides the input. Provenance: HumanEval/25 (entry point factorize). Level: Medium — trial division with running state (divide the factor out and re-test it), emitting repeats in a defined ascending order; more than a single expression or plain accumulator loop.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 732 OUT674 |
| ✓ | ProductOfThreePrimesModel decides whether a number is the product of exactly three prime numbers (with multiplicity). Provenance: HumanEval/75 (entry point is_multiply_prime). Level: Medium — factor the number and check that the total count of prime factors, counting repeats, equals exactly three.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 721 OUT156 |
| × | SumDigitsOfLargestPrimeModel returns the digit sum of the largest prime in a list. Provenance: HumanEval/94 (entry point skjkasdkd). Level: Medium — primality test, tracking the running maximum prime, then summing its digits.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-181: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '{'.
Values:
run.Compiled = false |
Failed | IN8 709 OUT1 308 |
| ✓ | TribonacciLikeSequenceModel writes an SC# function returning the first n+1 values of a Tribonacci-like sequence. Provenance: HumanEval/130 (entry point tri). Level: Hard — the odd-index rule references tri(n+1), a forward reference that would loop under naive recursion; the solver must realize the referenced term is even-indexed and directly computable, then build the list iteratively.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 768 OUT1 927 |
Code Generation - Sequences & Sorting
12 passed3 failed |
80.0% |
IN130 712 OUT11 256 | |
| ✓ | CanSortByRightShiftsModel decides whether a list can be sorted ascending using only whole-array right shifts (rotations). Provenance: HumanEval/109 (entry point move_one_ball). Level: Medium — needs the insight that a rotation-sortable list has at most one circular descent; a couple of edge cases (empty, already sorted).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 741 OUT256 |
| ✓ | FindClosestPairModel finds the two closest values in a list and returns them in ascending order. Provenance: HumanEval/20 (entry point find_closest_elements). Level: Medium — an O(n^2) scan over all pairs tracking the minimal distance, plus ordering the winning pair.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 703 OUT188 |
| ✓ | IsSortedWithLimitedDuplicatesModel checks that a list is non-decreasing and no value occurs more than twice. Provenance: HumanEval/126 (entry point is_sorted). Level: Medium — two coupled checks (order plus a per-value occurrence count) with empty/single-element edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 724 OUT1 821 |
| ✓ | LargestDecreasingIndexModel returns the largest index whose element is smaller than its predecessor, or -1 if none. Provenance: HumanEval/135 (entry point can_arrange). Level: Easy — a single pass keeping the last decreasing index; -1 for empty and single-element lists.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 714 OUT87 |
| ✓ | MinChangesToPalindromeModel finds the minimum number of element changes needed to make a list a palindrome. Provenance: HumanEval/73 (entry point smallest_change). Level: Medium — the spec no longer names the pair-counting trick, so the model must derive that only mismatched symmetric pairs cost a change (the middle element of odd-length lists is free) and index them without off-by-one.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 704 OUT371 |
| ✓ | PluckSmallestEvenNodeModel returns the smallest even value together with its first index as [value, index], or an empty list when there is none. Provenance: HumanEval/68 (entry point pluck). Level: Medium — track a running minimum plus its index, break ties toward the earliest index, and handle empty / no-even inputs; a few interacting edge cases beyond a plain scan.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 720 OUT665 |
| ✓ | RemoveNonUniqueElementsModel removes every value that appears more than once, keeping only values that occur exactly once. Provenance: HumanEval/26 (entry point remove_duplicates).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 710 OUT147 |
| ✓ | RescaleToUnitIntervalModel linearly rescales a list of doubles so the smallest becomes 0 and the largest becomes 1. Provenance: HumanEval/21 (entry point rescale_to_unit).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 718 OUT822 |
| × | SecondSmallestDistinctValueModel returns the second smallest DISTINCT element of a list, or null when there is none. Provenance: HumanEval/90 (entry point next_smallest; canonical dedups via set before taking index 1). Level: Medium — dedup + sort + take second, with the null cases (empty, all-equal) and the distinct-value subtlety that flips [1,1,1,1,0] from 1-vs-something.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-sorted: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:16: Operator '==' cannot compare 'List<int>' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared.
Values:
run.Compiled = false |
Failed | IN8 697 OUT155 |
| ✓ | SortArrayByEndpointParityModel sorts a copy of a non-negative integer list ascending or descending depending on the parity of the sum of its first and last elements, without mutating the input. Provenance: HumanEval/88 (entry point sort_array). Level: Medium — compose a copy, an endpoint-parity decision that selects the sort direction, and a sort, plus the empty/single edge cases; more than a plain sort.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 731 OUT848 |
| ✓ | SortedUniqueIntegersModel returns the distinct elements of a list sorted ascending. Provenance: HumanEval/34 (entry point unique). Level: Easy — deduplicate then sort; two straightforward operations, one empty-list edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 684 OUT441 |
| × | SortNumberWordsModel sorts number words from zero to nine into ascending numeric order. Provenance: HumanEval/19 (entry point sort_numbers).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
19:33: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '('.
Values:
run.Compiled = false |
Failed | IN8 722 OUT342 |
| × | SortValuesAtEvenIndicesModel sorts the values at even indices, leaving odd indices in place. Provenance: HumanEval/37 (entry point sort_even). Level: Medium — collect even-index values, sort them, then scatter them back; index bookkeeping across two passes.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-small: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: Expected a type (bool, byte, int, long, decimal, double, char, string, List<...>, Dictionary<...>, HashSet<...> or a struct name), found '{'.
Values:
run.Compiled = false |
Failed | IN8 711 OUT1 102 |
| ✓ | SortValuesAtIndicesDivisibleByThreeModel sorts the values that sit at indices divisible by three, leaving the other positions in place. Provenance: HumanEval/33 (entry point sort_third). Level: Medium — collect every third value, sort them, then scatter them back by index; a few steps with index bookkeeping.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 712 OUT887 |
| ✓ | StrangeSortAlternatingMinMaxModel reorders a list by alternately taking the smallest and largest of the remaining values. Provenance: HumanEval/70 (entry point strange_sort_list). Level: Medium — recognise the sort-then-two-pointer pattern, keep state while alternating min/max, and preserve duplicates and odd-length handling.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 721 OUT3 124 |
Code Generation - String Algorithms
10 passed6 failed |
62.5% |
IN139 618 OUT10 728 | |
| ✓ | CountSentencesStartingWithIModel counts sentences that start with the standalone word "I". Sentences are delimited by '.', '?' or '!'. Provenance: HumanEval/91 (entry point is_bored). The exact "starts with word I" rule (I vs Is/It, standalone I at sentence end) is spelled out to make the count deterministic. Level: Hard — scan with sentence-boundary state, skip leading spaces, detect a standalone leading "I" while rejecting distractors like Is/It and mid-sentence I, across several subtle edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Hardsource:HumanEval |
Passed | IN8 762 OUT2 836 |
| × | DeleteCharsThenCheckPalindromeModel deletes characters that appear in a removal set and reports whether the remainder is a palindrome. Provenance: HumanEval/112 (entry point reverse_delete). SC# returns a result struct instead of Python's tuple. Level: Easy — filter characters, then a straightforward two-pointer palindrome check; the empty string counts as a palindrome.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-abcde: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
16:40: Expected ')' to close the StringBuilder constructor. Use 'new StringBuilder()'., found 'filteredText'.
Values:
run.Compiled = false |
Failed | IN8 733 OUT149 |
| ✓ | DictionaryKeysUniformCaseModel returns true when every key of a non-empty dictionary is lowercase, or every key is uppercase; false for empty, mixed, or titlecase keys. Provenance: HumanEval/95 (entry point check_dict_case). SC# Dictionary<string,string> keys are always strings, so Python's non-string-key case is intentionally not represented. Level: Medium — reduce over the keys checking a uniform-casing property, with the empty, mixed and titlecase edge cases and the "cased letters required" subtlety.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 707 OUT162 |
| × | DigitNamesDescendingModel sorts the integers, walks them in descending order, keeps only values 1..9 and replaces each with its English name in title case. Provenance: HumanEval/105 (entry point by_length). Level: Medium — chain sort + descending traversal + range filter + name mapping, with the out-of-range and empty edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-long: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:25: Unexpected character '`'.
1:37: Unexpected character '`'.
Values:
run.Compiled = false |
Failed | IN8 734 OUT622 |
| ✓ | FlipCaseOrReverseNoLettersModel swaps the case of every letter, or reverses the string when it contains no letters. Provenance: HumanEval/161 (entry point solve). Level: Medium — one pass swaps each letter's case while tracking whether any letter was seen, then a second branch reverses the whole string when none was; two distinct output modes plus the empty/no-letter edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 718 OUT259 |
| × | IntegerToLowercaseRomanModel converts an integer in 1..1000 to its lowercase Roman numeral, using subtractive forms. Provenance: HumanEval/156 (entry point int_to_mini_roman). Level: Medium — greedy subtraction against the full value/symbol table including all subtractive pairs; one wrong table entry fails a case.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.ResultJson == expectedJson
Context: baseline-19: expected "xix", got "xcix"
Values:
run.ResultJson = "\"xcix\""
expectedJson = "\"xix\"" |
Failed | IN8 727 OUT272 |
| ✓ | KeepWordsWithPrimeLengthModel keeps only the words whose length is prime, preserving order, and rejoins them. Provenance: HumanEval/143 (entry point words_in_sentence). Level: Medium — split, a per-word primality sub-check, order-preserving filter, and rejoin (~3 composed steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 715 OUT684 |
| ✓ | NormalizeSpaceRunsModel rewrites runs of spaces: one or two become underscores, three or more become a single hyphen. Provenance: HumanEval/140 (entry point fix_spaces). Level: Medium — run-length replacement with a threshold, applied at the start, middle and end of the string.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 765 OUT1 132 |
| ✓ | RemoveVowelsFromTextModel returns the input string with every vowel removed. Provenance: HumanEval/51 (entry point remove_vowels). Level: Easy — single pass filtering characters against a small vowel set.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 701 OUT138 |
| × | SelectWordsByConsonantCountModel selects the words containing exactly n consonants, preserving order. Provenance: HumanEval/117 (entry point select_words). Level: Easy — tokenize on spaces and count consonants per word; a couple of edge cases (empty string, repeated spaces).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-little: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
23:26: Unknown variable 'Char'.
Values:
run.Compiled = false |
Failed | IN8 741 OUT1 051 |
| × | ShortestPalindromeExtensionModel extends a string into the shortest palindrome that begins with it. Provenance: HumanEval/10 (entry point make_palindrome). Level: Medium — requires the insight of finding the longest palindromic suffix and appending the reversed remaining prefix, with edge cases (empty, single char, already a palindrome).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.ResultJson == expectedJson
Context: baseline-xyz: expected "xyzyx", got "xyzzyx"
Values:
run.ResultJson = "\"xyzzyx\""
expectedJson = "\"xyzyx\"" |
Failed | IN8 726 OUT262 |
| ✓ | SortCharactersWithinWordsModel sorts the characters within each space-separated word ascending by ASCII value, keeping word order and every space. Provenance: HumanEval/86 (entry point anti_shuffle). Level: Medium — tokenise on spaces while preserving blanks (including consecutive and trailing spaces), sort each word's characters, and rejoin without dropping or collapsing whitespace.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 717 OUT1 292 |
| ✓ | SplitWordsByCommaOrSpaceModel splits a string into words separated by commas or spaces, dropping empty tokens. Provenance: HumanEval/101 (entry point words_string). Level: Easy — a single-pass tokenizer over two separators with the empty/repeated-separator edge.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 705 OUT135 |
| ✓ | StrongestClassExtensionModel picks the extension whose strength (uppercase letters minus lowercase letters) is greatest, breaking ties by first occurrence, and formats ClassName.Extension. Provenance: HumanEval/153 (entry point Strongest_Extension). Level: Medium — compute a per-extension score, take the argmax with a first-wins tie rule, then format (~3 steps).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 725 OUT1 447 |
| ✓ | SumUppercaseAsciiCodesModel sums the ASCII codes of the uppercase English letters in a string, ignoring all other characters. Provenance: HumanEval/66 (entry point digitSum). Level: Easy — a single pass with one character-range condition and an accumulator.1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 701 OUT69 |
| × | ValidateFileNameModel validates a file name against a set of naming rules. Provenance: HumanEval/141 (entry point file_name_check). Level: Medium — five independent rules (digit count, exactly one dot, non-empty stem, leading latin letter, allowed extension) with several distinct edge cases.1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.ResultJson == expectedJson
Context: baseline-two-dots: expected "No", got "Yes"
Values:
run.ResultJson = "\"Yes\""
expectedJson = "\"No\"" |
Failed | IN8 741 OUT218 |
Code Generation - Strings & Text
13 passed6 failed |
68.4% |
IN165 414 OUT6 826 | |
| × | AllPrefixesModel returns all non-empty prefixes from shortest to longest. Provenance: HumanEval/14 (entry point all_prefixes).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
4:14: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared.
Values:
run.Compiled = false |
Failed | IN8 684 OUT94 |
| ✓ | CanConcatenateToBalancedModel checks whether two parenthesis strings can be concatenated in some order to form a balanced string. Provenance: HumanEval/119 (entry point match_parens).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 731 OUT163 |
| × | ClosestVowelBetweenConsonantsModel finds the rightmost vowel that lies strictly between two consonants. Provenance: HumanEval/118 (entry point get_closest_vowel).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-yogurt: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
8:99: Char literals must contain exactly one character and close with '''.
28:119: Char literals must contain exactly one character and close with '''.
36:18: Char literals must contain exactly one character and close with '''.
Values:
run.Compiled = false |
Failed | IN8 734 OUT562 |
| ✓ | ConcatenateStringsModel concatenates a list of strings in order. Provenance: HumanEval/28 (entry point concatenate).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 683 OUT48 |
| ✓ | CountDistinctCharactersModel counts distinct characters case-insensitively. Provenance: HumanEval/16 (entry point count_distinct_characters).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 674 OUT98 |
| × | CountOverlappingOccurrencesModel counts overlapping substring occurrences. Provenance: HumanEval/18 (entry point how_many_times).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
8:19: Expected ')' to close the parenthesized expression, found '='.
Values:
run.Compiled = false |
Failed | IN8 679 OUT86 |
| ✓ | CountUppercaseVowelsAtEvenIndicesModel counts uppercase vowels located at even indices of a string. Provenance: HumanEval/98 (entry point count_upper).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 691 OUT98 |
| ✓ | CountVowelsWithTrailingYModel counts vowels, additionally counting a trailing y as a vowel. Provenance: HumanEval/64 (entry point vowels_count).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 718 OUT856 |
| ✓ | FilterStringsByPrefixModel filters strings by prefix, preserving order. Provenance: HumanEval/29 (entry point FilterByPrefix).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 690 OUT617 |
| ✓ | FlipCaseModel flips ASCII letter case. Provenance: HumanEval/27 (entry point flip_case).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 685 OUT374 |
| × | LastCharIsStandaloneLetterModel reports whether the final character is a standalone single-letter word. Provenance: HumanEval/134 (entry point check_if_last_char_is_a_letter).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-apple: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:14: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared.
Values:
run.Compiled = false |
Failed | IN8 732 OUT111 |
| × | ListWithSmallerCharTotalModel returns whichever list has the smaller total character count, preferring the first on a tie. Provenance: HumanEval/74 (entry point total_match).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-empty-both: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:1: 'var' can only be used for local variables and foreach variables, not here. Use an explicit type.
Values:
run.Compiled = false |
Failed | IN8 703 OUT543 |
| ✓ | LongestStringModel returns the first longest string, or null for empty input. Provenance: HumanEval/12 (entry point longest).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 690 OUT123 |
| ✓ | OddDigitCountSentencesModel builds a fixed sentence per digit string where every letter 'i' is replaced by the odd-digit count. Provenance: HumanEval/113 (entry point odd_count).1 AI query
0 tool callscategory:CodeGenerationlevel:Mediumsource:HumanEval |
Passed | IN8 841 OUT1 324 |
| ✓ | PalindromeWithinWeightModel returns true when a list is a palindrome and its sum fits a weight limit. Provenance: HumanEval/72 (entry point will_it_fly).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 701 OUT127 |
| ✓ | RotateAlphabetCipherModel rotates each lowercase letter forward through the alphabet with wraparound. Provenance: HumanEval/89 (entry point encrypt).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 722 OUT1 143 |
| × | SameDistinctCharactersModel checks whether two strings share exactly the same distinct characters. Provenance: HumanEval/54 (entry point SameChars).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval Assertion expression evaluated to false: run.Compiled
Context: baseline-true-long: candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
9:16: Type 'HashSet<char>' has no supported method named 'Equals' with 1 argument(s).
Values:
run.Compiled = false |
Failed | IN8 687 OUT75 |
| ✓ | SpaceDelimitedSequenceModel builds a space-delimited sequence of integers from 0 through n. Provenance: HumanEval/15 (entry point string_sequence).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 689 OUT293 |
| ✓ | StringIsPalindromeModel checks whether a string is a palindrome. Provenance: HumanEval/48 (entry point IsPalindrome).1 AI query
0 tool callscategory:CodeGenerationlevel:Easysource:HumanEval |
Passed | IN8 680 OUT91 |