Introduction

This article was inspired by a video by Protorikis. It tests several open-weight models on FinQA, a benchmark of 1,147 numerical questions about company annual reports. No model passed 80% of the questions, and the video concludes that financial reports are very hard for LLMs. We wanted to verify this:

How much of the gap comes from the models, and how much from the benchmark itself?

Before we fixed anything, our results were in the same range. On the original tasks, Qwen 3.6 27B in instruct mode passed about 76% of the questions. Without the tasks whose two FinQA reference answers disagree, it passed about 81%. In the video, the models passed between 70% and 78%. So in both cases about 25% of the answers counted as wrong.

Our review then showed that about 23% of the original questions have a problem: a wrong reference answer, an ambiguous question or missing data. The next sections describe these problems with examples, explain how we fixed them, and show how Qwen 3.6 27B does on the corrected tests.

• Model: Qwen 3.6 27B, our main local model.
• Modes: instruct mode and thinking mode.
• Quantization: FP8.
• Runtime: vLLM, runs from 4 to 5 October 2026.
• Benchmark: the Financial Reasoning Benchmark based on FinQA, with 1,125 tasks. The report excerpt and the question are in the prompt, and the model answers without tools.

Every test runs once, and its first answer counts. Failed tests are never rerun. The aim is to show what a user can expect from a single request, not the best possible result.

The chart below shows the results on the original tasks. The video tested other models, and we do not know how its portal grades answers. Our runs accepted an answer both as a fraction and as a percentage.

FinQA

FinQA was published at EMNLP 2021. Each task contains an excerpt from an S&P 500 annual report: some text, a table and more text. The question needs one or more figures from the excerpt and a short calculation, such as a difference, a ratio or a growth rate.

For every task, the dataset stores a short calculation formula, for example subtract(5829, 5735) or subtract(5829, 5735), divide(#0, 5735), where #0 is the result of the first step. FinQA calls it a program. The result of this formula is the reference answer. It also stores a human-written answer, such as "94" or "14.5%". The public test split has 1,147 tasks with answers. Another 919 tasks are private and have no answers, so we did not use them.

The difficulty depends on the number of operations. Most tasks need one or two operations. A few need four or five, for example an average growth rate over several years.

Problems in the original dataset

About 23% of the tasks needed a fix. We reviewed every task that one of our runs failed, and every task where the two reference fields of FinQA disagree. The same rule applied to every decision:

The chart below shows the result of the review. Most problems fall into two groups: wrong reference answers and ambiguous questions.

Wrong reference answers

In about 130 tasks, the stored formula computes something other than the question asks. A model that answers correctly fails such a task. The examples below show typical cases, with the data from the report that the question needs. In some tasks, the human-written answer of FinQA is right and only the formula is wrong. In others, both are wrong.

Example 1: Apple, tax expense

What was the 2006 tax expense?

Example 2: Entergy, net revenue without two effects

What would 2015 net revenue have been without the retail price change and the volume/weather effect?

Example 3: WestRock, balance with interest and penalties

What was the total balance in 2018 including interest and penalty liabilities?

Example 4: Snap-on, change of the dividend

What was the percentage change in the cash dividend per share from 2006 to 2007?

Example 5: Arista, a yes/no question

Are purchase commitments greater than the expected 2016 lease commitments?

Example 6: Lockheed Martin, an average of three years

What was the average weighted average common shares outstanding for diluted computations from 2012 to 2014?

Example 7: Union Pacific, an improvement in the wrong direction

What will the 2010 operating ratio be if the average of the 2009 and 2008 changes occurs in 2010?

We fixed the reference answer in these tasks. The question stays as it is.

Ambiguous questions

In about 110 tasks, the stored answer is one correct reading of the question, but another reading is just as reasonable. A model that chooses the other reading fails. These problems are hard to see, because a model often happens to choose the author's reading, and the test passes.

Typical patterns:

• One sum or one value per year. "In 2010 and 2009, what was the total fair value of the segregated assets?" can mean one sum (68.9) or one total per year (34.7 and 34.2).
• Which date. "What is the rate of return of the Nasdaq from 2010 to 2011?" The table has three dates in this period: 1/2/2010, 1/1/2011 and 12/31/2011.
• Index points or percent. "What was the five-year change in the S&P 500 index?" can be 105 index points or a growth of 1.05 (105%).
• Which row. Two pension plans in the table, but the question names none of them.
• The sign. "What was the percentage improvement in unbilled receivables?" when the balance fell: is the answer −31% or 31%?

We made these questions precise and kept the reference answer. For example, the first question now asks for "the combined total for 2010 and 2009 together". For questions about a difference or a decrease, both signs are accepted.

Example: one sum or one value per year

The original question allows two readings:

Example: which end date

The original question allows two readings:

Many of these problems did not appear in one run. Instruct mode often chose the author's reading, and thinking mode or a different test setup chose the other one, for example one value per year instead of a sum. A passing test therefore does not prove that a question is clear.

Format of the answer

FinQA does not define the format of the answer. In about 45% of the numeric tasks, the reference formula returns a fraction such as 0.145, while the human answer says "14.5%". Amounts are given in the unit of the table, often thousands or millions, but the question rarely says so.

Without a fixed format, the grading has to guess. A grader that checks only one scale fails correct answers given in the other scale. A grader that accepts every scale can pass wrong answers. We do not know how other FinQA runs, including the one in the video, handle this.

Missing data and broken tables

25 tasks could not be fixed. In these tasks, the excerpt does not contain the data that the question needs, or the question has no clear meaning. Examples:

• "What is the average share price for the shares issued in the U.K.?" The excerpt gives only the number of shares, no price and no amount.
• A table where two column groups have the same header "end of period 2008, 2007, 2006". In the original report, the second group is probably average balances, but the excerpt does not say so.

We left 22 of these tasks out of the migration. Three tasks were removed after the first publication. They stay in the benchmark as skipped tests with the reason, so that existing benchmark reports remain valid.

How we migrated FinQA

Answer contract and grading

The system prompt defines the format of the answer, as our coding tests define the exact function signature:

• the last line is ANSWER: X;
• X is a plain number, without currency signs, thousands separators or unit words;
• percentages, ratios and growth rates are decimal fractions (14.5% is 0.145);
• amounts are in the unit the question asks for, otherwise in the unit of the report;
• at least four significant digits, without rounding, and without changing the unit (thousands stay thousands);
• yes or no for yes/no questions.

The test compares the number with the reference answer with a relative tolerance of 1%. "14.5%" is read as 0.145. Questions about a difference or a decrease accept both signs. The grading is deterministic: the same answer always gets the same result.

A test is skipped only when the task itself is defective. A broken, unfinished or looping answer of the model counts as a failure.

The levels follow the number of steps. One operation is Medium, because every task is a full report page with many numbers that could be confused. Two or three operations are Hard, four or more are VeryHard.

What a test looks like

Each FinQA task becomes one SC# test. The generator puts the report excerpt into the test, asks the model, and checks the answer against the reference value. A complete test from the reasoning benchmark:

/// what was the percent of the decline in amortization expense for other intangible assets from 2016 to 2017
/// @tags Test category:Reasoning category:ContextProcessing level:Hard source:FinQA
void FinQaHum2017DeclineAmortizationExpense()
{
    // FinQA HUM/2017/page_118.pdf-1; reference program: subtract(75, 77), divide(#0, 77)
    var report = @"humana inc . notes to consolidated financial statements 2014 ( continued ) amortization expense for other intangible assets was approximately $ 75 million in 2017 , $ 77 million in 2016 , and $ 93 million in 2015 . the following table presents our estimate of amortization expense for each of the five next succeeding fiscal years: .

|  | ( in millions ) |
| for the years ending december 31, |  |
| 2018 | $ 64 |
| 2019 | 54 |
| 2020 | 52 |
| 2021 | 19 |
| 2022 | 16 |

.";
    var answer = AskFinQa(report, @"what was the percent of the decline in amortization expense for other intangible assets from 2016 to 2017");
    AssertWithToleranceIgnoringSign(answer, -0.02597);
}

The question asks about a decline, and a decline of 2.6% can be written as 0.026 or as −0.026. So the test uses AssertWithToleranceIgnoringSign, which accepts both signs. Questions where the sign matters use AssertWithTolerance, and yes/no questions use AssertYesNo.

AskFinQa sends the excerpt and the question to the model, with the answer contract in the system prompt, and returns the text after ANSWER:. The helper then reads the number from this text and accepts a difference of up to 1% of the reference value. So −0.02597 and −0.026 both pass, but a rounded −0.03 fails:

// Reads the first number in the answer in financial notation: thousands separators and "$" are ignored,
// "(12.5)" is -12.5 (financial statements write negative amounts in parentheses) and "14.5%" is 0.145.
// No rounding and no unit conversion. Null when the answer has no number.
double? ParseFirstFinancialNumber(string answer)
{
    var text = answer.Replace(",", "").Replace("$", "").Replace("\u2212", "-");
    var match = Regex.Match(text, @"(\(?)(-?\d+(?:\.\d+)?)\s*(%?)");
    if (!match.Success) { return null; }
    var value = double.Parse(match.Groups[2]);
    if (match.Groups[1] == "(" && value > 0) { value = -value; }
    if (match.Groups[3] == "%") { value = value / 100; }
    return value;
}

// For questions about a difference, decrease or decline, which do not fix the sign: 0.026 and -0.026 both pass.
void AssertWithToleranceIgnoringSign(string answer, double expected)
{
    var actual = ParseFirstFinancialNumber(answer);
    Assert(actual != null, "answer: " + answer);
    var difference = Math.Abs(Math.Abs(actual.Value) - Math.Abs(expected));
    var tolerance = Math.Max(Math.Abs(expected) * 0.01, 0.00001);
    Assert(difference <= tolerance, "answer: " + answer + ", expected: " + expected.ToString() + " (either sign)");
}

Checking the review itself

A review can make its own mistakes. A reviewer who sees a model's answer may also follow it. To check this, we gave a sample of tasks to new reviewers who saw only the excerpt and the question: no reference answer, no model answer and no FinQA formula. Their answers matched our reference answers in all 60 sampled corrected tasks and in 19 of 20 unchanged tasks. The one mismatch was a FinQA task with missing data, which we then removed.

These reviewers were AI models of the same family as the first reviewers, so they may share some blind spots. Errors that a model makes in exactly the same way as FinQA are also not visible to this method, because such tests pass.

We also checked every question that we had made precise after a model failure. A model failure alone is not a reason to change a question: the goal is a task with one clear answer, not a passing model. So new reviewers got the original wording only, again without a reference answer, a model answer or the FinQA formula. A fix stayed only if the reviewer found two reasonable readings or reached a different answer.

Of 134 reworded questions, 85 kept the fix. In 45, the original wording had one clear answer, so we restored it. In 4 more, an earlier rewording had changed the question to fit a wrong FinQA value. There we restored the original question and corrected the reference answer instead.

The questions that we made precise after the final runs went through the same check, with two independent reviewers for each question. A fix stayed only if both reviewers found a second reasonable reading.

Results on the corrected tests

On the corrected tests, Qwen 3.6 27B passes about 98% of the tasks in both instruct and thinking mode. The report excerpt and the question are in the user message, and the model answers in one reply, without tools. This is the classic FinQA setup.

The level of a task changes the score very little. At the medium and hard levels, the two modes are within about one percentage point. The very hard group has only 26 tasks, so one failed task changes its score by about four points.

Thinking mode

Thinking mode does not raise the score, but it writes much longer answers. It uses almost three times as many output tokens per task as instruct mode, so each answer also takes longer.

Remaining failures

Most of the remaining failures are model errors. We checked every failed task that had not been checked before, in the same way as the fixes: two reviewers got only the excerpt and the question. Where both reached our reference answer and found no second reading, the task stayed unchanged.

The model errors fall into a few groups:

• A wrong figure. The model takes a value from the wrong row, column or year. This is the most common error.
• A wrong unit. The model converts millions or thousands into whole dollars, although the answer contract says to keep the unit of the report.
• A wrong sign. The model reports an increase as a decrease, or the other way round, where the question defines the direction.
• Loops. Four times in thinking mode, the model repeated itself until the generation was stopped. Such a test fails.

Recommendation

Instruct mode and thinking mode get a similar score. Thinking mode writes almost three times as many output tokens for it.

These results cover one-page report excerpts with a short calculation. We do not know how well the model processes long reports, several documents at once or questions that need many figures, because these were not tested. Define the expected unit, scale and sign of the answer in the prompt: most of the errors in the original benchmark come from questions that leave them open.

References

This article is based on the FinQA dataset, our review of its test split, and two benchmark runs with FP8 quantization:

• Qwen 3.6 27B, Instruct: #83
• Qwen 3.6 27B, Thinking: #84

Other sources

• FinQA website and GitHub repository (code MIT, data CC BY 4.0)
• Chen et al., FinQA: A Dataset of Numerical Reasoning over Financial Data, EMNLP 2021
• Protorikis: Meta's Muse vs. 4 AI Models: 1,147 Financial Questions