In a report published October 19, OpenAI presents the public openai/MultiLangBench suite alongside its claim of stronger coding in less common languages. It reports an average test pass rate of 87.4% for GPT-6 Astra across 20 languages, compared with GPT-5.6 Sol's 26.8%. The accompanying public suite lets readers inspect what those percentages are trying to measure.
The language changes.
The examiner stays outside.
A candidate program receives an existing database and a SQL request. It must leave a modified database that the real SQLite implementation can read. The test checks a specified query against that file. Printing a convincing answer is insufficient.
This is a useful way to compare implementations written in languages with very different tools and conventions. The interface is a common artifact, rather than language-specific unit tests. The published contract requires the chosen implementation language and an offline environment stripped of existing SQLite implementations.
One small exam, outside the implementation
One published fixture starts with every pair of numbers from 1 to 6. Follow the query to the rows the reference reader expects.
The fixture supplies every combination of x and y from 1 to 6.
The complete example in text and tables
Input. Table t2 holds 36 rows. Each cell below represents one (x,y) row. Coordinates are arranged for reading; this grid does not reproduce insertion order or database storage.
| y / x | x=1 | x=2 | x=3 | x=4 | x=5 | x=6 |
|---|---|---|---|---|---|---|
| y=1 | 1,1 | 2,1 | 3,1 | 4,1 | 5,1 | 6,1 |
| y=2 | 1,2 | 2,2 | 3,2 | 4,2 | 5,2 | 6,2 |
| y=3 | 1,3 | 2,3 | 3,3 | 4,3 | 5,3 | 6,3 |
| y=4 | 1,4 | 2,4 | 3,4 | 4,4 | 5,4 | 6,4 |
| y=5 | 1,5 | 2,5 | 3,5 | 4,5 | 5,5 | 6,5 |
| y=6 | 1,6 | 2,6 | 3,6 | 4,6 | 5,6 | 6,6 |
Choose. Keep x=2 and y IN(2,4,5). The matching rows are (2,2), (2,4) and (2,5).
Order and write. The query creates table r from those rows with ORDER BY x ASC,y DESC. Since every selected x is 2, descending y determines their order.
Read. Reference SQLite runs SELECT x,y FROM r ORDER BY rowid. The fixture’s expected output is:
| Row | x | y |
|---|---|---|
| 1 | 2 | 5 |
| 2 | 2 | 4 |
| 3 | 2 | 2 |
A schematic of a published test, not a generated model run. The fixture also creates t3 and indexes, omitted from this drawing. Reading the expected rows checks this observable result; it does not establish that every database behavior is correct.
Inspect the input, query, reference read and expected output.
Selection is only half the answer.
The rows have to survive.
In this example, the implementation needs to understand a request, find three matching pairs, put them in the requested order and write a table the reference reader can recover. The check crosses the boundary between the candidate's internal representation and a persistent database file.
The golden-output generator makes that boundary inspectable. It pins SQLite 3.40.1, builds the starting database and produces the reference result. It then opens the output read-only and serializes the validation query's rows. That script establishes expected answers; it does not run a coding model.
A common exam.
A bounded slice of behavior.
The repository describes roughly 24,000 converted SQLite tests and recommends a 272-test subset for routine evaluation. A passing case establishes the answer to its validation query. It cannot establish every database behavior, execution plan or performance property.
The gain reported by OpenAI is substantial, including in COBOL, its hardest language in this comparison. But these are the provider's measurements. We inspected the contract, fixture, scripts and history; we did not reproduce model scores. The repository contains no candidate implementations or complete agent-run records that would independently establish them.
The strongest competing reading is therefore specific: models may have become much better at this database-construction task without improving equally on every unfamiliar business system. The useful next experiment is to make that distinction observable.
