Polars CSV schema: preserve IDs

1. Run the disposable exercise

Download lab.py into a fresh scratch folder and read it. The script accepts no input paths, loads no customer files and performs no import. It builds CSV bytes in memory, compares them with Python's standard-library CSV reader and raises an error if a checked outcome changes. Its checks remain active even when Python runs with optimization enabled.

The checked package is the official Python Polars 2.1.0 release. The command below uses uv 0.13.6 syntax from the official script-dependency guide. Install uv using its official instructions first. The same command works in a Windows terminal or a Unix shell; we executed the Linux path only. uv may download Python and the pinned package. That setup uses the network; the lab itself makes no network requests.

uv run --no-project --python 3.11 --with polars==2.1.0 python lab.py

The result is one JSON object containing eight cases and the original rejected record. A successful run exits normally. The two expected parser failures are caught and reported as ComputeError; an unexpected result stops the script. Do not substitute the latest package and label it the same experiment. The script deliberately refuses a different Polars version.

No special Polars build, Rust compiler or database account is needed for this Python exercise. Use a compatible Python environment and the official package for your platform. If installation fails, resolve that environment issue rather than weakening a value-preservation check to get a green run.

2. Read the eight outcomes, not just the exit status

Two numeric-only controls contain order_id values 0007 and 0042 with amounts 12 and 34. One control quotes both identifiers. The longer fixture has 101 data rows: identifiers 0001 through 0101, amounts 1 through 100, then the text unknown in the final row. There are no empty fields or physical blank records in these inputs. The 2.1.0 API defaults to a 100-row inference window, so the conflicting amount is just outside that sample.

The quoted control is useful because quotation marks look like a request to keep text, but here they only served as CSV syntax. The failed late-value reads prove a different point: no completed frame was available. They do not themselves demonstrate identifier conversion. That behavior is established by the successful numeric controls and full-inference read.

The last case is the misleading success. The row count remained 101, yet the original unknown token was no longer present in the amount column. A row count alone would have missed it. The lab checks exact identifier lists, the final amount and per-column null counts, so a successful read cannot silently stand in for all those conditions.

3. Choose types before identifiers become numbers

According to the Polars 2.1.0 CSV API source, inference samples an initial window, schema_overrides fixes selected types, and infer_schema=False makes unspecified columns strings. For this fixture, the following are two alternative reads. Each snippet assumes late is the exact bytes generated by the downloadable lab; neither is a complete standalone import program.

import polars as pl

# Choose the two known columns explicitly:
text = pl.read_csv(late, schema_overrides={
    "order_id": pl.String,
    "amount": pl.String,
})

# Or inspect all columns as strings:
text = pl.read_csv(late, infer_schema=False)

Both reads preserved every expected decoded value in this fixture. An identifier override alone did not make the late amount readable; the columns need separate decisions. Keeping amount as text makes the unexpected token inspectable, not acceptable as a numeric amount. Decide what your application permits before converting it.

Full inference is a diagnostic option, not the default recommendation for large files. It resolved this mixed amount column but still chose integers for the independent ID column. The tagged inference implementation classifies observed values; it cannot know that your all-digit value is an order code. Once 0007 has become integer 7, converting that integer to text cannot recover the original padding without another contract or source.

4. Inspect original values before deciding what to reject

The lab makes a separate, non-strict integer conversion while retaining the original text frame. This Polars 2.1.0 snippet uses the tagged Series cast and null-check APIs and DataFrame filter API. Here text means the complete all-string frame from the previous section.

parsed = text["amount"].cast(pl.Int64, strict=False)
rejected = text.filter(parsed.is_null())
print(rejected.to_dicts())
# This fixture: [{"order_id": "0101", "amount": "unknown"}]

Because this fixture starts with no null amounts, the null mask identifies its one failed conversion. On real input the same mask can include preexisting nulls. Define missing-value policy separately; do not call every null an invalid token. Keep the original identifier and amount beside any parsed column so a reviewer can see what was actually supplied.

An import contract also needs rules this exercise does not enforce: required fields, identifier width and uniqueness, permitted ranges and destination relationships. A String column alone guarantees none of those. For a separate shell-based schema trial, use the qsv CSV validation exercise. That is an optional companion, not a prerequisite for Polars or proof that your destination accepted anything.

Method, limits and the next useful check

The downloadable code is Built with Rust's original synthetic fixture. Python's 3.11 CSV reader reference supplies the independent string-oriented comparison for decoded input fields. We also reproduced the central full-inference-versus-String result with a separately authored four-row fixture and a two-row inference window. Neither check used customer records, analytics data or an external dataset.

The eight published cases use eager read_csv with the normal CPU implementation, not PyArrow or GPU parsing. The lab does not test malformed quoting, ragged records, arbitrary encodings, overflow, locale-specific numbers, dates, large-file memory use or blank-line handling. String reads still have null rules: do not assume a bare empty field, quoted empty field and literal NULL mean the same thing. Add explicit fixtures for your own conventions before depending on them.

For lazy processing, the 2.1.0 scan_csv reference in the tagged source describes a separate reader entry point. This exercise does not establish eager/lazy equivalence. Constructing a lazy plan is not a completed read, and a projection that skips a troublesome column cannot establish that column's validity. We measured no speed or memory advantage here.

Next, copy the acceptance idea, not an unreviewed import policy: build a synthetic example of your identifier, a late unexpected value and your missing-value convention. State the exact rows and values you expect before reading. Then verify the complete output and review failures before connecting any destination. Explore the Data & AI projects when choosing the surrounding workflow.

Common questions

Does quoting a CSV identifier preserve leading zeros in Polars?

Not in our Polars 2.1.0 numeric-only control: both 0007 and quoted 0007 became integer 7. Set that column to String when reading, then compare the exact expected identifiers.

Will infer_schema_length=None fix a Polars CSV import?

It resolved the late text amount in our 101-row fixture, but the separate identifier column still became integers. Full inference is not an identifier or business-rule contract.

Is ignore_errors=True a safe way to keep every value?

No. In our Polars 2.1.0 fixture, the late unknown amount became null while the row remained. Inspect the original text before deciding how to handle a failed conversion.