Breaking News — The popular mssql-python driver now supports fetching SQL Server data directly as Apache Arrow structures, eliminating the performance penalty of per-row Python object creation. This major update, contributed by community developer Felix Graßl, promises to dramatically accelerate data pipelines for Polars, Pandas, DuckDB, and other Arrow-native libraries.
"Fetching a million rows used to mean a million Python objects and considerable garbage-collector pressure," explained Felix Graßl, the developer behind the feature. "With Arrow, the entire fetch loop runs in C++ and writes directly into shared-memory buffers. The DataFrame library simply receives a pointer and starts processing immediately."
Background: The Cost of Row-by-Row Fetching
Traditionally, retrieving large datasets from SQL Server with Python involved constructing one Python object per row—each with its own memory allocation and type conversion. This process created significant overhead, especially for temporal types like DATETIME and DATETIMEOFFSET, where per-value conversions added latency.
The result was high memory usage and slower fetch times, limiting the throughput of data engineering workflows. Developers often had to resort to workarounds or accept performance bottlenecks.
How Apache Arrow Eliminates Bottlenecks
Apache Arrow introduces a columnar in-memory format that stores all values for a column contiguously in typed buffers. Nulls are tracked via a compact bitmap—no None objects per cell. The key enabler is the Arrow C Data Interface, a cross-language ABI that allows drivers and libraries to exchange data by passing a pointer, without serialization or copying.
"Zero-copy language interoperability is the core insight behind Arrow," noted Sumit Sarabhai, who reviewed the feature. "A C++ database driver and a Python DataFrame library can work on the exact same memory without either knowing about each other."
What This Means for Developers
For users of mssql-python, the new Arrow support translates into four concrete benefits:
- Speed: Fetching becomes noticeably faster, especially for temporal types, because Python-side per-value conversions are eliminated entirely.
- Lower memory usage: A column of one million integers is stored as a single contiguous C array, not a million Python objects.
- Seamless interoperability: Polars, Pandas (via
ArrowDtype), DuckDB, and Hugging Face datasets can all consume Arrow data directly—no intermediate format conversion needed. - Reduced garbage-collector pressure: Because fewer Python objects are created, the GC runs less often, improving overall pipeline stability.
Subsequent operations—filters, joins, aggregations—also work in-place on those same shared-memory buffers. A Polars pipeline reading from mssql-python never needs to materialize intermediate Python objects at any stage.
The Arrow C Data Interface: How It Works
The Arrow C Data Interface is an ABI specification that defines a stable shared-memory layout. Any language can produce or consume it by exchanging a pointer, with no serialization or re-parsing. This makes it the foundation for high-throughput data processing across diverse ecosystems.
Immediate Impact on Data Engineering Workflows
Data engineers using SQL Server as a source for analytics pipelines will see the most benefit. Fetching millions of rows for ETL/ELT jobs, machine learning feature extraction, or real-time dashboards can now be done with fraction of the previous memory footprint.
"This change effectively removes a major bottleneck for Python-based data processing with SQL Server," said Sumit Sarabhai. "It opens the door to handling larger datasets without costly infrastructure upgrades."
Availability and Next Steps
The Apache Arrow support is included in the latest release of mssql-python. Developers can install or upgrade via pip install mssql-python --upgrade. The feature is community-contributed and open source, inviting further collaboration.
For full documentation, visit the mssql-python GitHub repository.