Results of 1 - 10 of about 16 for pyarrow (0.001 sec.)
- SQL on Apache Arrow 13533
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
...es Arrow Tables stored in local variables can be queried as if they are regular tables within DuckDB. import duckdb import pyarrow as pa # connect to an in-memory database con = duckdb.connect() my_arrow_table = pa.Table.from_pydict({'i': [1, 2, 3, 4],...translate down into the dataset scan operation so that only the necessary data is pulled into memory. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds # connect to an in-memory database con = d...t, but using Arrow compute operations instead. Arrow can use asynchronous IO to quickly access files. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds import pyarrow.compute as pc # connect to... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Export to Apache Arrow 10743
- All results of a query can be exported to an Apache Arrow Table using the to_arrow_table function. Alternatively, results can be returned as a RecordBatchReader using the to_arrow_reader function and results can be read one batch at a time.
...reader . Results of con.execute() and the duckdb module no longer have them. Export to an Arrow Table import duckdb import pyarrow as pa my_arrow_table = pa.Table.from_pydict({'i': [1, 2, 3, 4], 'j': ["one", "two", "three", "four"]}) # query the Apache...mplements __arrow_c_stream__ of the Arrow PyCapsule interface , so libraries that accept an Arrow stream read it directly: pyarrow.table(rel) , pyarrow.RecordBatchReader.from_stream(rel) and polars.DataFrame(rel) . Polars reads it this way even without...pyarrow installed. import duckdb import polars as pl import pyarrow as pa rel = duckdb.sql("SELECT i, 'v' || i AS s FROM range(3) AS t(i)") print(pa.table(rel).to_pydict()) print(pl.DataFram... - https://man.mtool.work/duckdb/2.0/guides/python/export_arrow.html
- Integration with PyTorch 10639
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
...s are converted to tensors. Installation pip install -U duckdb numpy pyarrow torch DuckDB to PyTorch This example queries a Parquet-backed Arrow dataset, streams the result as RecordBatch objects and...converts each batch into feature and label tensors. import pathlib import tempfile import duckdb import numpy as np import pyarrow as pa import pyarrow.dataset as ds import pyarrow.parquet as pq import torch base_path = pathlib.Path(tempfile.mkdtemp())... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
- Integration with Polars 9652
- Polars is a DataFrames library built in Rust with bindings for Python and Node.js. It uses Apache Arrow's columnar format as its memory model. DuckDB can read Polars DataFrames and convert query results to Polars DataFrames. It does this in
...ternally using the efficient Apache Arrow integration. Note that the pyarrow library must be installed for the integration to work. Installation pip install -U duckdb 'polars[pyarrow]' Polars to Duck...PROJECT */2 COLUMNS A relation also implements the Arrow PyCapsule interface , so pl.DataFrame(rel) reads it too, without pyarrow installed. pl.from_arrow(rel) returns a Series with one struct per row instead of a DataFrame. To learn more about Polars,... - https://man.mtool.work/duckdb/2.0/guides/python/polars.html
- ADBC Client 9253
- Installation To use the DuckDB ADBC client, download the libduckdb archive for your platform and follow the instructions below . For Python, install the duckdb package instead, as described in the Python section . The latest stable version
...its adbc_driver_duckdb module, so Python needs no libduckdb archive. Install it together with the ADBC driver manager and pyarrow : pip install duckdb adbc_driver_manager pyarrow pyarrow to fetch rows ( fetchone and fetchall ). It is not needed for fet...b.dbapi with adbc_driver_duckdb.dbapi.connect("test.db") as conn, conn.cursor() as cur: cur.execute("SELECT 42") # fetch a pyarrow table tbl = cur.fetch_arrow_table() print(tbl) Alongside fetch_arrow_table , the cursor implements the standard DB-API met...hods, such as fetchone and fetchall . To load Arrow data into a table, pass the table name and the data, such as a pyarrow record batch or table, to cur.adbc_ingest . By default it creates the table and fails if the table exists; its mode argume... - https://man.mtool.work/duckdb/2.0/clients/adbc.html
- Python Function API 9253
- You can create a DuckDB user-defined function (UDF) from a Python function so it can be used in SQL queries. Similarly to regular functions , they need to have a name, a return type and parameter types. Creating Functions To register a Pyth
...w. This parameter specifies the return type of the function. type (optional): DuckDB supports both native Python types and PyArrow Arrays. By default, type = 'native' is assumed, but you can specify type = 'arrow' to use PyArrow Arrays. In general, usin...d to receive arrow arrays, set the type parameter to 'arrow' . This will let the system know to provide each argument as a pyarrow.ChunkedArray of at most 2048 rows to the function, and also expect an array of the same amount of rows to be returned from...g an Arrow UDF will be much more efficient than native because it will be able to operate in batches. import duckdb import pyarrow as pa from duckdb.sqltypes import VARCHAR from pyarrow import compute as pc def mirror(strings: pa.Array, sep: pa.Array) -... - https://man.mtool.work/duckdb/2.0/clients/python/function.html
- Python API 8248
- Installation To use the DuckDB Python client, visit the Python installation page . The latest stable version of the DuckDB Python client is 2.0.0. Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please s
..._df = pl.DataFrame({"a": [42]}) duckdb.sql("SELECT * FROM polars_df") ┌───────┐ │ a │ │ int64 │ ├───────┤ │ 42 │ └───────┘ PyArrow To directly query a PyArrow table, run: import duckdb import pyarrow as pa arrow_table = pa.Table.from_pydict({"a": [42]})... - https://man.mtool.work/duckdb/2.0/clients/python/overview.html
- Import from Apache Arrow 8144
- CREATE TABLE AS and INSERT INTO can be used to create a table from any query. We can then create tables or insert into existing tables by referring to the Apache Arrow object in the query. This example imports from an Arrow Table , but Duck
...DB can query different Apache Arrow formats as seen in the SQL on Arrow guide . import duckdb import pyarrow as pa # connect to an in-memory database my_arrow = pa.Table.from_pydict({'a': [42]}) # create the table "my_table" from t... - https://man.mtool.work/duckdb/2.0/guides/python/import_arrow.html
- Delta Extension 7953
- The delta extension adds support for the Delta Lake open-source storage format . It is built using the Delta Kernel . The extension reads Delta tables, local and remote, and writes them: it can create tables and append to them. For implemen
...larger data, use Arrow Datasets to read it lazily. The example needs deltalake , the Python package of delta-rs , with its pyarrow extra: pip install duckdb "deltalake[pyarrow]" Full example import deltalake as dl import duckdb import pyarrow as pa # Cr... - https://man.mtool.work/duckdb/2.0/core_extensions/delta.html
- Parquet Encryption 7659
- Starting with version 0.10.0, DuckDB supports reading and writing encrypted Parquet files. DuckDB broadly follows the Parquet Modular Encryption specification with some limitations . Reading and Writing Encrypted Files Using the PRAGMA add_
...ey: 'key256'}); Interoperability DuckDB can read uniformly encrypted Parquet files written by the Arrow C++ API (e.g., via PyArrow), as long as the same encryption key is used for both the footer and all columns. Limitations DuckDB's Parquet encryption... - https://man.mtool.work/duckdb/2.0/data/parquet/encryption.html
DuckDB 2.0 Documentation / man.mtool.work