Results of 1 - 5 of about 5 for RecordBatchReader (0.002 sec.)
- SQL on Apache Arrow 12446
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchReader
...ders are read in full and filtered afterwards. A RecordBatchReader can be scanned only once; to query its data more than once, read it into a table with read_all() before the first query. A...arrow_scanner" and return as an Arrow Table results = con.sql("SELECT * FROM arrow_scanner").to_arrow_table() Apache Arrow RecordBatchReaders Arrow RecordBatchReaders are a reader for Arrow's streaming binary format and can also be queried directly as if they wer...duckdb.connect() my_recordbatch = pa.RecordBatch.from_pydict({'i': [1, 2, 3, 4], 'j': ["one", "two", "three", "four"]}) my_recordbatchreader = pa.ipc.RecordBatchReader.from_batches(my_recordbatch.schema, [my_recordbatch]) # query the Apache Arrow RecordBatchReade... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Export to Apache Arrow 9927
- All results of a query can be exported to an Apache Arrow Table using the to_arrow_table function. Alternatively, results can be returned as a RecordBatchReader using the to_arrow_reader function and results can be read one batch at a time.
...rrow_table" and return as an Arrow Table results = duckdb.sql("SELECT * FROM my_arrow_table").to_arrow_table() Export as a RecordBatchReader import duckdb con = duckdb.connect() rel = con.sql("SELECT i FROM range(5) AS t(i)") with rel.to_arrow_reader(batch_size=2...the Relational API. A relation can be converted to an Arrow table using DuckDBPyRelation.to_arrow_table , and to an Arrow record batch reader using DuckDBPyRelation.to_arrow_reader . import duckdb # connect to an in-memory database con = duckdb.connect() con.exe...xport that result to Arrow res = rel.aggregate("sum(i)") arrow_table = res.to_arrow_table() # You can also create an Arrow record batch reader from a relation arrow_batch_reader = res.to_arrow_reader() for batch in arrow_batch_reader: # Process a single chunk her... - https://man.mtool.work/duckdb/2.0/guides/python/export_arrow.html
- Data Ingestion 8236
- This page contains examples for data ingestion to Python using DuckDB. First, import the DuckDB package: import duckdb Then, proceed with any of the following sections. CSV Files CSV files can be read using the read_csv function, called eit
...s querying multiple types of Apache Arrow objects including tables , record batches, arrays and chunked arrays, datasets , RecordBatchReaders and scanners . See the Python guides for more examples. The filters of a query are pushed into Arrow datasets and Polars...les, record batches, readers, scanners and Polars DataFrames are read whole, and the filter applies to what they return. A RecordBatchReader can be read once. A second scan raises InvalidInputException: This RecordBatchReader has already been consumed and cannot... - https://man.mtool.work/duckdb/2.0/clients/python/data_ingestion.html
- Troubleshooting 7758
- Troubleshooting Running EXPLAIN Renders Newlines In Python, the output of the EXPLAIN statement contains hard line breaks ( \n ): In [1]: import duckdb ...: duckdb.sql("EXPLAIN SELECT 42 AS x") Out[1]: ┌───────────────┬─────────────────────
...after a Subinterpreter Was Created On Windows, once the process has created a subinterpreter, a query that scans a pyarrow.RecordBatchReader whose batches come from a Python generator can hang. DataFrame Queries or Boxplots Fail in JupySQL With JupySQL 0.11.1, ru... - https://man.mtool.work/duckdb/2.0/clients/python/known_issues.html
- Python API 7519
- Installation To use the DuckDB Python client, visit the Python installation page . The latest stable version of the DuckDB Python client is 2.0.0. Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please s
...Polars DataFrame duckdb.sql("SELECT 42").to_arrow_table() # Arrow Table duckdb.sql("SELECT 42").to_arrow_reader() # Arrow RecordBatchReader duckdb.sql("SELECT 42").fetchnumpy() # NumPy Arrays Writing Data to Disk DuckDB supports writing Relation objects directly... - https://man.mtool.work/duckdb/2.0/clients/python/overview.html
PREV
1
NEXT
DuckDB 2.0 Documentation / man.mtool.work