Results of 1 - 10 of about 35 for batch (0.001 sec.)
- Export to Apache Arrow 12427
- All results of a query can be exported to an Apache Arrow Table using the to_arrow_table function. Alternatively, results can be returned as a RecordBatchReader using the to_arrow_reader function and results can be read one batch at a time.
...first, or run the other queries on a cursor from con.cursor() . Deprecated On relations, fetch_arrow_table , fetch_record_batch and fetch_arrow_reader are deprecated aliases of to_arrow_table and to_arrow_reader . Results of con.execute() and the duc...able" and return as an Arrow Table results = duckdb.sql("SELECT * FROM my_arrow_table").to_arrow_table() Export as a RecordBatchReader import duckdb con = duckdb.connect() rel = con.sql("SELECT i FROM range(5) AS t(i)") with rel.to_arrow_reader(batch_...size=2) as reader: for batch in reader: print(batch.to_pydict()) {'i': [0, 1]} {'i': [2, 3]} {'i': [4]} The small batch size makes the batches visible... - https://man.mtool.work/duckdb/2.0/guides/python/export_arrow.html
- Handle Results 12380
- Overview Beyond reading a result set row by row , the Rust client can hand a query result to Apache Arrow as a stream of record batches, register an Arrow batch as a queryable table, and return results as Polars data frames. Each of these c
...s through duckdb::arrow . Reading a Result as Arrow Call query_arrow() on a prepared statement to get an iterator of RecordBatch . Collecting it materializes the whole result, while iterating it reads one batch at a time: use duckdb::{Connection, Resu...lt}; use duckdb::arrow::record_batch::RecordBatch; use duckdb::arrow::util::pretty::print_batches; let conn = Connection::open_in_memory()?; let mut stmt = con...n.prepare("SELECT * FROM generate_series(1, 5)")?; let batches: Vec<RecordBatch> = stmt.query_arrow([])?.collect(); print_batches(&batches).unwrap(); get_schema() on the returned hand... - https://man.mtool.work/duckdb/2.0/clients/rust/result_handling.html
- Integration with PyTorch 11298
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
...kdb numpy pyarrow torch DuckDB to PyTorch This example queries a Parquet-backed Arrow dataset, streams the result as RecordBatch objects and converts each batch into feature and label tensors. import pathlib import tempfile import duckdb import numpy...quet_dir)) reader = con.sql(""" SELECT feature_0, feature_1, label FROM train_dataset WHERE label = 1 """).to_arrow_reader(batch_size=2) for batch in reader: features = torch.tensor( np.column_stack( [ batch.column(0).to_numpy(), batch.column(1).to_nu...mpy(), ] ), dtype=torch.float32, ) labels = torch.tensor(batch.column(2).to_numpy(), dtype=torch.int64) print(features.shape, labels) torch.Size([2, 2]) tensor([1, 1]) DuckDB pushes the... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
- Import Data 11059
- Overview The JDBC driver offers two ways to load data into DuckDB in bulk: the high-performance Appender and a JDBC batch writer. This page describes both and when to use each. Appender The Appender is available in the DuckDB JDBC driver vi
...Appender is already the fastest way to bulk-load data, but a few properties of how it works determine its throughput: Row batching is fixed and internal. The Appender writes rows in fixed-size batches of 2,048 rows, one DuckDB vector, and flushes a b...atch automatically as soon as it fills. There is no configurable flush count, so adding an application-level batch size on top of the Appender does not improve performance. Prefer primitive columns over complex types. Nested types such a...which is faster and appends nested and collection types. DuckDBSingleValueAppender is kept only for existing applications. Batch Writer The DuckDB JDBC driver offers batch write functionality. The batch writer supports prepared statements to mitigate... - https://man.mtool.work/duckdb/2.0/clients/java/data_import.html
- ODBC API on macOS 9738
- A driver manager is required to manage communication between applications and the ODBC driver. DuckDB supports unixODBC , which is a complete ODBC driver manager for macOS and Linux. Users can install it from the command line via Homebrew :
...batch brew install unixodbc DuckDB releases ODBC drivers for macOS on Apple Silicon ( arm64 ) and Intel ( x86_64 ) CPUs. To down...load the driver for Apple Silicon, run: batch wget https://github.com/duckdb/duckdb-odbc/releases/download/v2.0.0.0/duckdb_odbc-macos-arm64.zip To download the driver f...or Intel, run: batch wget https://github.com/duckdb/duckdb-odbc/releases/download/v2.0.0.0/duckdb_odbc-macos-amd64.zip The archive contains the... - https://man.mtool.work/duckdb/2.0/clients/odbc/macos.html
- ODBC Extension Functions 9197
- odbc_begin_transaction odbc_bind_params odbc_close odbc_commit odbc_connect odbc_copy odbc_create_params odbc_list_data_sources odbc_list_drivers odbc_query odbc_rollback odbc_begin_transaction odbc_begin_transaction(conn_handle BIGINT) ->
...tab1" (or with a schema name: "schema1"."tab1" ) dest_query ( VARCHAR ): query to be executed in remote DB for each source batch, must have the number of ODBC parameter placeholders ? equal to the source_columns_count * batch_size , cannot be specifie...d if dest_table is specified, example: CALL import_city(?,?,?,?) dest_query_single ( VARCHAR ): only used when the batch_size>0 and the number of rows read in the last source batch are less than the batch_size , in this case used instead of de...: pass (unsigned) TINYINT , SMALLINT , INTEGER and BIGINT parameters as SQL_C_NUMERIC . Optional named parameters (other): batch_size ( UINTEGER , default: 16 ): number of records to be inserted (or executed in case of dest_query ) in a single SQLExec... - https://man.mtool.work/duckdb/2.0/core_extensions/odbc/functions.html
- ODBC API on Linux 8767
- Driver Manager A driver manager is required to manage communication between applications and the ODBC driver. We tested and support unixODBC that is a complete ODBC driver manager for Linux. Users can install it from the command line: On De
...b_odbc.so : the DuckDB driver. unixodbc_setup.sh : a setup script to aid the configuration on Linux. To extract them, run: batch mkdir duckdb_odbc && unzip duckdb_odbc-linux-amd64.zip -d duckdb_odbc The unixodbc_setup.sh script performs the configurat...her option -u or -s to configure DuckDB ODBC. The -u option based on the user home directory to setup the ODBC init files. batch ./unixodbc_setup.sh -u The -s option changes the system level files that will be visible for all users, because of that it...requires root privileges. batch sudo ./unixodbc_setup.sh -s The option --help shows the usage of unixodbc_setup.sh prints the help. batch ./unixodbc_setup... - https://man.mtool.work/duckdb/2.0/clients/odbc/linux.html
- Conversion between DuckDB and Python 8767
- This page documents the rules for converting DuckDB results to Python and Python objects to DuckDB . Result Conversion: DuckDB Results to Python A result is read from a relation or, when using the DB-API / con.execute() , from the connectio
...urn relations. Each has its own methods: Result Relation DB-API 2.0 ( con.execute() ) Python tuples fetchall() , iter_rows(batch_size=1024) fetchone() , fetchmany(size=1) , fetchall() Python dicts to_dicts() NumPy fetchnumpy() fetchnumpy() Pandas df()...es vectors_per_chunk (1 by default) rows. Apache Arrow to_arrow_table() fetches the data as an Arrow table to_arrow_reader(batch_size) returns an Arrow record batch reader with at most batch_size rows per batch (the default is 131072) arrow() is an ol...r() first fetch the rows and then convert them, so they make an extra copy of the result, and the reader returns its first batch only after the whole query ran. They warn: UserWarning: Arrow output on a DB-API result makes an extra copy of the rows, u... - https://man.mtool.work/duckdb/2.0/clients/python/conversion.html
- Overview 8465
- How is DuckDB Tested? Testing is vital to make sure that DuckDB works properly and keeps working properly. For that reason, we put a large emphasis on thorough and frequent testing: * We run a batch of small tests on every commit using GitH
...ub Actions , and run a more exhaustive batch of tests on pull requests and commits in the main branch. * We use a fuzzer , which automatically reports issues found thr... - https://man.mtool.work/duckdb/2.0/dev/sqllogictest/overview.html
- Import Data 8226
- Overview The Rust client offers two ways to get data into DuckDB in bulk: the high-performance Appender for inserting rows from Rust, and reading directly from data files such as Parquet, CSV, and JSON. Both are described below. Appender Th
...he params! macro: use duckdb::{params, Connection, Result}; fn insert_rows(conn: &Connection) -> Result<()> { conn.execute_batch("CREATE TABLE foo (a INTEGER, b INTEGER)")?; let mut app = conn.appender("foo")?; app.append_row(params![1, 2])?; app.appe...arams, Connection, DropBehavior, Result}; fn main() -> Result<()> { let mut db = Connection::open_in_memory()?; db.execute_batch( "CREATE TABLE test (id INTEGER NOT NULL, area CHAR(6), age TINYINT NOT NULL, active TINYINT NOT NULL)", )?; let row_count...n), and Panic . Appending Arrow Data With the appender-arrow feature enabled, an Appender can append an Apache Arrow RecordBatch directly, which avoids building a parameter list per row. See Handle Results for the Arrow integration. Reading Data Files... - https://man.mtool.work/duckdb/2.0/clients/rust/data_import.html
DuckDB 2.0 Documentation / man.mtool.work