Results of 1 - 10 of about 37 for reader (0.002 sec.)
- Node.js Client (Neo) 14396
- Installation To use the DuckDB Node.js client, visit the Node.js installation page . The latest stable version of the DuckDB Node.js (Neo) client is 2.0.0. An API for using DuckDB in Node.js . The primary package, @duckdb/node-api , is a hi
...that case, all of the statements are executed in order, and the returned result is the result of the last statement: const reader = await connection.runAndReadAll( `create table t (id integer); insert into t values (1); from t;` ); reader.getRows(); //...olumnNames = result.columnNames(); const columnTypes = result.columnTypes(); Read Result Data Run and read all data: const reader = await connection.runAndReadAll('from test_all_types()'); const rows = reader.getRows(); // OR: const columns = reader.ge...tColumns(); Stream and read up to (at least) some number of rows: const reader = await connection.streamAndReadUntil( 'from range(5000)', 1000 ); const rows = reader.getRows(); // rows.length === 2048.... - https://man.mtool.work/duckdb/2.0/clients/node_neo/overview.html
- Export to Apache Arrow 12912
- All results of a query can be exported to an Apache Arrow Table using the to_arrow_table function. Alternatively, results can be returned as a RecordBatchReader using the to_arrow_reader function and results can be read one batch at a time.
...Call these functions on the relation that duckdb.sql , con.sql or DuckDB's Relational API returns: its reader streams the result. On a con.execute() result they first copy all rows and warn with a UserWarning , and the reader does n...ot stream. While a reader is partly read, other queries on its connection raise a ResourceInUseException . Read the reader to the end or close it fi...er queries on a cursor from con.cursor() . Deprecated On relations, fetch_arrow_table , fetch_record_batch and fetch_arrow_reader are deprecated aliases of to_arrow_table and to_arrow_reader . Results of con.execute() and the duckdb module no longer ha... - https://man.mtool.work/duckdb/2.0/guides/python/export_arrow.html
- Handle Results 9897
- Overview DuckDB is columnar, and beyond reading a result set row by row through database/sql , the Go client can exchange whole results with Apache Arrow as a stream of record batches, and register an Arrow stream as a queryable view. Both
...raw DuckDB connection with duckdb.NewArrowFromConn() , then call QueryContext() to run a query and get back an array.RecordReader . Iterate it with Next() , reading one arrow.RecordBatch at a time, and release it when done: connector, err := duckdb.New...err != nil { log.Fatal(err) } defer conn.Close() ar, err := duckdb.NewArrowFromConn(conn) if err != nil { log.Fatal(err) } reader, err := ar.QueryContext(context.Background(), "SELECT * FROM generate_series(1, 10)") if err != nil { log.Fatal(err) } def...er reader.Release() for reader.Next() { record := reader.RecordBatch() // Process one record batch at a time. log.Printf("%d rows",... - https://man.mtool.work/duckdb/2.0/clients/go/result_handling.html
- CSV Import 9282
- Examples The following examples use the flights.csv file. Read a CSV file from disk, auto-infer options: SELECT * FROM 'flights.csv'; Use the read_csv function with custom options: SELECT * FROM read_csv('flights.csv', delim = '|', header =
...loading them a challenge. CSV files come in many different varieties, are often corrupt, and do not have a schema. The CSV reader needs to cope with all of these different situations. The DuckDB CSV reader can automatically infer which configuration fl.... This will work correctly in most situations, and should be the first option attempted. In rare situations where the CSV reader cannot figure out the correct configuration it is possible to manually configure the CSV reader to correctly parse the CSV...gs can be suppressed for specific columns with force_not_null . VARCHAR or VARCHAR[] (empty) parallel Use the parallel CSV reader. BOOL true quote String used to quote values. VARCHAR " rejects_scan Name of the temporary table where information on faul... - https://man.mtool.work/duckdb/2.0/data/csv/overview.html
- Integration with PyTorch 9140
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
..., 0], } ) pq.write_to_dataset(table, str(parquet_dir)) con = duckdb.connect() train_dataset = ds.dataset(str(parquet_dir)) reader = con.sql(""" SELECT feature_0, feature_1, label FROM train_dataset WHERE label = 1 """).to_arrow_reader(batch_size=2) for...batch in reader: features = torch.tensor( np.column_stack( [ batch.column(0).to_numpy(), batch.column(1).to_numpy(), ] ), dtype=torch.floa...the selected columns into the dataset scan, so only the requested rows and columns are converted to tensors. The to_arrow_reader method of a relation streams the result in batches instead of materializing the full result in memory at once. While the l... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
- Conversion between DuckDB and Python 8982
- This page documents the rules for converting DuckDB results to Python and Python objects to DuckDB . Result Conversion: DuckDB Results to Python A result is read from a relation or, when using the DB-API / con.execute() , from the connectio
...n. con.sql() , con.table() , the readers and the other relational methods return relations. Each has its own methods: Result Relation DB-API 2.0 ( con.execute() )...ts to_dicts() NumPy fetchnumpy() fetchnumpy() Pandas df() df() , fetch_df_chunk() Apache Arrow to_arrow_table() , to_arrow_reader() to_arrow_table() , to_arrow_reader() , both make a copy Polars pl() PyTorch torch() torch() TensorFlow tf() tf() Relatio...self. To get specific rows of a result with the relational API you would use limit(n).fetchall() , iter_rows() or to_arrow_reader() instead. Note that with the DB-API, after a partial read with fetchone() or fetchmany() , the df() , fetchnumpy() and Ar... - https://man.mtool.work/duckdb/2.0/clients/python/conversion.html
- Handle Results 8461
- Overview Beyond the standard JDBC ResultSet , the DuckDB driver can hand results to Apache Arrow , stream large results instead of materializing them, and expose raw columnar data chunks. This page covers each of these result-handling optio
...are read from the VectorSchemaRoot . import org.apache.arrow.memory.RootAllocator; import org.apache.arrow.vector.ipc.ArrowReader; import org.duckdb.DuckDBResultSet; try (var conn = DriverManager.getConnection("jdbc:duckdb:"); var stmt = conn.prepareSt...ate_series(2000)"); var resultset = (DuckDBResultSet) stmt.executeQuery(); var allocator = new RootAllocator()) { try (var reader = (ArrowReader) resultset.arrowExportStream(allocator, 256)) { while (reader.loadNextBatch()) { System.out.println(reader....mport org.apache.arrow.c.Data; import org.apache.arrow.memory.RootAllocator; import org.apache.arrow.vector.ipc.ArrowStreamReader; import org.duckdb.DuckDBConnection; import org.duckdb.DuckDBResultSet; // Arrow binding try (var allocator = new RootAllo... - https://man.mtool.work/duckdb/2.0/clients/java/result_handling.html
- Migrating Python Code to DuckDB 2.0 8366
- This page lists what Python code written for DuckDB 1.x must change to run on DuckDB 2.0. Each item gives the 1.x form, the 2.0 form and the page that explains it. For changes outside the Python client, see Migrating to DuckDB v2.0 . Python
..._record_batch() , arrow() and pl() . 2.0: these methods are gone from execute() results, and to_arrow_table() and to_arrow_reader() there copy the rows and warn. Use a relation: con.sql(query).to_arrow_table() , con.sql(query).to_arrow_reader() or con....for row in rel.iter_rows(): ... ; this keeps the complete result in engine memory. For streamed batches, use rel.to_arrow_reader() . Replace rel.execute() with rel.materialize() when you want to compute and retain the rows. Plain strings in expression...me in batches of 1000000 rows by default. 2.0: the default is 131072 rows. Pass batch_size to to_arrow_table() or to_arrow_reader() for another size. See to_arrow_reader . Database files. 1.x: a database file that DuckDB 1.5 wrote opens in DuckDB 1.5.... - https://man.mtool.work/duckdb/2.0/clients/python/migration.html
- Multiple Python Threads 8271
- This page demonstrates how to simultaneously insert into and read from a DuckDB database across multiple Python threads. This could be useful in scenarios where new data is flowing in and an analysis should be periodically re-run. Note that
...ecute(""" CREATE OR REPLACE TABLE my_inserts ( thread_name VARCHAR, insert_time TIMESTAMP DEFAULT current_timestamp ) """) Reader and Writer Functions Next, define functions to be executed by the writer and reader threads. Each thread must use the .cur...AS read_time FROM my_inserts """, (thread_name,)).fetchall() print(results) Create Threads We define how many writers and readers to use, and define a list to track all of the threads that will be created. Then, create first writer and then reader thr...eads. Next, shuffle them so that they will be kicked off in a random order to simulate simultaneous writers and readers. Note that the threads have not yet been executed, only defined. write_thread_count = 50 read_thread_count = 5 threads =... - https://man.mtool.work/duckdb/2.0/guides/python/multiple_threads.html
- Order Preservation 8129
- For many operations, DuckDB preserves the order of rows, similarly to data frame libraries such as Pandas. Example Take the following table for example: CREATE TABLE tbl AS SELECT * FROM (VALUES (1, 'a'), (2, 'b'), (3, 'c')) t(x, y); SELECT
...a stable algorithm ) Scalar subqueries Insertion Order By default, the following components preserve insertion order: CSV reader ( read_csv function) JSON reader ( read_json function) Parquet reader ( read_parquet function) The preserve_insertion_orde... - https://man.mtool.work/duckdb/2.0/sql/dialect/order_preservation.html
DuckDB 2.0 Documentation / man.mtool.work