Results of 1 - 10 of about 48 for Thread (0.003 sec.)
- Multiple Python Threads 14010
- This page demonstrates how to simultaneously insert into and read from a DuckDB database across multiple Python threads. This could be useful in scenarios where new data is flowing in and an analysis should be periodically re-run. Note that
...rary. Note: if using Pandas, add import pandas at the top of the script as well (as it must be imported prior to the multi-threading). Then connect to a file-backed DuckDB database and create an example table to store inserted data. This table will tra...ck the name of the thread that completed the insert and automatically insert the timestamp when that insert occurred using the DEFAULT expression ....import duckdb from threading import Thread, current_thread import random duckdb_con = duckdb.connect('my_persistent_db.duckdb') # Use connect withou... - https://man.mtool.work/duckdb/2.0/guides/python/multiple_threads.html
- PostgreSQL Extension Connection Pool 12874
- PostgreSQL server spawns a backend process for every incoming client connection. This model leads to the two following points with the postgres extension that can cause performance problems: opening a new connection is relatively expensive
...nnections are kept open (and idle) after use and can be reused for subsequent queries. Parallel Scans Due to DuckDB's multithreaded query processing, postgres extension can open multiple connections even for queries that performs a scan for a single Po...stgreSQL table. The number of parallel queries running for a single table scan can be as high as the number of worker threads of the current DB instance ( threads global configuration option). When the threads parameter is set too high (that may b...ode. If connection for them is not available (the maximum number of connection in the pool is exceeded) - then that worked thread does not participate futher in the processing of this table scan, that proceeds with the worker threads that were able to... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/connection_pool.html
- PostgreSQL Extension Functions 11814
- postgres_clear_cache postgres_configure_pool postgres_execute postgres_hstore_get postgres_hstore_to_json postgres_query read_postgres_binary postgres_clear_cache postgres_clear_cache() -> TABLE Clears cached schema entries (like table name
...of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken. enable_thread_local_cache ( BOOLEAN ): whether to enable the connection caching in thread-local cache. Such connections are getting pinn...ed to the threads and are not made available to other threads, while still taking the place in the pool. max_lifetime_millis ( UBIGINT ): m...his number is checked when the connection is taken from the pool and returned to the pool. When the connection pool reaper thread is enabled ( enable_reaper_thread argument), then this number is checked in background periodically. idle_timeout_millis (... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/functions.html
- Parquet Tips 9193
- Below is a collection of tips to help when dealing with Parquet files. Tips for Reading Parquet Files Use union_by_name When Loading Files with Different Schemas The union_by_name option can be used to unify the schema of files that have di
...glob pattern upon read or a Hive partitioning structure are good ways to transparently handle multiple files. Enabling PER_THREAD_OUTPUT If the final number of Parquet files is not important, writing one file per thread can significantly improve perfor...mance: COPY (FROM generate_series(10_000_000)) TO 'test.parquet' (FORMAT parquet, PER_THREAD_OUTPUT); Selecting a ROW_GROUP_SIZE The ROW_GROUP_SIZE parameter specifies the minimum number of rows in a Parquet row gro...the row group size, the more opportunities to compress the data. On the other hand, larger row group sizes mean that each thread keeps more data in memory before flushing when streaming results. Another argument for smaller row group sizes is that Duc... - https://man.mtool.work/duckdb/2.0/data/parquet/tips.html
- Profile and Monitor 8739
- Overview The Rust client can observe a running or completed query: it can retrieve per-operator profiling metrics after a query runs, and interrupt a long-running query from another thread. Both are shown below. Profiling Profiling must be
...ts metrics map and its children through children . Interrupting Queries A long-running query can be cancelled from another thread. Call interrupt_handle() on the connection to get an Arc<InterruptHandle> . InterruptHandle is both Send and Sync , so you...can move it, or the connection, to another thread and call interrupt() from wherever is convenient. The interrupted query fails with an error: use duckdb::{Connection, Resu...lt}; fn run_query(conn: Connection) -> Result<()> { let interrupt_handle = conn.interrupt_handle(); let join_handle = std::thread::spawn(move || { conn.execute("⟨expensive query⟩", []) }); // ... later, from this thread, cancel the query: interrupt_han... - https://man.mtool.work/duckdb/2.0/clients/rust/profiling.html
- Connect 8512
- Overview Every DuckDB feature reachable from Java starts with a Connection . This page covers the JDBC URL forms the driver accepts, how DuckDB and driver options are passed and which of them win, how connections come to share (or not share
...) an underlying database instance, how DuckDB's threads relate to that instance, how to open or attach a DuckLake , and how to reach a remote DuckDB server over Quack . Opening...ly way to configure DuckDB in tools that accept a connection string but no Properties object: jdbc:duckdb:/tmp/my_database;threads=4;memory_limit=4GB;jdbc_stream_results=true Each entry is a key=value pair and surrounding whitespace is trimmed. The dri...the same option appears in both places, the value in the URL wins: Properties props = new Properties(); props.setProperty("threads", "8"); // The connection is opened with threads = 4. Connection conn = DriverManager.getConnection("jdbc:duckdb:/tmp/my_... - https://man.mtool.work/duckdb/2.0/clients/java/connecting.html
- Python API 8194
- Installation To use the DuckDB Python client, visit the Python installation page . The latest stable version of the DuckDB Python client is 2.0.0. Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please s
...ckDB using conda : conda install python-duckdb -c conda-forge . Python version: DuckDB requires Python 3.11 or newer. Free-threaded builds of CPython are not supported. If you are upgrading from DuckDB 1.x, see Migrating Python Code to DuckDB 2.0 . Bas...config dictionary, where configuration options can be specified. For example: import duckdb con = duckdb.connect(config={"threads": 1}) Besides its worker threads, each database starts a pool of asynchronous I/O threads : by default four times the num...ber of CPU cores, at most 256. To start fewer, set async_threads : con = duckdb.connect(config={"threads": 4, "async_threads": 4}) To specify the storage version , pass the storage_compa... - https://man.mtool.work/duckdb/2.0/clients/python/overview.html
- Connect 8148
- Overview Every DuckDB feature reachable from Rust starts with a Connection . Opening in-memory and file-backed databases, configuring a database instance with Config , opening read-only, pooling connections across threads, the crate's thread
...DuckDB validates the option: use duckdb::{Config, Connection, Result}; let config = Config::default() .max_memory("4GB")? .threads(4)?; let conn = Connection::open_with_flags("my_database.duckdb", config)?; Config exposes typed methods for the most com...mon options, including access_mode() , max_memory() , threads() , default_order() , default_null_order() , enable_external_access() , enable_object_cache() , custom_user_agent() , and...h_flags("my_database.duckdb", config)?; AccessMode has three variants: Automatic (the default), ReadOnly , and ReadWrite . Thread Safety A Connection is Send but not Sync : it can be moved to another thread, but it cannot be shared between threads at t... - https://man.mtool.work/duckdb/2.0/clients/rust/connecting.html
- I/O 7997
- Warning DuckDB ships with safe defaults. The configuration options described on this page are advanced options, so proceed with caution when changing them. Asynchronous I/O New Asynchronous I/O was introduced in DuckDB 2.0. Starting with v2
....0, DuckDB supports asynchronous I/O. Instead of blocking a worker thread until requested data arrives, DuckDB issues reads in the background and keeps multiple requests in flight while worker thr...is used automatically for the supported formats, including when they are read through data lake formats such as DuckLake . Thread Pools DuckDB uses two thread pools: The regular pool contains the worker threads, which perform the actual query processin...g (e.g., decoding, joins and aggregations). Its size is controlled by the threads setting and defaults to the number of CPU cores. Regular threads prioritize query processing but can also perform I/O tas... - https://man.mtool.work/duckdb/2.0/data/io.html
- Profile and Monitor 7861
- Overview The JDBC driver exposes several ways to observe a running or completed query: cancelling and timing out queries, polling query progress, retrieving profiling output, and emitting memory-usage events through Java Flight Recorder . T
...his page covers each of them. Query Progress and Cancellation A running query can be cancelled from another thread with the standard JDBC Statement.cancel() method, which interrupts the query executing on the statement's connection. Sett....getQueryProgress() returns a QueryProgress snapshot while the query executes. It is intended to be called from a separate thread and exposes getPercentage() , getRowsProcessed() , and getTotalRowsToProcess() . import java.sql.DriverManager; import jav...BPreparedStatement) conn.prepareStatement("SELECT count(*) FROM range(1_000_000_000)")) { // Run the query on a background thread so its progress can be polled while it executes. CompletableFuture<Void> query = CompletableFuture.runAsync(() -> { try {... - https://man.mtool.work/duckdb/2.0/clients/java/profiling.html
DuckDB 2.0 Documentation / man.mtool.work