Results of 1 - 10 of about 26 for parallel (0.002 sec.)
- PostgreSQL Extension Connection Pool 14530
- PostgreSQL server spawns a backend process for every incoming client connection. This model leads to the two following points with the postgres extension that can cause performance problems: opening a new connection is relatively expensive
...pool. With it a limited number of connections are kept open (and idle) after use and can be reused for subsequent queries. Parallel Scans Due to DuckDB's multithreaded query processing, postgres extension can open multiple connections even for queries th...at performs a scan for a single PostgreSQL table. The number of parallel queries running for a single table scan can be as high as the number of worker threads of the current DB instance ( thread...es on DuckDB side) this can lead to attempts to open an excessive number of PostgreSQL connections. To prevent this, these parallel queries (for the same table scan) are additionally limited by the maximum number of allowed connection in the connection p... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/connection_pool.html
- Write User Defined Functions 10199
- Overview The JDBC driver can register user-defined functions (UDFs) written in Java, including scalar functions and table functions. This page covers building and registering both, from the simple functional-interface overloads to the vecto
...s the global state. localInit (optional) runs once per execution thread and returns thread-local state. Override it when a parallel scan needs per-thread state; the default returns null . apply writes rows into the output vectors and returns the number o...it throws is not propagated to the query. A state that only implements AutoCloseable , or a null state, is left untouched. Parallel Execution To run a table function on multiple threads, set the thread count with info.setMaxThreads() in the init callback.... Parallel output may also require the preserve_insertion_order option set to false , in the connection string or the connection prop... - https://man.mtool.work/duckdb/2.0/clients/java/functions.html
- Querying Parquet Files 10199
- To run a query directly on a Parquet file, use the read_parquet function in the FROM clause of a query. SELECT * FROM read_parquet('input.parquet'); The Parquet file will be processed in parallel. Filters will be automatically pushed down i
... - https://man.mtool.work/duckdb/2.0/guides/file_formats/query_parquet.html
- Python API 9951
- Installation To use the DuckDB Python client, visit the Python installation page . The latest stable version of the DuckDB Python client is 2.0.0. Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please s
...lobal database, which can cause hard to debug issues if used from within multiple different packages. Using Connections in Parallel Python Programs A connection runs one call at a time. A call from another thread while the connection is busy does not wai...odule and duckdb.connect(":default:") all use one shared default connection, so the same holds for them. To run queries in parallel, give each thread its own cursor: import threading import duckdb con = duckdb.connect() con.execute("CREATE TABLE numbers...tabase. A cursor has its own transaction and its own session settings, such as search_path , so cursors can run queries in parallel. A global setting, such as threads or memory_limit , applies to the whole database, also when a cursor sets it. Closing a... - https://man.mtool.work/duckdb/2.0/clients/python/overview.html
- Multiple Python Threads 9951
- This page demonstrates how to simultaneously insert into and read from a DuckDB database across multiple Python threads. This could be useful in scenarios where new data is flowing in and an analysis should be periodically re-run. Note that
...e consistent results when testing random.shuffle(threads) Run Threads and Show Results Now, kick off all threads to run in parallel, then wait for all of them to finish before printing out the results. Note that the timestamps of readers and writers are...interspersed as expected due to the randomization. # Kick off all threads in parallel for thread in threads: thread.start() # Ensure all threads complete before printing final results for thread in threads: t... - https://man.mtool.work/duckdb/2.0/guides/python/multiple_threads.html
- Pragmas 8926
- The PRAGMA statement is a SQL extension adopted by DuckDB from SQLite. PRAGMA statements can be issued in a similar manner to regular SQL statements. PRAGMA commands may alter the internal state of the database engine, and can influence the
...refore, the actual memory consumption can be higher than the specified memory limit. Threads Set the amount of threads for parallel query execution: SET threads = 4; Maximum Execution Time New As of DuckDB 2.0, the max_execution_time setting bounds the w...able query verification: PRAGMA enable_verification; Disable query verification: PRAGMA disable_verification; Enable force parallel query processing: PRAGMA verify_parallelism; Disable force parallel query processing: PRAGMA disable_verify_parallelism; B... - https://man.mtool.work/duckdb/2.0/configuration/pragmas.html
- My Workload Is Slow 8926
- If you find that your workload in DuckDB is slow, we recommend performing the following checks. More detailed instructions are linked for each point. Do you have enough memory? DuckDB works best if you have 1-4 GB memory per thread . Is you
...and file sizes between 100 MB to 10 GB? Does the query plan look right? Study it with EXPLAIN . Is the workload running in parallel ? Use htop or the operating system's task manager to observe this. Is DuckDB using too many threads? Try limiting the amou... - https://man.mtool.work/duckdb/2.0/guides/performance/my_workload_is_slow.html
- PostgreSQL Extension Functions 8678
- postgres_clear_cache postgres_configure_pool postgres_execute postgres_hstore_get postgres_hstore_to_json postgres_query read_postgres_binary postgres_clear_cache postgres_clear_cache() -> TABLE Clears cached schema entries (like table name
...allowed to be cached in a connection pool for each attached Postgres database. This number can be temporary exceeded when parallel scans are used. wait_timeout_millis ( UBIGINT ): maximum number of milliseconds to wait when acquiring a connection from a...y acquire-mode and no connection was available at the time - so no connection was provided by the pool; note that when the parallel scans are performed by the Postgres extension - worker threads always use the try acquire mode; when SET threads = ⟨number... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/functions.html
- EXPLAIN ANALYZE: Profile Queries 8430
- Prepending a query with EXPLAIN ANALYZE both pretty-prints the query plan, and executes it, providing run-time performance numbers and the actual cardinality (number of rows produced) for every operator. EXPLAIN ANALYZE SELECT * FROM tbl; N
...the cumulative wall-clock time that is spent on every operator is shown. When multiple threads are processing the query in parallel, the total processing time of the query may be lower than the sum of all the times spent on the individual operators. For... - https://man.mtool.work/duckdb/2.0/guides/meta/explain_analyze.html
- CSV Import 8166
- Examples The following examples use the flights.csv file. Read a CSV file from disk, auto-infer options: SELECT * FROM 'flights.csv'; Use the read_csv function with custom options: SELECT * FROM read_csv('flights.csv', delim = '|', header =
.... Matching against these strings can be suppressed for specific columns with force_not_null . VARCHAR or VARCHAR[] (empty) parallel Use the parallel CSV reader. BOOL true quote String used to quote values. VARCHAR " rejects_scan Name of the temporary tab... - https://man.mtool.work/duckdb/2.0/data/csv/overview.html
DuckDB 2.0 Documentation / man.mtool.work