Results of 1 - 10 of about 27 for processing (0.002 sec.)
- Tuning Workloads 14154
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...n_order The preserve_insertion_order option Using all available cores Row group sizing and the threads setting Parallelism Processing data larger than memory Configuring the spill-to-disk directory Larger-than-memory workloads Diagnosing a slow query Inspe...details on which operations preserve row order and which do not, see the Order Preservation page . Parallelism (Multi-Core Processing) The Effect of Row Groups on Parallelism DuckDB parallelizes the workload based on row groups , i.e., groups of rows that...ases, it's worth manually limiting the number of threads using SET threads = X . Larger-than-Memory Workloads (Out-of-Core Processing) A key strength of DuckDB is support for larger-than-memory workloads, i.e., it is able to process datasets that are large... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- EXPLAIN ANALYZE: Profile Queries 10418
- Prepending a query with EXPLAIN ANALYZE both pretty-prints the query plan, and executes it, providing run-time performance numbers and the actual cardinality (number of rows produced) for every operator. EXPLAIN ANALYZE SELECT * FROM tbl; N
...ote that the cumulative wall-clock time that is spent on every operator is shown. When multiple threads are processing the query in parallel, the total processing time of the query may be lower than the sum of all the times spent on the indi... - https://man.mtool.work/duckdb/2.0/guides/meta/explain_analyze.html
- PostgreSQL Extension Connection Pool 10187
- PostgreSQL server spawns a backend process for every incoming client connection. This model leads to the two following points with the postgres extension that can cause performance problems: opening a new connection is relatively expensive
...ept open (and idle) after use and can be reused for subsequent queries. Parallel Scans Due to DuckDB's multithreaded query processing, postgres extension can open multiple connections even for queries that performs a scan for a single PostgreSQL table. The...stance ( threads global configuration option). When the threads parameter is set too high (that may be necessary for local processing of other queries on DuckDB side) this can lead to attempts to open an excessive number of PostgreSQL connections. To preve...le (the maximum number of connection in the pool is exceeded) - then that worked thread does not participate futher in the processing of this table scan, that proceeds with the worker threads that were able to acquire a connection from the pool. See the de... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/connection_pool.html
- Spatial Extension 9726
- The spatial extension provides support for geospatial data processing in DuckDB. For an overview of the extension, see our blog post . Installing and Loading To install the spatial extension, run: INSTALL spatial; Note that the spatial exte
... - https://man.mtool.work/duckdb/2.0/core_extensions/spatial/overview.html
- Performance Guide 9726
- DuckDB aims to automatically achieve high performance by using well-chosen default configurations and having a forgiving architecture. Of course, there are still opportunities for tuning the system for specific workloads. The Performance Gu
...ide's pages contain guidelines and tips for achieving good performance when loading and processing data with DuckDB. The guides include several microbenchmarks. You may find details about these on the Benchmarks page .... - https://man.mtool.work/duckdb/2.0/guides/performance/overview.html
- Command Line Arguments 9019
- The table below summarizes DuckDB's command line options. To list all command line options, use the command: duckdb -help For a list of dot commands available in the CLI shell, see the Dot Commands page . Argument Description -agent Render
...put row separator. Default: \n -no-agent Never render output for an AI coding agent. See the Agent Mode page -no-init Skip processing the init file ( ~/.duckdbrc or the file passed to -init ) -noheader Turn headers off -no-stdin Exit after processing optio... - https://man.mtool.work/duckdb/2.0/clients/cli/arguments.html
- Indexes 9019
- Index Types DuckDB has two built-in index types. Indexes can also be defined via extensions . Min-Max Index (Zonemap) A min-max index (also known as zonemap or block range index) is automatically created for columns of all general-purpose d
...ART Indexes ART indexes create a secondary copy of the data in a second location. Maintaining that second copy complicates processing. Thus, certain limitations currently apply when it comes to modifying data that is also stored in secondary indexes. As ex...written into a DELETE , followed by an INSERT . This rewrite happens per chunk of data (2048 rows) moving through DuckDB's processing pipeline. When updating i = 2047 to i = 2048 , we do not yet know that 2048 becomes 2049, and so forth. That is because we... - https://man.mtool.work/duckdb/2.0/sql/indexes.html
- Schema 8788
- Types It is important to use the correct type for encoding columns (e.g., BIGINT , DATE , DATETIME ). While it is always possible to use string types ( VARCHAR , etc.) to encode more specific values, this is not recommended. Strings use mor
...s small values stored in larger data types will not take up larger sizes on disk, but they will take up more memory during processing – hence, using the most restrictive types possible when creating columns is necessary to reduce memory consumption. Microb...E 3.3 GB 0.9 s VARCHAR 5.2 GB 3.9 s The results show that using the DATETIME value yields smaller storage sizes and faster processing. Microbenchmark: Joining on Strings We illustrate the difference caused by joining on different types by computing a self-... - https://man.mtool.work/duckdb/2.0/guides/performance/schema.html
- I/O 8558
- Warning DuckDB ships with safe defaults. The configuration options described on this page are advanced options, so proceed with caution when changing them. Asynchronous I/O New Asynchronous I/O was introduced in DuckDB 2.0. Starting with v2
.... Thread Pools DuckDB uses two thread pools: The regular pool contains the worker threads, which perform the actual query processing (e.g., decoding, joins and aggregations). Its size is controlled by the threads setting and defaults to the number of CPU...cores. Regular threads prioritize query processing but can also perform I/O tasks when idle. The asynchronous pool contains threads dedicated to blocking I/O. As these threa... - https://man.mtool.work/duckdb/2.0/data/io.html
- Pragmas 8327
- The PRAGMA statement is a SQL extension adopted by DuckDB from SQLite. PRAGMA statements can be issued in a similar manner to regular SQL statements. PRAGMA commands may alter the internal state of the database engine, and can influence the
...rmation in raw JSON format. For certain errors, extra information or decomposed information is provided for easier machine processing. For example: SET errors_as_json = true; Then, running a query that results in an error produces a JSON output: SELECT * F...fication: PRAGMA enable_verification; Disable query verification: PRAGMA disable_verification; Enable force parallel query processing: PRAGMA verify_parallelism; Disable force parallel query processing: PRAGMA disable_verify_parallelism; Block Sizes When p... - https://man.mtool.work/duckdb/2.0/configuration/pragmas.html
DuckDB 2.0 Documentation / man.mtool.work