Results of 1 - 9 of about 9 for Parallelism (0.004 sec.)
- Tuning Workloads 13193
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...rve_insertion_order The preserve_insertion_order option Using all available cores Row group sizing and the threads setting Parallelism Processing data larger than memory Configuring the spill-to-disk directory Larger-than-memory workloads Diagnosing a slow...ucing memory usage. For details on which operations preserve row order and which do not, see the Order Preservation page . Parallelism (Multi-Core Processing) The Effect of Row Groups on Parallelism DuckDB parallelizes the workload based on row groups , i.e...ows that are stored together at the storage level. The default row group size in DuckDB's database format is 122,880 rows. Parallelism starts at the level of row groups, therefore, for a query to run on k threads, it needs to scan at least k * 122,880 rows.... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- Avro Extension 11018
- The avro extension enables DuckDB to read Apache Avro files. The avro extension was released as a community extension in late 2024 and became a core extension in early 2025. The read_avro Function The extension adds a single DuckDB function
...ing to allow reading of Avro files from memory. Limitations and Future Plans This extension currently does not make use of parallelism when reading either a single (large) Avro file or when reading a list of files. Adding support for parallelism in the latt... - https://man.mtool.work/duckdb/2.0/core_extensions/avro.html
- Pragmas 8843
- The PRAGMA statement is a SQL extension adopted by DuckDB from SQLite. PRAGMA statements can be issued in a similar manner to regular SQL statements. PRAGMA commands may alter the internal state of the database engine, and can influence the
...rification; Disable query verification: PRAGMA disable_verification; Enable force parallel query processing: PRAGMA verify_parallelism; Disable force parallel query processing: PRAGMA disable_verify_parallelism; Block Sizes When persisting a database to dis... - https://man.mtool.work/duckdb/2.0/configuration/pragmas.html
- I/O 8358
- Warning DuckDB ships with safe defaults. The configuration options described on this page are advanced options, so proceed with caution when changing them. Asynchronous I/O New Asynchronous I/O was introduced in DuckDB 2.0. Starting with v2
...reads = 48; SET http_retries = 8; SET http_retry_wait_ms = 50; SET http_retry_backoff = 2; As the row group is the unit of parallelism for Parquet scans, asynchronous I/O works best if Parquet files have at least as many row groups as the number of threads.... - https://man.mtool.work/duckdb/2.0/data/io.html
- File Formats 8358
- Handling Parquet Files DuckDB has advanced support for Parquet files, which includes directly querying Parquet files . When deciding on whether to query these files directly or to first load them to the database, you need to consider severa
...r example, with a machine having 10 threads, both 10 files with 1 row group or 1 file with 10 row groups will achieve full parallelism. It is also beneficial to keep the size of individual Parquet files moderate. Bestpractice The ideal range is between 100... - https://man.mtool.work/duckdb/2.0/guides/performance/file_formats.html
- Storage Versions and Format 8124
- Compatibility Backward Compatibility Backward compatibility refers to the ability of a newer DuckDB version to read storage files created by an older DuckDB version. Version 0.10 is the first release of DuckDB that supports backward compati
...izontal partitions of the data. This concept is equivalent to Parquet's row groups . Several features in DuckDB, including parallelism and compression are based on row groups. The row group size can be specified as an option of the ATTACH statement: ATTACH... - https://man.mtool.work/duckdb/2.0/internals/storage.html
- Vector Similarity Search Extension 7873
- The vss extension is an experimental extension for DuckDB that adds indexing support to accelerate vector similarity search queries using DuckDB's new fixed-size ARRAY type. See the announcement blog post and the “What's New in the Vector S
...'s faster to create the index after the table has been populated with data as the initial bulk load can make better use of parallelism on large tables. Deletes are not immediately reflected in the index, but are instead “marked” as deleted, which can cause... - https://man.mtool.work/duckdb/2.0/core_extensions/vss.html
- Connect 7639
- Overview Every DuckDB feature reachable from Java starts with a Connection . This page covers the JDBC URL forms the driver accepts, how DuckDB and driver options are passed and which of them win, how connections come to share (or not share
...) call can occupy every thread in the pool, and blocking JDBC calls from many application threads do not increase DuckDB's parallelism. The pool belongs to the database instance , not to the Connection . Every connection to the same instance, whether create... - https://man.mtool.work/duckdb/2.0/clients/java/connecting.html
- Spatial Functions 7387
- Function Index Scalar Functions Function Summary DuckDB_PROJ_Compiled_Version Returns a text description of the PROJ library version that this instance of DuckDB was compiled against. DuckDB_Proj_Version Returns a text description of the PR
...g in the GEOMETRY type yet. Note that GDAL is single-threaded, so this table function will not be able to make full use of parallelism. By using ST_Read , the spatial extension also provides “replacement scans” for common geospatial file formats, allowing y... - https://man.mtool.work/duckdb/2.0/core_extensions/spatial/functions.html
PREV
1
NEXT
DuckDB 2.0 Documentation / man.mtool.work