Results of 1 - 10 of about 13 for intermediate (0.002 sec.)
- Metrics 14850
- DuckDB provides a set of metrics that can be used to monitor the performance and health of the database. Metrics are named ⟨group⟩.⟨metric⟩ , e.g., query.total_time or operator.timing . The set of collected metrics is controlled with the tr
...ription query.cpu_time double seconds CPU time spent on the query query.sql string The SQL string of the query query.total_intermediate_rows uint64 rows Cumulative number of intermediate rows produced by the query query.total_intermediate_size_bytes uint64 b...ytes Cumulative size in bytes of all intermediate results produced by the query query.total_row_groups_scanned uint64 row_groups Cumulative number of row groups scanned by...n operator.cpu_time double seconds CPU time spent in the operator operator.extra_info map Unique operator metrics operator.intermediate_rows uint64 rows Number of intermediate rows produced by the operator operator.intermediate_size_bytes uint64 bytes Interm... - https://man.mtool.work/duckdb/2.0/dev/metrics.html
- Azure Extension 8657
- The azure extension lets DuckDB read and write files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2. Installing and Loading The azure extension will be transparently autoloaded on first use from the official extension reposit
...er the write completes. To commit earlier, set azure_write_staged_blocks_per_commit to the number of blocks after which an intermediate commit happens. Warning An intermediate commit makes an incomplete file visible to readers. A Parquet file cannot be read...ee Writing Large Files . UBIGINT 8*1024*1024 azure_write_staged_blocks_per_commit Number of uploaded blocks after which an intermediate commit happens. With 0 , all blocks are committed when the file is closed. Non-zero values make partial writes visible soo... - https://man.mtool.work/duckdb/2.0/core_extensions/azure.html
- Profile and Monitor 8498
- Overview The Rust client can observe a running or completed query: it can retrieve per-operator profiling metrics after a query runs, and interrupt a long-running query from another thread. Both are shown below. Profiling Profiling must be
...lowercase (see the metrics list ): the root's children are the root plan operator, with metrics such as type , timing and intermediate_rows , followed by one node per metric group, such as query with total_time . Read a node's metrics through its metrics ma... - https://man.mtool.work/duckdb/2.0/clients/rust/profiling.html
- Tuning Workloads 8498
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...hat are larger than the available system memory (also known as out-of-core processing ). It can also run queries where the intermediate results cannot fit into memory. This section explains the prerequisites, scope, and known limitations of larger-than-memor...rting are holistic, i.e., they need all inputs before the aggregation can start. As DuckDB cannot yet offload some complex intermediate aggregate states to disk, these functions can cause an out-of-memory exception when run on large datasets. Profiling If yo... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- Appender 8340
- Appenders are the most efficient way of loading data into DuckDB from within the C interface, and are recommended for fast data loading. The appender is much faster than using prepared statements or individual INSERT INTO statements. Append
...Return Value DuckDBSuccess on success or DuckDBError on failure. duckdb_appender_close Closes the appender by flushing all intermediate states and closing it for further appends. If flushing the data triggers a constraint violation or any other error, then a...turn Value DuckDBSuccess on success or DuckDBError on failure. duckdb_appender_destroy Closes the appender by flushing all intermediate states to the table and destroying it. By destroying it, this function de-allocates all memory associated with the appende... - https://man.mtool.work/duckdb/2.0/clients/c/appender.html
- PEG Parser 8340
- New Starting with DuckDB 2.0, the PEG parser is the default parser. In earlier versions it was an experimental, opt-in feature enabled with enable_peg_parser() , which is no longer required or available. DuckDB uses a parser based on PEG (
...you can expect extensions that expose entirely new SQL syntax. Performance The parser uses packrat parsing, which memoizes intermediate results to avoid exponential backtracking on deeply nested expressions. As a tradeoff, parsing uses additional memory and... - https://man.mtool.work/duckdb/2.0/sql/peg_parser.html
- Profiling 8269
- Profiling is essential to help understand why certain queries exhibit specific performance characteristics. DuckDB contains several built-in features to enable query profiling, which this page covers. For a high-level example of using EXPLA
...onal and named. You can also pass metrics as an unnamed parameter: CALL enable_profiling(['query.total_time', 'query.total_intermediate_size_bytes']); disable_profiling() The disable_profiling() function turns off profiling. CALL disable_profiling(); Metrics... - https://man.mtool.work/duckdb/2.0/dev/profiling.html
- Environment 8269
- The environment where DuckDB is run has an obvious impact on performance. This page focuses on the effects of the hardware configuration and the operating system used. Hardware Configuration CPU DuckDB's officially supported architectures a
...ries have a larger effect on the memory requirement. Workloads containing large joins over many-to-many tables yield large intermediate datasets and thus require more memory for their evaluation to fully fit into the memory. As an approximation, aggregation-... - https://man.mtool.work/duckdb/2.0/guides/performance/environment.html
- PostgreSQL Extension Connection Pool 8181
- PostgreSQL server spawns a backend process for every incoming client connection. This model leads to the two following points with the postgres extension that can cause performance problems: opening a new connection is relatively expensive
...hen DuckDB connects directly to a PostgreSQL server and a new backend process is started for every new connection. When an intermediate transparent proxy is used between DuckDB and PostgreSQL (for example, PgBouncer ) then opening a new connections only caus... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/connection_pool.html
- Jupyter Notebooks 8181
- DuckDB's Python client can be used directly in Jupyter notebooks with no additional configuration, as the first section shows. Additional libraries can simplify SQL query development; the rest of this guide uses the JupySQL package. See oth
...ing module in JupySQL runs computations in the SQL engine. This delegates memory management to the engine and ensures that intermediate computations do not keep eating up memory, efficiently plotting massive datasets. Boxplot & Histogram To create a boxplot,... - https://man.mtool.work/duckdb/2.0/guides/python/jupyter.html
DuckDB 2.0 Documentation / man.mtool.work