Results of 1 - 10 of about 20 for data processing (0.003 sec.)
data (258), processing (27)
- Tuning Workloads 20865
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...re optimizing for and on the characteristics of your data, hardware, and environment. Rather than copying a fixed value, identify which of the concerns below match your workload, a...pply the corresponding change, and measure its effect. If your priority is… Consider Covered in Loading or exporting data larger than memory Disabling preserve_insertion_order The preserve_insertion_order option Using all available cores Row gr...oup sizing and the threads setting Parallelism Processing data larger than memory Configuring the spill-to-disk directory Larger-than-memory workloads Diagnosing a slow query Inspe... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- INSERT Statements 14919
- INSERT statements are the standard way of loading data into a relational database. When using INSERT statements, the values are supplied row-by-row. While simple, there is significant overhead involved in parsing and processing individual I
...tatements when inserting more than a few rows (i.e., avoid using INSERT statements as part of a loop). When bulk inserting data, try to maximize the amount of data that is inserted per statement. If you must use INSERT statements to load data in a lo...op, avoid executing the statements in auto-commit mode. After every commit, the database is required to sync the changes made to disk to ensure no data is lost. In auto-commit mode every single statement wil... - https://man.mtool.work/duckdb/2.0/data/insert.html
- Schema 14446
- Types It is important to use the correct type for encoding columns (e.g., BIGINT , DATE , DATETIME ). While it is always possible to use string types ( VARCHAR , etc.) to encode more specific values, this is not recommended. Strings use mor
...ion mechanism to determine the correct types for CSV inputs. If you run in a memory-constrained environment, using smaller data types (e.g., TINYINT ) can reduce the amount of memory and disk space required to complete a query. DuckDB's bitpacking co...mpression means small values stored in larger data types will not take up larger sizes on disk, but they will take up more memory during processing – hence, using the most r...E 3.3 GB 0.9 s VARCHAR 5.2 GB 3.9 s The results show that using the DATETIME value yields smaller storage sizes and faster processing. Microbenchmark: Joining on Strings We illustrate the difference caused by joining on different types by computing a self-... - https://man.mtool.work/duckdb/2.0/guides/performance/schema.html
- Spatial Extension 14183
- The spatial extension provides support for geospatial data processing in DuckDB. For an overview of the extension, see our blog post . Installing and Loading To install the spatial extension, run: INSTALL spatial; Note that the spatial exte
... - https://man.mtool.work/duckdb/2.0/core_extensions/spatial/overview.html
- Performance Guide 14183
- DuckDB aims to automatically achieve high performance by using well-chosen default configurations and having a forgiving architecture. Of course, there are still opportunities for tuning the system for specific workloads. The Performance Gu
...ide's pages contain guidelines and tips for achieving good performance when loading and processing data with DuckDB. The guides include several microbenchmarks. You may find details about these on the Benchmarks page .... - https://man.mtool.work/duckdb/2.0/guides/performance/overview.html
- Hugging Face Support 14159
- The httpfs extension introduces support for the hf:// protocol to access datasets hosted in Hugging Face repositories. See the announcement blog post for details. Usage Hugging Face repositories can be queried using the following URL patter
...n: hf://datasets/⟨my_username⟩/⟨my_dataset⟩/⟨path_to_file⟩ For example, to read a CSV file, you can use the following query: SELECT * F...ROM 'hf://datasets/datasets-examples/doc-formats-csv-1/data.csv'; Where: datasets-examples is the name of the user/organization doc-forma...ently handle large datasets and perform comprehensive queries across multiple files, simplifying your data inspections and processing tasks. Here, you can see how you can look for questions that contain the word “planet” in astronomy: SELECT count(*) AS co... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/hugging_face.html
- I/O 14157
- Warning DuckDB ships with safe defaults. The configuration options described on this page are advanced options, so proceed with caution when changing them. Asynchronous I/O New Asynchronous I/O was introduced in DuckDB 2.0. Starting with v2
....0, DuckDB supports asynchronous I/O. Instead of blocking a worker thread until requested data arrives, DuckDB issues reads in the background and keeps multiple requests in flight while worker threads process data tha...This can significantly speed up queries when synchronous I/O does not saturate the available bandwidth, e.g., when reading data from object storage such as S3. For a detailed explanation of the design and benchmark results, see the “Asynchronous I/O.... Thread Pools DuckDB uses two thread pools: The regular pool contains the worker threads, which perform the actual query processing (e.g., decoding, joins and aggregations). Its size is controlled by the threads setting and defaults to the number of CPU... - https://man.mtool.work/duckdb/2.0/data/io.html
- Indexes 13883
- Index Types DuckDB has two built-in index types. Indexes can also be defined via extensions . Min-Max Index (Zonemap) A min-max index (also known as zonemap or block range index) is automatically created for columns of all general-purpose data
...o drop an ART index , use the DROP INDEX statement . Limitations of ART Indexes ART indexes create a secondary copy of the data in a second location. Maintaining that second copy complicates processing. Thus, certain limitations currently apply when...it comes to modifying data that is also stored in secondary indexes. As expected, indexes have a strong effect on performance, slowing down loading a...written into a DELETE , followed by an INSERT . This rewrite happens per chunk of data (2048 rows) moving through DuckDB's processing pipeline. When updating i = 2047 to i = 2048 , we do not yet know that 2048 becomes 2049, and so forth. That is because we... - https://man.mtool.work/duckdb/2.0/sql/indexes.html
- PostgreSQL Extension 13735
- The postgres extension allows DuckDB to directly read and write data from a running PostgreSQL database instance. The data can be queried directly from the underlying PostgreSQL database. Data can be loaded from PostgreSQL tables into DuckD
...sitory. If you would like to install and load it manually, run: INSTALL AND LOAD postgres; Connecting To make a PostgreSQL database accessible to DuckDB, use the ATTACH command with the postgres or postgres_scanner type. To connect to the public sche...mentation . dbname=postgresscanner host=localhost port=5432 dbname=mydb connect_timeout=10 Name Description Default dbname Database name [user] host Name of host to connect to localhost hostaddr Host IP address localhost passfile Name of file passwor...read the value for a given key, or postgres_hstore_to_json to convert the whole set of key/value pairs to JSON for further processing. SELECT postgres_hstore_get('a=>b, c=>d', 'a'); -- b SELECT postgres_hstore_get('a=>b, c=>d', 'missingkey'); -- NULL SELEC... - https://man.mtool.work/duckdb/2.0/core_extensions/postgres/overview.html
- Out-of-Memory Issues 13418
- Configuration Options If you are experiencing out-of-memory issues with DuckDB, try tweaking the following configuration options: Reduce the number of threads using the SET threads = ... command. Setting the memory limit lower than the defa
...ory errors. While this configuration sounds counter-intuitive, it helps because some of DuckDB's operations circumvent the database's buffer manager and thus they can reserve more memory than allowed by the memory limit. If this happens (e.g., DuckDB...t 50-60% of the total system memory by using the SET memory_limit = '...' statement. If your query reads a large amount of data from a file or writes a large amount of data, try setting the preserve_insertion_order option to false : SET preserve_inse...s small values stored in larger data types will not take up larger sizes on disk, but they will take up more memory during processing – hence, using the most restrictive types possible when creating columns is necessary to reduce memory consumption. Indexe... - https://man.mtool.work/duckdb/2.0/guides/performance/oom.html
DuckDB 2.0 Documentation / man.mtool.work