Results of 1 - 10 of about 136 for memory (0.001 sec.)
- Out-of-Memory Issues 10245
- Configuration Options If you are experiencing out-of-memory issues with DuckDB, try tweaking the following configuration options: Reduce the number of threads using the SET threads = ... command. Setting the memory limit lower than the defa
...ult 80% can help prevent out of memory errors. While this configuration sounds counter-intuitive, it helps because some of DuckDB's operations circumvent the dat...abase's buffer manager and thus they can reserve more memory than allowed by the memory limit. If this happens (e.g., DuckDB is killed by the operating system or an OOM reaper process...), set the memory limit to just 50-60% of the total system memory by using the SET memory_limit = '...' statement. If your query reads a lar... - https://man.mtool.work/duckdb/2.0/guides/performance/oom.html
- Out of Memory Errors 10209
- DuckDB has a state of the art out-of-core query engine that can spill to disk for larger-than-memory processing. We continuously strive to improve DuckDB's scalability and prevent out of memory errors whenever possible. That said, you may s
...till experience out-of-memory errors if you run queries with multiple blocking operators , certain aggregate functions (such as string_agg() or aggregat...es with an ORDER BY clause ), or if you have very little available memory compared to the dataset size. Types of “Out of Memory” Errors Out of memory errors mainly occur in two forms: OutOfMemoryE...xception Most of the time DuckDB runs out of memory with an OutOfMemoryException followed by a message from the memory allocator. For example: Out of Memory Error: failed to... - https://man.mtool.work/duckdb/2.0/guides/troubleshooting/oom_errors.html
- Environment 10137
- The environment where DuckDB is run has an obvious impact on performance. This page focuses on the effects of the hardware configuration and the operating system used. Hardware Configuration CPU DuckDB's officially supported architectures a
...led to other architecture such as LoongArch and RISC-V . However, there are no performance guarantees for these platforms. Memory Bestpractice Aim for 1-4 GB memory per thread. Minimum Required Memory As a rule of thumb, DuckDB requires a minimum of 12...5 MB of memory per thread. For example, if you use 8 threads, you need at least 1 GB of memory. If you are working in a memory-constraine...d environment, consider limiting the number of threads , e.g., by issuing: SET threads = 4; Memory for Ideal Performance The amount of memory required for ideal performance depends on several factors, including the datase... - https://man.mtool.work/duckdb/2.0/guides/performance/environment.html
- Tuning Workloads 9873
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...rresponding change, and measure its effect. If your priority is… Consider Covered in Loading or exporting data larger than memory Disabling preserve_insertion_order The preserve_insertion_order option Using all available cores Row group sizing and the...threads setting Parallelism Processing data larger than memory Configuring the spill-to-disk directory Larger-than-memory workloads Diagnosing a slow query Inspecting the query plan Pro...Option When importing or exporting datasets (from/to the Parquet or CSV formats), which are much larger than the available memory, an out of memory error may occur: Out of Memory Error: failed to allocate data of size ... (.../... used) In these cases,... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- Profile and Monitor 9344
- Overview The JDBC driver exposes several ways to observe a running or completed query: cancelling and timing out queries, polling query progress, retrieving profiling output, and emitting memory-usage events through Java Flight Recorder . T
...e(1000)").close(); String profile = conn.getProfilingInformation(ProfilerPrintFormat.JSON); System.out.println(profile); } Memory Monitoring with JFR On a JFR-capable JVM , the driver can emit Java Flight Recorder events describing DuckDB's internal me...age, so that any JFR-aware tool — JDK Mission Control, the jfr command line tool, continuous profilers — can ingest DuckDB memory metrics alongside the rest of the JVM signal. The feature is strictly opt-in and is a silent no-op on JVMs without JFR sup...as an active JFR recording that enables the event. Enabling Emission Monitoring is opt-in per connection: set the jdbc_jfr_memory_monitor property to an identifier for the connection's database instance. The value is an arbitrary label that is attached... - https://man.mtool.work/duckdb/2.0/clients/java/profiling.html
- Connect 8804
- Overview Every DuckDB feature reachable from Java starts with a Connection . This page covers the JDBC URL forms the driver accepts, how DuckDB and driver options are passed and which of them win, how connections come to share (or not share
...tion: DuckDBConnection conn2 = ((DuckDBConnection) conn).duplicate(); Its main purpose is to reach a connection-private in-memory database instance — the one created by the plain jdbc:duckdb: URL — which no other URL can address. On a named in-memory o...;⟨key⟩=⟨value⟩ The database part determines what the connection opens: JDBC URL Opens jdbc:duckdb: A connection-private in-memory database. jdbc:duckdb:memory: The same as above, written explicitly. jdbc:duckdb:memory:⟨label⟩ A named in-memory database.... jdbc:duckdb:ducklake:⟨metadata_path⟩ A DuckLake catalog. When using the jdbc:duckdb: URL alone, a connection-private in-memory database is created: the instance belongs to that single connection and cannot be shared, other than through duplicate() .... - https://man.mtool.work/duckdb/2.0/clients/java/connecting.html
- Known Issues 8732
- Incorrect Memory Values on Old Linux Distributions and WSL 2 On Windows Subsystem for Linux 2 (WSL2), when querying the max_memory or memory_limit from the duckdb_settings , the values may be inaccurate on certain Ubuntu versions (e.g., 20.
...rchar │ varchar │ ├──────────────┼────────────┼─────────────────────────────────────────────┼────────────┼─────────┤ │ max_memory │ 1638.3 PiB │ The maximum memory of the system (e.g. 1GB) │ VARCHAR │ GLOBAL │ │ memory_limit │ 1638.3 PiB │ The maximum... - https://man.mtool.work/duckdb/2.0/clients/cli/known_issues.html
- Connect 8539
- Overview Every DuckDB feature reachable from Rust starts with a Connection . Opening in-memory and file-backed databases, configuring a database instance with Config , opening read-only, pooling connections across threads, the crate's threa
...ees, and closing a connection are all covered below. Opening a Connection A Connection is created with Connection::open_in_memory() for an in-memory database, or Connection::open(path) for a database file. Both return a Result , so the ? operator propa...gates any error: use duckdb::{Connection, Result}; // In-memory database: nothing is persisted to disk. let conn = Connection::open_in_memory()?; // File-backed database: created if it d...oes not exist. let conn = Connection::open("my_database.duckdb")?; For an in-memory database no data is persisted to disk, so all data is lost when the process exits. A file-backed database is created if th... - https://man.mtool.work/duckdb/2.0/clients/rust/connecting.html
- My Workload Is Slow 8503
- If you find that your workload in DuckDB is slow, we recommend performing the following checks. More detailed instructions are linked for each point. Do you have enough memory? DuckDB works best if you have 1-4 GB memory per thread . Is you
...r system maybe overcommitting memory, forcing the operating system to swap? Try lowering the amount of memory available from the default 80% of the total RAM u...sing SET memory_limit = '...'; . While this sounds counter-intuitive, it can sometimes improve query performance, especially in memory-con...strained environments where other processes are likely using more than 20% of the total system memory. Are you using a fast disk? Network-attached disks (such as cloud block storage) cause write-intensive and larger than mem... - https://man.mtool.work/duckdb/2.0/guides/performance/my_workload_is_slow.html
- CHECKPOINT Statement 8359
- The CHECKPOINT statement synchronizes data in the write-ahead log (WAL) to the database data file. Examples Synchronize data in the default database: CHECKPOINT; Synchronize data in the specified database: CHECKPOINT file_db; Synchronize da
...ECKPOINT aborted any in-progress transactions. From v1.4, it waits until it can grab the checkpoint lock. Checkpointing In-Memory Tables Starting with v1.4.0, in-memory tables support checkpointing. This has two key benefits: In-memory tables also supp...ort compression. This is disabled by default – you can turn it on using: sql ATTACH ':memory:' AS memory_compressed (COMPRESS); USE memory_compressed; Checkpointing triggers vacuuming deleted rows, allowing space to...ementation this requires ~25% of rows to be deleted in adjacent row groups. Since v1.4.0, checkpointing also applies to in-memory databases (see Checkpointing In-Memory Tables ), where it likewise vacuums deleted rows and reclaims space. Warning The VA... - https://man.mtool.work/duckdb/2.0/sql/statements/checkpoint.html
DuckDB 2.0 Documentation / man.mtool.work