Results of 1 - 10 of about 77 for storage (0.002 sec.)
- Logging 12252
- DuckDB implements a logging mechanism that provides users with detailed information about events such as query execution, performance metrics and system events. Warning Logs may contain sensitive information, including secrets and credentia
...re stored in a special view named duckdb_logs , which can be queried like any standard table. Example: CALL enable_logging(storage = 'memory'); -- Run some queries... SELECT * FROM duckdb_logs; In the CLI client , the default log storage is shell_log_st...orage , which prints log messages to the console but does not support querying. Therefore, the example above passes storage = 'memory' . In other clients, the default log storage is memory , so CALL enable_logging(); suffices. See Log Storages fo...he fact, enable logging and query the duckdb_logs view, filtering on the log level: CALL enable_logging(level = 'warning', storage = 'memory'); -- Run some queries that may emit warnings... SELECT timestamp, message FROM duckdb_logs WHERE log_level = 'W... - https://man.mtool.work/duckdb/2.0/operations_manual/logging/overview.html
- Storage Versions and Format 12142
- Compatibility Backward Compatibility Backward compatibility refers to the ability of a newer DuckDB version to read storage files created by an older DuckDB version. Version 0.10 is the first release of DuckDB that supports backward compati
...bility in the storage format. DuckDB 0.10 can read and operate on files created by the previous DuckDB version – DuckDB 0.9. For future DuckDB v...les between versions. Forward Compatibility Forward compatibility refers to the ability of an older DuckDB version to read storage files produced by a newer DuckDB version. DuckDB 0.9 is partially forward compatible with DuckDB 0.10 . Certain files crea...by DuckDB 0.10 can be read by DuckDB 0.9. Forward compatibility is provided on a best effort basis. While stability of the storage format is important – there are still many improvements and innovations that we want to make to the storage format in the... - https://man.mtool.work/duckdb/2.0/internals/storage.html
- Azure Extension 10733
- The azure extension lets DuckDB read and write files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2. Installing and Loading The azure extension will be transparently autoloaded on first use from the official extension reposit
...and load it manually, run: INSTALL AND LOAD azure; Usage Queries need a secret ; for a public container, a secret with the storage account name is enough (see the CONFIG provider ). Then you can query Azure storage as follows: Azure Blob Storage Allowed...csv'; SELECT * FROM 'az://⟨my_container⟩/⟨path⟩/**'; Or with a fully qualified path syntax: SELECT count(*) FROM 'az://⟨my_storage_account⟩.blob.core.windows.net/⟨my_container⟩/⟨path⟩/⟨my_file⟩.⟨parquet_or_csv⟩'; SELECT * FROM 'az://⟨my_storage_account⟩....blob.core.windows.net/⟨my_container⟩/⟨path⟩/*.csv'; Azure Data Lake Storage (ADLS) Allowed URI schemes: abfss or abfs (both connect over HTTPS) SELECT count(*) FROM 'abfss://⟨my_filesystem⟩/⟨path⟩/⟨... - https://man.mtool.work/duckdb/2.0/core_extensions/azure.html
- Migrating to DuckDB 2.0 10733
- DuckDB 2.0.0 is packed with new features and performance improvements, including query optimization and asynchronous I/O. It also ships a small set of breaking changes, most notably a new default storage format, which DuckDB versions before
..., especially those that run many DuckDB processes, require more care. We expect the most common source of issues to be the storage format, which is described first below. Breaking Changes Storage Format DuckDB versions before 2.0.0 cannot read database...files that use the DuckDB 2.0.0 storage format. By default, every new database file created by v2.0.0 uses this format ( storage version 69 ). The other direction...keeps working: DuckDB 2.0.0 can read and write existing database files created by earlier versions. These files keep their storage version when v2.0.0 writes to them, so they remain readable by the earlier version. For example, creating a database with... - https://man.mtool.work/duckdb/2.0/operations_manual/migration.html
- Metrics 9683
- DuckDB provides a set of metrics that can be used to monitor the performance and health of the database. Metrics are named ⟨group⟩.⟨metric⟩ , e.g., query.total_time or operator.timing . The set of collected metrics is controlled with the tr
...pe, unit FROM duckdb_available_metrics(); All Metrics The metrics are organized into the following groups: query system io storage operator optimizer parser planner physical_planner query Metrics Metrics measured over the entire query. Name Type Unit De...ated uint64 bytes The total memory allocated by the buffer manager io Metrics Metrics on the data read from and written to storage. Name Type Unit Description io.total_bytes_read uint64 bytes The total amount of bytes read from storage io.total_bytes_wr...itten uint64 bytes The total amount of bytes written to storage storage Metrics Metrics on storage operations, such as attaching databases, writing to the WAL and checkpointing. Name Typ... - https://man.mtool.work/duckdb/2.0/dev/metrics.html
- Catch C/C++ Tests 9296
- While we prefer the sqllogic tests for testing most functionality, for certain tests only SQL is not sufficient. This typically happens when you want to test the C++ API. When using pure SQL is really not an option it might be necessary to
...C++ test using Catch. Catch tests reside in the test directory as well. Here is an example of a catch test that tests the storage of the system: #include "catch.hpp" #include "test_helpers.hpp" TEST_CASE("Test simple storage", "[storage]") { auto confi...g = GetTestConfig(); unique_ptr<QueryResult> result; auto storage_database = TestCreatePath("storage_test"); // make sure the database does not exist DeleteDatabase(storage_database); { //...create a database and insert values DuckDB db(storage_database, config.get()); Connection con(db); REQUIRE_NO_FAIL(con.Query("CREATE TABLE test (a INTEGER, b INTEGER);")); REQU... - https://man.mtool.work/duckdb/2.0/dev/sqllogictest/catch.html
- Profile and Monitor 9061
- Overview The Go client can observe query execution: it can retrieve per-operator profiling metrics after a query runs, and register a log storage that receives DuckDB's log messages. Both are described below. Profiling Profiling information
...rict them, set the tracked_metrics option. See the Profiling page and the metrics list for the available metric names. Log Storage DuckDB can forward its log messages to a custom storage registered from Go. Register one on a Connector with duckdb.Regist...erLogStorage() , giving it a name and a duckdb.LoggerCallbacks whose DefaultLoggerCallback receives each entry's level, type, and messa...ge: connector, err := duckdb.NewConnector("", nil) check(err) defer connector.Close() err = duckdb.RegisterLogStorage(connector, "my_log_storage", duckdb.LoggerCallbacks{ DefaultLoggerCallback: func(level, logType, logMsg string) { log.Prin... - https://man.mtool.work/duckdb/2.0/clients/go/profiling.html
- Geometry Data Type 9020
- Name Description GEOMETRY Geospatial entity The GEOMETRY data type is used to store and manipulate geometric objects, such as points, lines, and polygons. The GEOMETRY type was part of the spatial extension but became a built-in data type i
...n DuckDB 1.5. Most of the benefits of having GEOMETRY as a built-in type (e.g., storage optimizations, statistics, etc.) are therefore only available in databases using storage version v1.5 and above. However,...valid geometrical representation (e.g., the intersection of two non-overlapping geometries is an empty geometry). Geometry Storage Internally GEOMETRY values are stored as a sequence of bytes, similarly to DuckDB's BLOB types. The exact binary format is...not yet stabilized and may change in a future release, but as of DuckDB storage version v1.5 it is based on little-endian Well-Known Binary (WKB) , which is a standard binary encoding for geometries. In... - https://man.mtool.work/duckdb/2.0/sql/data_types/geometry.html
- Environment 8896
- The environment where DuckDB is run has an obvious impact on performance. This page focuses on the effects of the hardware configuration and the operating system used. Hardware Configuration CPU DuckDB's officially supported architectures a
.... While HDDs are supported, they will result in low performance, especially for write operations. Disk-based vs. in-memory storage. Counter-intuitively, using a disk-based DuckDB instance can be faster than an in-memory instance. This is because the def...ault settings prescribe compression for disk-based storage but they do turn compression off for in-memory storage. Read more in the “How to Tune Workloads” page . File systems. On L...t is important that the disks are reliable. As a general rule of thumb, this is true for locally attached disks, and block storage in the cloud. If your workload is larger than memory and/or fast data loading is important, you need fast disks, preferabl... - https://man.mtool.work/duckdb/2.0/guides/performance/environment.html
- ATTACH and DETACH Statements 8357
- DuckDB allows attaching to and detaching from database files. Examples Attach the database file.db with the alias inferred from the name ( file ): ATTACH 'file.db'; Attach the database file.db with an explicit alias ( file_db ): ATTACH 'fil
...ations.duckdb' AS stations_db; ATTACH 's3://⟨blobs-duckdb⟩/databases/stations.duckdb' AS stations_db (READ_ONLY); Explicit Storage Versions DuckDB 1.2.0 introduced the STORAGE_VERSION option , which allows explicitly specifying the storage version. This...n be read by the specified version and all newer versions of DuckDB. Since DuckDB 2.0.0, new database files use the latest storage version by default (equivalent to STORAGE_VERSION 'latest' ), so they cannot be read by older DuckDB versions. Therefore,...the main use of this option is to create database files that remain readable by older DuckDB versions: ATTACH 'file.db' (STORAGE_VERSION 'v1.5.0'); You can check the storage version of attached databases using the tags column of the duckdb_databases()... - https://man.mtool.work/duckdb/2.0/sql/statements/attach.html
DuckDB 2.0 Documentation / man.mtool.work