Results of 1 - 10 of about 21 for compressed (0.001 sec.)
- Reading and Writing Parquet Files 11687
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.par
...ts of the entire database as parquet: EXPORT DATABASE 'target_directory' (FORMAT parquet); Parquet Files Parquet files are compressed columnar files that are efficient to load and process. DuckDB provides support for both reading and writing Parquet files...e the COPY Statement page for details, including all possible parameters for the COPY statement. Write a query to a Snappy-compressed Parquet file: COPY (SELECT * FROM tbl) TO 'result-snappy.parquet' (FORMAT parquet); Write tbl to a zstd-compressed Parquet...file: COPY tbl TO 'result-zstd.parquet' (FORMAT parquet, COMPRESSION zstd); Write tbl to a zstd-compressed Parquet file with the lowest compression level yielding the fastest compression: COPY tbl TO 'result-zstd.parquet' (FORMAT... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- Execution Format 11687
- Vector is the container format used to store in-memory data during execution. DataChunk is a collection of Vectors, used for instance to represent a column list in a PhysicalProjection operator. Data Flow DuckDB uses a vectorized query exec
..., which allow the system to store the same logical data with a different physical representation . This allows for a more compressed representation, and potentially allows for compressed execution throughout the system. Below the list of supported vector...formats is shown. Flat Vectors Flat vectors are physically stored as a contiguous array, this is the standard uncompressed vector format. For flat vectors the logical and physical representations are identical. Constant Vectors Constant vectors...mpression. Just like constant vectors, dictionary vectors are also emitted by the storage. When deserializing a dictionary compressed column segment, we store this in a dictionary vector so we can keep the data compressed during query execution. Sequence V... - https://man.mtool.work/duckdb/2.0/internals/vector.html
- Advanced Installation Methods 10858
- Downloading Extensions Directly from S3 Downloading an extension directly can be helpful when building a Lambda service or container that uses DuckDB. DuckDB extensions are stored in public S3 buckets, but the directory structure of those b
...NSTALL command can be used with the path to a .duckdb_extension file: INSTALL 'path/to/httpfs.duckdb_extension'; Note that compressed .duckdb_extension.gz files need to be decompressed beforehand. It is also possible to specify remote paths. Loading an Ext...s will skip any currently installed extensions and load the specified extension directly. Note that using remote paths for compressed files is currently not possible. Building and Installing Extensions from Source For building and installing extensions fro... - https://man.mtool.work/duckdb/2.0/extensions/advanced_installation_methods.html
- CHECKPOINT Statement 10655
- The CHECKPOINT statement synchronizes data in the write-ahead log (WAL) to the database data file. Examples Synchronize data in the default database: CHECKPOINT; Synchronize data in the specified database: CHECKPOINT file_db; Synchronize da
...y tables also support compression. This is disabled by default – you can turn it on using: sql ATTACH ':memory:' AS memory_compressed (COMPRESS); USE memory_compressed; Checkpointing triggers vacuuming deleted rows, allowing space to be reclaimed after del... - https://man.mtool.work/duckdb/2.0/sql/statements/checkpoint.html
- Tuning Workloads 10232
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...quent queries. Querying Remote Files Starting with version 2.0, DuckDB uses asynchronous I/O when reading Parquet files, uncompressed, seekable UTF-8 CSV files and DuckDB database files (e.g., attached over HTTPS or S3), as well as when Parquet and CSV fil...our network bandwidth, tune the async_threads and read_ahead_depth settings instead. Other formats, such as JSON files and compressed CSV files, still use synchronous I/O: each DuckDB thread can make at most one HTTP request at a time. If a query on such f...PRAGMA tpch(1); We run this script using three DuckDB prompts: Database setup DuckDB prompt Execution time In-memory DB (uncompressed) duckdb 4.22 s In-memory DB (compressed) duckdb -cmd "ATTACH ':memory:' AS db (COMPRESS); USE db;" 0.55 s Persistent DB (c... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- File Formats 9826
- Handling Parquet Files DuckDB has advanced support for Parquet files, which includes directly querying Parquet files . When deciding on whether to query these files directly or to first load them to the database, you need to consider severa
...ways be somewhat faster, which over time amortizes the initial load time. High decompression times: Some Parquet files are compressed using heavyweight compression algorithms such as gzip. In these cases, querying the Parquet files will necessitate an expe...ips on reading and writing Parquet files, see the Parquet Tips page . Loading CSV Files CSV files are often distributed in compressed format such as GZIP archives ( .csv.gz ). DuckDB can decompress these files on the fly. In fact, this is typically faster...than decompressing the files first and loading them due to reduced IO. Schema Load time Load from GZIP-compressed CSV files ( .csv.gz ) 107.1 s Decompressing (using parallel gunzip ) and loading from decompressed CSV files 121.3 s Loadi... - https://man.mtool.work/duckdb/2.0/guides/performance/file_formats.html
- Importing Data 9420
- The first step to using a database system is to insert data into that system. DuckDB can directly connect to many popular data sources and offers several data ingestion methods that allow you to easily and efficiently fill up the database.
...alse); Or use the COPY statement : COPY tbl FROM 'test.csv' (HEADER false); It is also possible to read data directly from compressed CSV files (e.g., compressed with gzip ): SELECT * FROM 'test.csv.gz'; DuckDB can create a table from the loaded data using... - https://man.mtool.work/duckdb/2.0/data/overview.html
- Querying Parquet Metadata 8794
- Parquet Metadata The parquet_metadata function can be used to query the metadata contained within a Parquet file, which reveals various internal details of the Parquet file such as the statistics of the different columns. This can be useful
...compression VARCHAR encodings VARCHAR index_page_offset BIGINT dictionary_page_offset BIGINT data_page_offset BIGINT total_compressed_size BIGINT total_uncompressed_size BIGINT key_value_metadata MAP(BLOB, BLOB) bloom_filter_offset BIGINT bloom_filter_leng...th BIGINT min_is_exact BOOLEAN max_is_exact BOOLEAN row_group_compressed_bytes BIGINT Parquet Schema The parquet_schema function can be used to query the internal schema contained within a Parque... - https://man.mtool.work/duckdb/2.0/data/parquet/metadata.html
- Load Extensions 8592
- Overview DuckDB-Wasm's (dynamic) extension loading is modeled after regular DuckDB's extension loading, with a few relevant differences due to the difference in platform. A growing subset of core, community, and external extensions is suppo
...uckdb-wasm is added to the folder structure, and the file is served as a .wasm file. DuckDB-Wasm extensions are served pre-compressed using Brotli compression. While fetched from a browser, extensions will be transparently uncompressed. If you want to fetc...h the duckdb-wasm extension manually, you can use curl --compressed extensions.duckdb.org/<...>/icu.duckdb_extension.wasm . Serving Extensions from a Third-Party Repository As with regular D... - https://man.mtool.work/duckdb/2.0/clients/wasm/extensions.html
- EXPORT and IMPORT DATABASE Statements 7983
- The EXPORT DATABASE command allows you to export the contents of the database to a specific directory. The IMPORT DATABASE command allows you to then read the contents again. Examples Export the database to the target directory target_direc
...EXPORT DATABASE 'target_directory' (FORMAT parquet); Export to directory target_directory , tables serialized as Parquet, compressed with Zstd, with a ROW_GROUP_SIZE of 100,000: EXPORT DATABASE 'target_directory' ( FORMAT parquet, COMPRESSION zstd, ROW_GR... - https://man.mtool.work/duckdb/2.0/sql/statements/export.html
DuckDB 2.0 Documentation / man.mtool.work