Results of 1 - 10 of about 33 for compression (0.003 sec.)
- COPY Statement 12561
- Examples Read a CSV file into the lineitem table, using auto-detected CSV options: COPY lineitem FROM 'lineitem.csv'; Read a CSV file into the lineitem table, using manually specified CSV options: COPY lineitem FROM 'lineitem.csv' (DELIMITE
...v; Write the result of a query to a Parquet file: COPY (SELECT l_orderkey, l_partkey FROM lineitem) TO 'lineitem.parquet' (COMPRESSION zstd); Copy the entire content of database db1 to database db2 : COPY FROM DATABASE db1 TO db2; Copy only the schema (cata...Format-Specific Options CSV Options The below options are applicable when writing CSV files. Name Description Type Default COMPRESSION The compression type for the file. By default this will be detected automatically from the file extension (e.g., file.csv....VARCHAR (empty) Parquet Options The below options are applicable when writing Parquet files. Name Description Type Default COMPRESSION The compression format to use ( uncompressed , snappy , gzip , zstd , brotli , lz4 , lz4_raw ). VARCHAR snappy COMPRESSION... - https://man.mtool.work/duckdb/2.0/sql/statements/copy.html
- Reading and Writing Parquet Files 11743
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.par
...quet file : SELECT * FROM parquet_schema('test.parquet'); Write the results of a query to a Parquet file using the default compression (Snappy): COPY (SELECT * FROM tbl) TO 'result-snappy.parquet' (FORMAT parquet); Write the results from a query to a Parque...t file with specific compression and row group size: COPY (FROM generate_series(100_000)) TO 'test.parquet' (FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SI...tions into the Parquet file scans. Parquet datasets differ based on the number of files, the size of individual files, the compression algorithm used, row group size, etc. These have a significant effect on performance. Please consult the Performance Guide... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- Storage Versions and Format 11438
- Compatibility Backward Compatibility Backward compatibility refers to the ability of a newer DuckDB version to read storage files created by an older DuckDB version. Version 0.10 is the first release of DuckDB that supports backward compati
...data lazily, so databases with wide tables and large indexes open faster and use less memory; enables the DICT_FSST string compression method by default (see Compression ); stores row deletions more compactly; and performs stronger corruption validation whe...0.3.2 27 v0.3.1 25 v0.3.0 21 v0.2.9 18 v0.2.8 17 v0.2.7 15 v0.2.6 13 v0.2.5 11 v0.2.4 6 v0.2.3 4 v0.2.2 1 v0.2.1 and prior Compression DuckDB uses lightweight compression . By default, compression is only applied to persistent databases and is not applied t...o in-memory instances . To turn on compression for in-memory databases, use ATTACH with the COMPRESS option . Note that available compression algorithms depend on the st... - https://man.mtool.work/duckdb/2.0/internals/storage.html
- File Formats 10909
- Handling Parquet Files DuckDB has advanced support for Parquet files, which includes directly querying Parquet files . When deciding on whether to query these files directly or to first load them to the database, you need to consider severa
...the data into DuckDB. The queries will always be somewhat faster, which over time amortizes the initial load time. High decompression times: Some Parquet files are compressed using heavyweight compression algorithms such as gzip. In these cases, querying t...he Parquet files will necessitate an expensive decompression time every time the file is accessed. Meanwhile, lightweight compression methods like Snappy, LZ4, and zstd, are faster to...decompress. You may use the parquet_metadata function to find out the compression algorithm used. Microbenchmark: Running TPC-H on a DuckDB Database vs. Parquet The queries on the TPC-H benchmark run appr... - https://man.mtool.work/duckdb/2.0/guides/performance/file_formats.html
- Text Types 10620
- In DuckDB, strings can be stored in the VARCHAR field. The field allows storage of Unicode characters. Internally, the data is encoded as UTF-8. Name Aliases Description VARCHAR CHAR , BPCHAR , STRING , TEXT Variable-length character string
...th of 10 ); The VARCHAR field allows storage of Unicode characters. Internally, the data is encoded as UTF-8. Specifying a Compression Type You can specify a compression type for a string with the USING COMPRESSION clause. For example, to apply zstd compres...sion, run: CREATE TABLE tbl (s VARCHAR USING COMPRESSION zstd); Text Type Values Values of the text type are character strings, also known as string values or simply strings. At r... - https://man.mtool.work/duckdb/2.0/sql/data_types/text.html
- Pragmas 10460
- The PRAGMA statement is a SQL extension adopted by DuckDB from SQLite. PRAGMA statements can be issued in a similar manner to regular SQL statements. PRAGMA commands may alter the internal state of the database engine, and can influence the
...segment_type VARCHAR start BIGINT The start row id of this chunk count BIGINT The amount of entries in this storage chunk compression VARCHAR Compression type used for this column – see the “Lightweight Compression in DuckDB” blog post stats VARIANT has_up...rameter. When set to true , the result contains an additional segment_info VARCHAR column with per-segment detail for some compression methods, such as the DICT_FSST mode applied (for example, DICTIONARY ): SELECT compression, segment_info FROM pragma_stora...e.g., Parquet metadata: PRAGMA enable_object_cache; Disable caching of objects: PRAGMA disable_object_cache; Checkpointing Compression During checkpointing, the existing column data + any new changes get compressed. There exist a couple of settings to influ... - https://man.mtool.work/duckdb/2.0/configuration/pragmas.html
- Iceberg Options 8294
- This page lists the options of the iceberg extension: the parameters accepted by the Iceberg functions , the options of the ATTACH and CREATE SECRET statements used to connect to a catalog , and the global settings. ATTACH Options To make a
...Parameter Type Default Description allow_moved_paths BOOLEAN false Allows scanning Iceberg tables that are moved metadata_compression_codec VARCHAR '' Treats metadata files as gzip-compressed when set to 'gzip' snapshot_from_id UBIGINT NULL Access snapshot...tadata.json and {version}.metadata.json files, or v{version}.gz.metadata.json and {version}.gz.metadata.json when metadata_compression_codec = 'gzip' is specified. Other compression codecs are not supported. If any text file is provided through the version...version number in the metadata filename and the second is the location of the filename extension specified by the metadata_compression_codec . The behavior described above is provided by the default value v%s%s.metadata.json,%s%s.metadata.json . If you had... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/iceberg_options.html
- PHP Client (PDO) 7845
- The PHP DuckDB PDO extension is a tertiary client and is maintained by a third-party. To use pdo_duckdb , follow the instructions below . The latest version of pdo_duckdb supports DuckDB 2.0.0. pdo_duckdb is a native DuckDB database driver
...write Parquet files <?php $db = new PDO('duckdb::memory:'); $db->exec('CREATE TABLE table1 (id INTEGER, text VARCHAR USING COMPRESSION zstd, data JSON)'); $statement = $db->prepare('INSERT INTO table1 VALUES (?, ?, ?)'); $statement->execute([1, 'Hello DuckD...B ?', ['foo' => 'bar', 'baz' => 42]]); $db->exec("COPY (SELECT * FROM table1) TO '/tmp/table1.parquet' (COMPRESSION zstd)"); foreach ($db->query("SELECT * FROM '/tmp/table1.parquet'", PDO::FETCH_ASSOC) as $row) { print_r($row); } # Array...] => Array # ( # [log] => log text 2 # ) # ) $db->exec("COPY (SELECT * FROM '/tmp/logs.json') TO '/tmp/logs_json.parquet' (COMPRESSION zstd)"); Use structured columns with a fixed schema <?php # s is array{v: string, i: int, a: string[], d: float} $db = new... - https://man.mtool.work/duckdb/2.0/clients/tertiary_clients/php_pdo.html
- Parquet Export 7764
- To export the data from a table to a Parquet file, use the COPY statement : COPY tbl TO 'output.parquet' (FORMAT parquet); The result of queries can also be directly exported to a Parquet file: COPY (SELECT * FROM tbl) TO 'output.parquet' (
...FORMAT parquet); The flags for setting compression, row group size, etc. are listed in the Reading and Writing Parquet files page.... - https://man.mtool.work/duckdb/2.0/guides/file_formats/parquet_export.html
- Configuration 7684
- DuckDB has a number of configuration options that can be used to change the behavior of the system. The configuration options can be set using either the SET statement or the PRAGMA statement . They can be reset to their original values usi
...m in Parquet BOOLEAN false disable_timestamptz_casts Disable casting from timestamp to timestamptz. BOOLEAN false disabled_compression_methods Disable a specific, comma-separated set of compression methods. VARCHAR disabled_filesystems Disable specific file...ory. BOOLEAN true enable_fsst_vectors Allow scans on FSST compressed segments to emit compressed vectors to utilize late decompression. BOOLEAN false enable_geoparquet_conversion Attempt to decode/encode geometry data in/as GeoParquet files if the spatial e...older storage versions 6 and 7. This breaks storage backward-compatibility with older DuckDB versions. BOOLEAN false force_compression Forces the usage of a specific compression method. VARCHAR auto force_download_threshold Forces upfront download of files... - https://man.mtool.work/duckdb/2.0/configuration/overview.html
DuckDB 2.0 Documentation / man.mtool.work