Results of 1 - 10 of about 16 for Partitioning (0.002 sec.)
- Hive Partitioning 15283
- Examples Read data from a Hive partitioned dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); Write a table to a Hive partitioned dataset: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, mont
...reading, the partition columns are read from the directory structure and can be included or excluded depending on the hive_partitioning parameter. FROM read_parquet('test/*/*/*.parquet', hive_partitioning = false); -- will not include year, month columns FRO...M read_parquet('test/*/*/*.parquet', hive_partitioning = true); -- will include year, month partition columns Hive Partitioning Hive partitioning is a partitioning strategy that...file4.parquet │ └── file5.parquet └── month=12 └── file6.parquet Files stored in this hierarchy can be read using the hive_partitioning flag. SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); When we specify the hive_partitioning... - https://man.mtool.work/duckdb/2.0/data/partitioning/hive_partitioning.html
- Writing to Iceberg 8837
- The DuckDB iceberg extension supports writing to Iceberg tables when an Iceberg catalog is attached. Writing to a table read directly from storage with iceberg_scan is not supported: the catalog is required to commit the new table metadata.
...e they query the LoadTableInformation endpoint. Iceberg schema evolution is metadata-only, so no data files are rewritten. Partitioning Tables can be partitioned with the PARTITIONED BY clause using the Iceberg partition transforms : Transform Description ⟨c...by a date or timestamp component. bucket(⟨n⟩, ⟨column⟩) Hash the column's value into n buckets. This is useful for stable partitioning on a high-cardinality column. truncate(⟨w⟩, ⟨column⟩) Group rows by the first w characters (or by the column's value round...ed down to a multiple of w for numeric columns). This is useful for prefix-based partitioning. CREATE TABLE iceberg_catalog.default.events ( event_id BIGINT, user_id BIGINT, country VARCHAR, event_time TIMESTAMP ) PA... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/writing_to_iceberg.htm...
- Directly Reading Files 8716
- DuckDB allows directly reading files via the read_text and read_blob functions. These functions accept a filename, a list of filenames, or a glob pattern. They output the content of each file as a VARCHAR or BLOB , respectively, along with
...t VARCHAR YES NULL NULL NULL size BIGINT YES NULL NULL NULL last_modified TIMESTAMP WITH TIME ZONE YES NULL NULL NULL Hive Partitioning Data can be read from Hive partitioned datasets. SELECT * FROM read_blob('data/parquet-testing/hive-partitioning/simple/**... - https://man.mtool.work/duckdb/2.0/guides/file_formats/read_file.html
- Iceberg Extension 8543
- The iceberg extension implements support for the Apache Iceberg open table format . There are two ways to work with Iceberg in DuckDB: Individual tables are read directly from storage, by pointing at a table's metadata. This requires no cat
...uding writing. This page covers the basics of both. See also: Writing to Iceberg describes the supported write operations, partitioning and schema evolution. Iceberg Functions documents the functions provided by the extension. Iceberg Options documents the s...passed a fully qualified name, e.g., iceberg_snapshots(my_catalog.default.events) . For the full set of write operations — partitioning, UPDATE , DELETE , MERGE INTO , ALTER TABLE , and table properties — see Writing to Iceberg . For catalog-specific setup,... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/overview.html
- S3 API Support 8300
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...umn_a column_b filename 1 examplevalue1 s3://bucket-name/file1.parquet 2 examplevalue1 s3://bucket-name/file2.parquet Hive Partitioning DuckDB also offers support for the Hive partitioning scheme , which is available when using HTTP(S) and S3 endpoints. Writ... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Window Functions 8300
- DuckDB supports window functions , which can use multiple rows to calculate a value for each row. Window functions are blocking operators , i.e., they require their entire input to be buffered, making them one of the most memory-intensive o
...the diagram at the top of the page, and this diagram visually illustrates computation environment: Partition and Ordering Partitioning breaks the relation up into independent, unrelated pieces. Partitioning is optional, and if none is specified then the ent...istory.csv ). To load the data, run: CREATE TABLE "Generation History" AS FROM 'power-plant-generation-history.csv'; After partitioning by plant and ordering by date, it will have this layout: Plant Date MWh Boston 2019-01-02 564337 Boston 2019-01-03 507405... - https://man.mtool.work/duckdb/2.0/sql/functions/window_functions.html
- Reading and Writing Parquet Files 8248
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.par
...y listed column must exist in the files. Cannot be combined with schema , union_by_name or allow_empty , and requires hive_partitioning to be set explicitly. Added in DuckDB 2.0. STRUCT - can_have_nan Whether FLOAT and DOUBLE columns may contain NaN values....or compatibility reasons. BOOL false file_row_number Whether or not to include the file_row_number column. BOOL false hive_partitioning Whether or not to interpret the path as a Hive partitioned path . BOOL (auto-detected) int96_as How to read INT96 columns... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- Parquet Tips 8196
- Below is a collection of tips to help when dealing with Parquet files. Tips for Reading Parquet Files Use union_by_name When Loading Files with Different Schemas The union_by_name option can be used to unify the schema of files that have di
...parquet('flights*.parquet', union_by_name = true); Tips for Writing Parquet Files Using a glob pattern upon read or a Hive partitioning structure are good ways to transparently handle multiple files. Enabling PER_THREAD_OUTPUT If the final number of Parquet... - https://man.mtool.work/duckdb/2.0/data/parquet/tips.html
- Partitioned Writes 8127
- Examples Write a table to a Hive partitioned dataset of Parquet files: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)); Write a table to a Hive partitioned dataset of CSV files, allowing overwrites: COPY orders TO 'orde
...alue Hive layout. By default, the PARTITION_BY clause writes each partition to a col=value subdirectory following the Hive partitioning convention. The PARTITION_PATH option replaces this with an expression over the PARTITION_BY columns that computes the out... - https://man.mtool.work/duckdb/2.0/data/partitioning/partitioned_writes.html
- R Client 8075
- Installation To use the DuckDB R client, visit the R installation page . The latest stable version of the DuckDB R client is 2.0.0. Installation duckdb : R Client The DuckDB R client can be installed using the following command: install.pac
...he dataset in DuckDB to avoid reading 12 Parquet files into R's memory tbl(con, "read_parquet('dataset/**/*.parquet', hive_partitioning = true)") |> filter(month == "3") |> summarise(delay = mean(dep_time, na.rm = TRUE)) |> collect() Memory Limit You can use... - https://man.mtool.work/duckdb/2.0/clients/r.html
DuckDB 2.0 Documentation / man.mtool.work