Results of 1 - 10 of about 16 for partitioned (0.002 sec.)
- Partitioned Writes 13343
- Examples Write a table to a Hive partitioned dataset of Parquet files: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)); Write a table to a Hive partitioned dataset of CSV files, allowing overwrites: COPY orders TO 'orde
...rs' (FORMAT csv, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE); Write a table to a Hive partitioned dataset of GZIP-compressed CSV files, setting explicit data files' extension: COPY orders TO 'orders' (FORMAT csv, PARTITI...ON_BY (year, month), COMPRESSION gzip, FILE_EXTENSION 'csv.gz'); Partitioned Writes When the PARTITION_BY clause is specified for the COPY statement , the files are written in a Hive partitioned fold...mit the maximum number of files the system can keep open before flushing to disk when writing using PARTITION_BY , use the partitioned_write_max_open_files configuration option (default: 100): SET partitioned_write_max_open_files = 10; Bestpractice Writing... - https://man.mtool.work/duckdb/2.0/data/partitioning/partitioned_writes.html
- Hive Partitioning 12078
- Examples Read data from a Hive partitioned dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); Write a table to a Hive partitioned dataset: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, mont
...hin each folder, the partition key has a value that is determined by the name of the folder. Below is an example of a Hive partitioned file hierarchy. The files are partitioned on two keys ( year and month ). orders ├── year=2021 │ ├── month=1 │ │ ├── file1...── file4.parquet └── file5.parquet Auto-detection By default the system tries to infer if the provided files are in a hive partitioned hierarchy. And if so, the hive_partitioning flag is enabled automatically. The auto-detection will look at the names of th...ypes: DATE , TIMESTAMP and BIGINT . To switch off the auto-detection, the flag hive_types_autocast = 0 can be set. Writing Partitioned Files See the Partitioned Writes section.... - https://man.mtool.work/duckdb/2.0/data/partitioning/hive_partitioning.html
- Writing to Iceberg 10726
- The DuckDB iceberg extension supports writing to Iceberg tables when an Iceberg catalog is attached. Writing to a table read directly from storage with iceberg_scan is not supported: the catalog is required to commit the new table metadata.
...after creation, see Table Properties Functions . UPDATE and DELETE The UPDATE and DELETE statements are supported on both partitioned and unpartitioned tables: UPDATE iceberg_catalog.sales.events SET event_name = 'view' WHERE id = 1; DELETE FROM iceberg_ca...nformation endpoint. Iceberg schema evolution is metadata-only, so no data files are rewritten. Partitioning Tables can be partitioned with the PARTITIONED BY clause using the Iceberg partition transforms : Transform Description ⟨column⟩ Identity: partition...ng. CREATE TABLE iceberg_catalog.default.events ( event_id BIGINT, user_id BIGINT, country VARCHAR, event_time TIMESTAMP ) PARTITIONED BY (day(event_time), bucket(16, user_id), truncate(2, country)); The partition spec of an existing table can be changed wi... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/writing_to_iceberg.htm...
- S3 API Support 8924
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...high speed. Writing to S3 works for both CSV and Parquet: COPY table_name TO 's3://⟨your-bucket⟩/⟨filename⟩.⟨extension⟩'; Partitioned copy to S3 also works: COPY table TO 's3://⟨your-bucket⟩/partitioned' ( FORMAT parquet, PARTITION_BY (⟨part_col_a⟩, ⟨part_...latency). To disable this check and force writing, an OVERWRITE_OR_IGNORE flag is added: COPY table TO 's3://⟨your-bucket⟩/partitioned' ( FORMAT parquet, PARTITION_BY (⟨part_col_a⟩, ⟨part_col_b⟩), OVERWRITE_OR_IGNORE true ); The naming scheme of the written...files looks like this: s3://⟨your-bucket⟩/partitioned/part_col_a=⟨val⟩/part_col_b=⟨val⟩/data_⟨thread_number⟩.parquet Configuration Some additional configuration options exist f... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Unity Catalog Extension 8612
- The unity_catalog extension attaches a Unity Catalog catalog, on Databricks or Unity Catalog OSS, as a DuckDB database. You can list, read, insert into and create its Delta Lake tables. It uses the delta extension to access the tables. For
...rnal_table (id INTEGER, name VARCHAR) WITH (location = 's3://⟨my_bucket⟩/⟨path⟩'); Both managed and external tables can be partitioned: CREATE TABLE my_catalog.my_schema.my_partitioned_table (id INTEGER, part VARCHAR) PARTITIONED BY (part); CREATE TABLE IF... - https://man.mtool.work/duckdb/2.0/core_extensions/unity_catalog.html
- Directly Reading Files 7936
- DuckDB allows directly reading files via the read_text and read_blob functions. These functions accept a filename, a list of filenames, or a glob pattern. They output the content of each file as a VARCHAR or BLOB , respectively, along with
...YES NULL NULL NULL last_modified TIMESTAMP WITH TIME ZONE YES NULL NULL NULL Hive Partitioning Data can be read from Hive partitioned datasets. SELECT * FROM read_blob('data/parquet-testing/hive-partitioning/simple/**/*.parquet') WHERE part IN ('a', 'b') A... - https://man.mtool.work/duckdb/2.0/guides/file_formats/read_file.html
- COPY Statement 7867
- Examples Read a CSV file into the lineitem table, using auto-detected CSV options: COPY lineitem FROM 'lineitem.csv'; Read a CSV file into the lineitem table, using manually specified CSV options: COPY lineitem FROM 'lineitem.csv' (DELIMITE
...er the limit. VARCHAR or BIGINT (empty) PARTITION_BY The columns to partition by using a Hive partitioning scheme, see the partitioned writes section . VARCHAR[] (empty) PARTITION_PATH An expression over the PARTITION_BY columns that computes the output sub...o leading / , no .. segments) and non-empty. Requires PARTITION_BY and cannot be combined with HIVE_FILE_PATTERN . See the partitioned writes section . Added in DuckDB 2.0. VARCHAR (empty) ORDER_BY A parenthesized list of expressions to sort the rows by bef... - https://man.mtool.work/duckdb/2.0/sql/statements/copy.html
- Configuration 7797
- DuckDB has a number of configuration options that can be used to change the behavior of the system. The configuration options can be set using either the SET statement or the PRAGMA statement . They can be reset to their original values usi
...column_gap Byte gap under which Parquet prefetch I/O ranges are coalesced (NULL lets the cost model adapt it) UBIGINT NULL partitioned_write_flush_threshold The number of rows threshold after which a thread state is flushed when writing using PARTITION_BY ....UBIGINT 524288 partitioned_write_max_open_files The maximum amount of files the system can keep open before flushing to disk when writing using PARTI... - https://man.mtool.work/duckdb/2.0/configuration/overview.html
- Loading JSON 7711
- The DuckDB JSON reader can automatically infer which configuration flags to use by analyzing the JSON file. This will work correctly in most situations, and should be the first option attempted. In rare situations where the JSON reader cann
...unstructured , newline_delimited and array . VARCHAR auto hive_partitioning Whether or not to interpret the path as a Hive partitioned path . BOOL (auto-detected) ignore_errors Whether to ignore parse errors (only possible when format is newline_delimited )... - https://man.mtool.work/duckdb/2.0/data/json/loading_json.html
- Metrics 7711
- DuckDB provides a set of metrics that can be used to monitor the performance and health of the database. Metrics are named ⟨group⟩.⟨metric⟩ , e.g., query.total_time or operator.timing . The set of collected metrics is controlled with the tr
..._self_join , optimizer.projection_pullup , optimizer.outer_join_simplification , optimizer.row_number_rewriter , optimizer.partitioned_execution , optimizer.partial_aggregate_pushdown . parser Metrics Metrics on parsing the SQL query. Name Type Unit Descrip... - https://man.mtool.work/duckdb/2.0/dev/metrics.html
DuckDB 2.0 Documentation / man.mtool.work