Results of 1 - 10 of about 12 for hive (0.003 sec.)
- Hive Partitioning 13495
- Examples Read data from a Hive partitioned dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); Write a table to a Hive partitioned dataset: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, mont
...When reading, the partition columns are read from the directory structure and can be included or excluded depending on the hive_partitioning parameter. FROM read_parquet('test/*/*/*.parquet', hive_partitioning = false); -- will not include year, mont...h columns FROM read_parquet('test/*/*/*.parquet', hive_partitioning = true); -- will include year, month partition columns Hive Partitioning Hive partitioning is a partitioning.... Within each folder, the partition key has a value that is determined by the name of the folder. Below is an example of a Hive partitioned file hierarchy. The files are partitioned on two keys ( year and month ). orders ├── year=2021 │ ├── month=1 │... - https://man.mtool.work/duckdb/2.0/data/partitioning/hive_partitioning.html
- Partitioned Writes 11886
- Examples Write a table to a Hive partitioned dataset of Parquet files: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)); Write a table to a Hive partitioned dataset of CSV files, allowing overwrites: COPY orders TO 'orde
...rs' (FORMAT csv, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE); Write a table to a Hive partitioned dataset of GZIP-compressed CSV files, setting explicit data files' extension: COPY orders TO 'orders' (FORMAT...csv.gz'); Partitioned Writes When the PARTITION_BY clause is specified for the COPY statement , the files are written in a Hive partitioned folder hierarchy. The target is the name of the root directory (in the example above: orders ). The files are...specific filenames: {i} will be replaced by an index. {uuid} will be replaced by a 128 bits long UUID. Write a table to a Hive partitioned dataset of .parquet files, with an index in the filename: COPY orders TO 'orders' (FORMAT parquet, PARTITION_B... - https://man.mtool.work/duckdb/2.0/data/partitioning/partitioned_writes.html
- Directly Reading Files 8596
- DuckDB allows directly reading files via the read_text and read_blob functions. These functions accept a filename, a list of filenames, or a glob pattern. They output the content of each file as a VARCHAR or BLOB , respectively, along with
...ontent VARCHAR YES NULL NULL NULL size BIGINT YES NULL NULL NULL last_modified TIMESTAMP WITH TIME ZONE YES NULL NULL NULL Hive Partitioning Data can be read from Hive partitioned datasets. SELECT * FROM read_blob('data/parquet-testing/hive-partition... - https://man.mtool.work/duckdb/2.0/guides/file_formats/read_file.html
- Relational API 8384
- The Relational API builds a query one step at a time from Python. A relation, a DuckDBPyRelation , is a piece of SQL text: the methods of the Transformation and Functions sections return a new relation whose text extends the one they were c
...ULL . * filename : bool, keyword-only, default: None Add a column filename with the path of the file each row came from. * hive_partitioning : bool, keyword-only, default: None Interpret the path as a Hive partitioned path . * union_by_name : bool, k...eyword-only, default: None Align the columns of several files by name instead of by position. * hive_types : dict[str, str], keyword-only, default: None Types of the Hive partition columns, as a dict of column name to type...name. * hive_types_autocast : bool, keyword-only, default: None Whether to detect the types of the Hive partition columns. Example impo... - https://man.mtool.work/duckdb/2.0/clients/python/relational_api.html
- Reading and Writing Parquet Files 8118
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.par
...every listed column must exist in the files. Cannot be combined with schema , union_by_name or allow_empty , and requires hive_partitioning to be set explicitly. Added in DuckDB 2.0. STRUCT - can_have_nan Whether FLOAT and DOUBLE columns may contain...ept for compatibility reasons. BOOL false file_row_number Whether or not to include the file_row_number column. BOOL false hive_partitioning Whether or not to interpret the path as a Hive partitioned path . BOOL (auto-detected) int96_as How to read I... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- File Formats 8083
- Handling Parquet Files DuckDB has advanced support for Parquet files, which includes directly querying Parquet files . When deciding on whether to query these files directly or to first load them to the database, you need to consider severa
...f individual Parquet files moderate. Bestpractice The ideal range is between 100 MB and 10 GB per individual Parquet file. Hive Partitioning for Filter Pushdown When querying many files with filter conditions, performance can be improved by using a H...files, see the Parquet Tips page . Loading CSV Files CSV files are often distributed in compressed format such as GZIP archives ( .csv.gz ). DuckDB can decompress these files on the fly. In fact, this is typically faster than decompressing the files... - https://man.mtool.work/duckdb/2.0/guides/performance/file_formats.html
- S3 API Support 7959
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...: column_a column_b filename 1 examplevalue1 s3://bucket-name/file1.parquet 2 examplevalue1 s3://bucket-name/file2.parquet Hive Partitioning DuckDB also offers support for the Hive partitioning scheme , which is available when using HTTP(S) and S3 en... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Loading JSON 7959
- The DuckDB JSON reader can automatically infer which configuration flags to use by analyzing the JSON file. This will work correctly in most situations, and should be the first option attempted. In rare situations where the JSON reader cann
...or compatibility reasons. BOOL false format Can be one of auto , unstructured , newline_delimited and array . VARCHAR auto hive_partitioning Whether or not to interpret the path as a Hive partitioned path . BOOL (auto-detected) ignore_errors Whether... - https://man.mtool.work/duckdb/2.0/data/json/loading_json.html
- COPY Statement 7959
- Examples Read a CSV file into the lineitem table, using auto-detected CSV options: COPY lineitem FROM 'lineitem.csv'; Read a CSV file into the lineitem table, using manually specified CSV options: COPY lineitem FROM 'lineitem.csv' (DELIMITE
...es can be occasionally slightly over the limit. VARCHAR or BIGINT (empty) PARTITION_BY The columns to partition by using a Hive partitioning scheme, see the partitioned writes section . VARCHAR[] (empty) PARTITION_PATH An expression over the PARTITIO...N_BY columns that computes the output subdirectory for each partition, replacing the default col=value Hive layout. The result must be a relative path (no leading / , no .. segments) and non-empty. Requires PARTITION_BY and cannot...be combined with HIVE_FILE_PATTERN . See the partitioned writes section . Added in DuckDB 2.0. VARCHAR (empty) ORDER_BY A parenthesized list of... - https://man.mtool.work/duckdb/2.0/sql/statements/copy.html
- CSV Import 7924
- Examples The following examples use the flights.csv file. Read a CSV file from disk, auto-infer options: SELECT * FROM 'flights.csv'; Use the read_csv function with custom options: SELECT * FROM read_csv('flights.csv', delim = '|', header =
...ot of type VARCHAR , an error is raised. VARCHAR[] [] header First line of each file contains the column names. BOOL false hive_partitioning Interpret the path as a Hive partitioned path . BOOL (auto-detected) ignore_errors Ignore any parsing errors... - https://man.mtool.work/duckdb/2.0/data/csv/overview.html
DuckDB 2.0 Documentation / man.mtool.work