Results of 1 - 10 of about 119 for Parquet (0.001 sec.)
- Reading and Writing Parquet Files 10753
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.parquet
...quet'; If the file does not end in .parquet , use the read_parquet function: SELECT * FROM read_parquet('test.parq'); Use list parameter to read three Parquet files a...nd treat them as a single table: SELECT * FROM read_parquet(['file1.parquet', 'file2.parquet', 'file3.parquet']); Read Parquet files from a local directory, including its subdirector...ies: SELECT * FROM read_parquet('test'); A directory path recursively reads files with the .parquet extension. To read only files directly inside the dire... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- Querying Parquet Metadata 10168
- Parquet Metadata The parquet_metadata function can be used to query the metadata contained within a Parquet file, which reveals various internal details of the Parquet file such as the statistics of the different columns. This can be useful
...for figuring out what kind of skipping is possible in Parquet files, or even to obtain a quick overview of what the different columns contain. The function supports glob patterns to qu...ery metadata across multiple files in parallel: SELECT * FROM parquet_metadata('test.parquet'); SELECT * FROM parquet_metadata('data/*.parquet'); Below is a table of the columns returned by pa...ilter_offset BIGINT bloom_filter_length BIGINT min_is_exact BOOLEAN max_is_exact BOOLEAN row_group_compressed_bytes BIGINT Parquet Schema The parquet_schema function can be used to query the internal schema contained within a Parquet file. Note that thi... - https://man.mtool.work/duckdb/2.0/data/parquet/metadata.html
- Reading Multiple Files 9931
- DuckDB can read multiple files of different types (CSV, Parquet, JSON files) at the same time using either the glob syntax, or by providing a list of files to read. See the combining schemas page for tips on reading files with different sch
...utting a filename column: SELECT * FROM read_csv(['flights1.csv', 'flights2.csv'], union_by_name = true, filename = true); Parquet Read all files that match the glob pattern: SELECT * FROM 'test/*.parquet'; Read three Parquet files and treat them as a s...ingle table: SELECT * FROM read_parquet(['file1.parquet', 'file2.parquet', 'file3.parquet']); Read all Parquet files from two specific folders: SELECT * FROM read..._parquet(['folder1/*.parquet', 'folder2/*.parquet']); Read all Parquet files that match the glob pattern at any depth: SELECT * FRO... - https://man.mtool.work/duckdb/2.0/data/multiple_files/overview.html
- File Formats 9346
- Handling Parquet Files DuckDB has advanced support for Parquet files, which includes directly querying Parquet files . When deciding on whether to query these files directly or to first load them to the database, you need to consider severa
...l factors. Reasons for Querying Parquet Files Availability of basic statistics: Parquet files use a columnar storage format and contain basic statistics such as z...onemaps . Thanks to these features, DuckDB can leverage optimizations such as projection and filter pushdown on Parquet files. Therefore, workloads that combine projection, filtering, and aggregation tend to perform quite well when run on Par...quet files. Storage considerations: Loading the data from Parquet files will require approximately the same amount of space for the DuckDB database file. Therefore, if the available disk s... - https://man.mtool.work/duckdb/2.0/guides/performance/file_formats.html
- Parquet Import 9110
- To read data from a Parquet file, use the read_parquet function in the FROM clause of a query: SELECT * FROM read_parquet('input.parquet'); Alternatively, you can omit the read_parquet function and let DuckDB infer it from the extension: SE
...LECT * FROM 'input.parquet'; To create a new table using the result from a query, use the CREATE TABLE ... AS SELECT statement : CREATE TABLE new_tbl...AS SELECT * FROM read_parquet('input.parquet'); To load data into an existing table from a query, use INSERT INTO from a SELECT statement: INSERT INTO t...bl SELECT * FROM read_parquet('input.parquet'); Alternatively, use the COPY statement to load data from a Parquet file into an existing table: COPY tbl... - https://man.mtool.work/duckdb/2.0/guides/file_formats/parquet_import.html
- Parquet Encryption 9023
- Starting with version 0.10.0, DuckDB supports reading and writing encrypted Parquet files. DuckDB broadly follows the Parquet Modular Encryption specification with some limitations . Reading and Writing Encrypted Files Using the PRAGMA add_
...ion, named encryption keys of 128, 192, or 256 bits can be added to a session. These keys are stored in-memory: PRAGMA add_parquet_key('key128', '0123456789112345'); PRAGMA add_parquet_key('key192', '012345678911234501234567'); PRAGMA add_parquet_key('k...ey256', '01234567891123450123456789112345'); PRAGMA add_parquet_key('key256base64', 'MDEyMzQ1Njc4OTExMjM0NTAxMjM0NTY3ODkxMTIzNDU='); Writing Encrypted Parquet Files After specifying the...key (e.g., key256 ), files can be encrypted as follows: COPY tbl TO 'tbl.parquet' (ENCRYPTION_CONFIG {footer_key: 'key256'}); Reading Encrypted Parquet Files An encrypted Parquet file using a specific ke... - https://man.mtool.work/duckdb/2.0/data/parquet/encryption.html
- Hive Partitioning 9023
- Examples Read data from a Hive partitioned dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); Write a table to a Hive partitioned dataset: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, mont
...read from the directory structure and can be included or excluded depending on the hive_partitioning parameter. FROM read_parquet('test/*/*/*.parquet', hive_partitioning = false); -- will not include year, month columns FROM read_parquet('test/*/*/*.pa...file hierarchy. The files are partitioned on two keys ( year and month ). orders ├── year=2021 │ ├── month=1 │ │ ├── file1.parquet │ │ └── file2.parquet │ └── month=2 │ └── file3.parquet └── year=2022 ├── month=11 │ ├── file4.parquet │ └── file5.parquet...└── month=12 └── file6.parquet Files stored in this hierarchy can be read using the hive_partitioning flag. SELECT * FROM read_parquet('orders/*/*/*.parq... - https://man.mtool.work/duckdb/2.0/data/partitioning/hive_partitioning.html
- Parquet Tips 8711
- Below is a collection of tips to help when dealing with Parquet files. Tips for Reading Parquet Files Use union_by_name When Loading Files with Different Schemas The union_by_name option can be used to unify the schema of files that have di
...fferent or missing columns. For files that do not have certain columns, NULL values are filled in: SELECT * FROM read_parquet('flights*.parquet', union_by_name = true); Tips for Writing Parquet Files Using a glob pattern upon read or a Hive partiti...oning structure are good ways to transparently handle multiple files. Enabling PER_THREAD_OUTPUT If the final number of Parquet files is not important, writing one file per thread can significantly improve performance: COPY (FROM generate_series(10_0...00_000)) TO 'test.parquet' (FORMAT parquet, PER_THREAD_OUTPUT); Selecting a ROW_GROUP_SIZE The ROW_GROUP_SIZE parameter specifies the minimum number... - https://man.mtool.work/duckdb/2.0/data/parquet/tips.html
- Importing Data 8649
- The first step to using a database system is to insert data into that system. DuckDB can directly connect to many popular data sources and offers several data ingestion methods that allow you to easily and efficiently fill up the database.
...E ... AS SELECT statement : CREATE TABLE test AS SELECT * FROM 'test.csv'; For more details, see the page on CSV loading . Parquet Loading Parquet files can be efficiently loaded and queried using their filename: SELECT * FROM 'test.parquet'; Alternativ...ely, use the read_parquet function : SELECT * FROM read_parquet('test.parquet'); Or use the COPY statement : COPY tbl FROM 'test.parquet'; For more...details, see the page on Parquet loading . JSON Loading JSON files can be efficiently loaded and queried using their filename: SELECT * FROM 'test.json'; A... - https://man.mtool.work/duckdb/2.0/data/overview.html
- Data Ingestion 8599
- This page contains examples for data ingestion to Python using DuckDB. First, import the DuckDB package: import duckdb Then, proceed with any of the following sections. CSV Files CSV files can be read using the read_csv function, called eit
...write a relation to a CSV file, use write_csv . If you pass overwrite=False then it refuses to replace a file that exists. Parquet Files Parquet files can be read using the read_parquet function, called either from within Python or directly from within...SQL. Read from a single Parquet file: duckdb.read_parquet("example.parquet") Read multiple Parquet files from a folder: duckdb.read_parquet("folder/*.parq...uet") Read a Parquet file over https : duckdb.read_parquet("https://some.url/some_file.parquet") Read a list of Parquet files: duckdb.read_parq... - https://man.mtool.work/duckdb/2.0/clients/python/data_ingestion.html
DuckDB 2.0 Documentation / man.mtool.work