Results of 1 - 10 of about 144 for files (0.002 sec.)
- File Formats 10630
- Handling Parquet Files DuckDB has advanced support for Parquet files, which includes directly querying Parquet files . When deciding on whether to query these files directly or to first load them to the database, you need to consider severa
...l factors. Reasons for Querying Parquet Files Availability of basic statistics: Parquet files use a columnar storage format and contain basic statistics such as zonemap...s . Thanks to these features, DuckDB can leverage optimizations such as projection and filter pushdown on Parquet files. Therefore, workloads that combine projection, filtering, and aggregation tend to perform quite well when run on Parquet f...iles. Storage considerations: Loading the data from Parquet files will require approximately the same amount of space for the DuckDB database file. Therefore, if the available disk space i... - https://man.mtool.work/duckdb/2.0/guides/performance/file_formats.html
- Reading Multiple Files 10393
- DuckDB can read multiple files of different types (CSV, Parquet, JSON files) at the same time using either the glob syntax, or by providing a list of files to read. See the combining schemas page for tips on reading files with different sch
...emas. CSV Read all files with a name ending in .csv in the folder dir : SELECT * FROM 'dir/*.csv'; Read all files with a name ending in .csv , two...directories deep: SELECT * FROM '*/*/*.csv'; Read all files with a name ending in .csv , at any depth in the folder dir : SELECT * FROM 'dir/**/*.csv'; Read the CSV files flights1.cs...v and flights2.csv : SELECT * FROM read_csv(['flights1.csv', 'flights2.csv']); Read the CSV files flights1.csv and flights2.csv , unifying schemas by name and outputting a filename column: SELECT * FROM read_csv(['flight... - https://man.mtool.work/duckdb/2.0/data/multiple_files/overview.html
- Iceberg Functions 9305
- The iceberg extension provides the following functions: Function Description iceberg_scan(source) Reads an Iceberg table from a path. iceberg_metadata(source) Returns the manifest entries of an Iceberg table. iceberg_snapshots(source) Retur
...cs of an Iceberg table. iceberg_load_table_response(table) Returns the raw LoadTable response of the REST Catalog. iceberg_files(source) Returns the data and delete files of an Iceberg table snapshot. iceberg_manifests(source) Returns the manifest fil..._partition_specs(source) Returns the partition specs of an Iceberg table. iceberg_scan_plan(table) Returns the scan tasks (files to read) planned for an Iceberg table. iceberg_view_metadata(view) Returns the raw LoadView response of the REST Catalog....iceberg_rollback_to_snapshot(table, snapshot_id) Rolls back an Iceberg table to an earlier snapshot. iceberg_rewrite_data_files(table) Compacts the data files of an Iceberg table. iceberg_to_ducklake(iceberg_catalog, ducklake_catalog) Copies the Iceb... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/iceberg_functions.html
- Combining Schemas 9270
- Examples Read a set of CSV files combining columns by position: SELECT * FROM read_csv('flights*.csv'); Read a set of CSV files combining columns by name: SELECT * FROM read_csv('flights*.csv', union_by_name = true); Combining Schemas When
...reading from multiple files, we have to combine schemas from those files. That is because each file has its own schema that can differ from the other...files. DuckDB offers two ways of unifying schemas of multiple files: by column position and by column name . By default, DuckDB reads the schema of the first file provided, and then matches...the columns of subsequent files to it by column name: columns missing from the first file are dropped, and columns of the first file missing from a later... - https://man.mtool.work/duckdb/2.0/data/multiple_files/combining_schemas.html
- Reading and Writing Parquet Files 9021
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.par
...parquet , use the read_parquet function: SELECT * FROM read_parquet('test.parq'); Use list parameter to read three Parquet files and treat them as a single table: SELECT * FROM read_parquet(['file1.parquet', 'file2.parquet', 'file3.parquet']); Read Pa...rquet files from a local directory, including its subdirectories: SELECT * FROM read_parquet('test'); A directory path recursively rea...ds files with the .parquet extension. To read only files directly inside the directory, use test/*.parquet instead. See directory p... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- Storage Versions and Format 8844
- Compatibility Backward Compatibility Backward compatibility refers to the ability of a newer DuckDB version to read storage files created by an older DuckDB version. Version 0.10 is the first release of DuckDB that supports backward compati
...bility in the storage format. DuckDB 0.10 can read and operate on files created by the previous DuckDB version – DuckDB 0.9. For future DuckDB versions, our goal is to ensure that any DuckDB ver...sion released after can read files created by previous versions, starting from this release. We want to ensure that the file format is fully backward compati...ble. This allows you to keep data stored in DuckDB files around and guarantees that you will be able to read the files without having to worry about which version the file was wri... - https://man.mtool.work/duckdb/2.0/internals/storage.html
- Working with Huge Databases 8489
- This page contains information for working with huge DuckDB database files. While most DuckDB databases are well below 1 TB, in our 2024 user survey , 1% of respondents used DuckDB files of 2 TB or more (corresponding to roughly 10 TB of CS
...V files). DuckDB's native database format supports huge database files without any practical restrictions, however, there are a fe...w things to keep in mind when working with huge database files. Object storage systems have lower limits on file sizes than block-based storage systems. For example, AWS S3 limits the f...after adding a few rows to a table in the TPC-H SF1000 database takes approximately 5 seconds. On block-based storage, the file system has a significant effect on performance when working with large files. On Linux, DuckDB performs best with XFS on la... - https://man.mtool.work/duckdb/2.0/guides/performance/working_with_huge_databases...
- Gitignore for DuckDB 8347
- If you work in a Git repository, you may want to configure your Gitignore to disable tracking files created by DuckDB . These potentially include the DuckDB database, write-ahead log, and temporary files. Sample Gitignore Files In the follo
...wing, we present sample Gitignore configuration snippets for DuckDB. Ignore Temporary Files but Keep Database This configuration is useful if you would like to keep the database file in the version control system:...*.wal *.tmp/ Ignore Database and Temporary Files If you would like to ignore both the database and the temporary files, extend the Gitignore file to include the database f...his depends on the extension you use for your DuckDB databases ( .duckdb , .db , .ddb , etc.). For example, if your DuckDB files use the .duckdb extension, add the following lines to your .gitignore file: *.duckdb* *.wal *.tmp/... - https://man.mtool.work/duckdb/2.0/operations_manual/footprint_of_duckdb/gitignor...
- CSV Import 8312
- Examples The following examples use the flights.csv file. Read a CSV file from disk, auto-infer options: SELECT * FROM 'flights.csv'; Use the read_csv function with custom options: SELECT * FROM read_csv('flights.csv', delim = '|', header =
...M -first syntax to omit SELECT * . CREATE TABLE ontime AS FROM 'flights.csv'; CSV Loading CSV loading, i.e., importing CSV files to the database, is a very common, and yet surprisingly tricky, task. While CSVs seem simple on the surface, there are a l...ot of inconsistencies found within CSV files that can make loading them a challenge. CSV files come in many different varieties, are often corrupt, and do not have a s...This option is only supported by the read_csv function. BOOL false allow_empty If the file pattern or file list matches no files, return an empty result (with a single empty column of type BOOL ) instead of throwing an error. Not available in the COPY... - https://man.mtool.work/duckdb/2.0/data/csv/overview.html
- Directly Reading Files 8075
- DuckDB allows directly reading files via the read_text and read_blob functions. These functions accept a filename, a list of filenames, or a glob pattern. They output the content of each file as a VARCHAR or BLOB , respectively, along with
...metadata such as the file size and last modified time. read_text The read_text table function reads from the selected source(s) to a VARCHAR . Each f...entire content of the respective file. SELECT size, parse_path(filename), content FROM read_text('test/sql/table_function/files/*.txt'); size parse_path(filename) content 12 [test, sql, table_function, files, one.txt] Hello World! 2 [test, sql, table..._function, files, three.txt] 42 10 [test, sql, table_function, files, two.txt] Foo Bar\nFöö Bär DuckDB first validates the file content as... - https://man.mtool.work/duckdb/2.0/guides/file_formats/read_file.html
DuckDB 2.0 Documentation / man.mtool.work