Results of 1 - 10 of about 36 for filename (0.001 sec.)
- Reading Multiple Files 12694
- DuckDB can read multiple files of different types (CSV, Parquet, JSON files) at the same time using either the glob syntax, or by providing a list of files to read. See the combining schemas page for tips on reading files with different sch
...ghts1.csv', 'flights2.csv']); Read the CSV files flights1.csv and flights2.csv , unifying schemas by name and outputting a filename column: SELECT * FROM read_csv(['flights1.csv', 'flights2.csv'], union_by_name = true, filename = true); Parquet Read all...other extensions are ignored by directory discovery, even if they contain Parquet data. To read those files, supply their filenames explicitly. If the directory contains no matching files, the query raises an error. Directory paths do not change how sch...use union_by_name = true as described in combining schemas . List Parameter The read_parquet function can accept a list of filenames as the input parameter. Read three Parquet files and treat them as a single table: SELECT * FROM read_parquet(['file1.par... - https://man.mtool.work/duckdb/2.0/data/multiple_files/overview.html
- Loading JSON 12187
- The DuckDB JSON reader can automatically infer which configuration flags to use by analyzing the JSON file. This will work correctly in most situations, and should be the first option attempted. In rare situations where the JSON reader cann
...tions for Reading JSON Objects The following table functions are used to read JSON: Function Description read_json_objects(filename) Read a JSON object from filename , where filename can also be a list of files or a glob pattern. read_ndjson_objects(file...name) Alias for read_json_objects with the parameter format set to newline_delimited . read_json_objects_auto(filename) Alias for read_json_objects with the parameter format set to auto . Parameters These functions have the following paramet...e.g., t.json.gz will use gzip, t.json will use none). Options are none , gzip , zstd and auto_detect . VARCHAR auto_detect filename Whether or not an extra filename column should be included in the result. Since DuckDB 1.3.0, the filename column is added... - https://man.mtool.work/duckdb/2.0/data/json/loading_json.html
- Directly Reading Files 11252
- DuckDB allows directly reading files via the read_text and read_blob functions. These functions accept a filename, a list of filenames, or a glob pattern. They output the content of each file as a VARCHAR or BLOB , respectively, along with
...results in a single row with the content field holding the entire content of the respective file. SELECT size, parse_path(filename), content FROM read_text('test/sql/table_function/files/*.txt'); size parse_path(filename) content 12 [test, sql, table_fu...ext is 3.9 GiB. read_blob The read_blob table function reads from the selected source(s) to a BLOB : SELECT size, content, filename FROM read_blob('test/sql/table_function/files/*'); size content filename 178 PK\x03\x04\x0A\x00\x00\x00\x00\x00\xACi=X\x14...ead_text and read_blob are identical: DESCRIBE FROM read_text('README.md'); column_name column_type null key default extra filename VARCHAR YES NULL NULL NULL content VARCHAR YES NULL NULL NULL size BIGINT YES NULL NULL NULL last_modified TIMESTAMP WITH... - https://man.mtool.work/duckdb/2.0/guides/file_formats/read_file.html
- Command Line Arguments 9968
- The table below summarizes DuckDB's command line options. To list all command line options, use the command: duckdb -help For a list of dot commands available in the CLI shell, see the Dot Commands page . Argument Description -agent Render
...sv Set output mode to csv -dark-mode Use dark mode colors for syntax highlighting -echo Print commands before execution -f FILENAME Run the script in FILENAME and exit. Note that the ~/.duckdbrc is read and executed first (if it exists) -format Format SQ...L read from stdin , writing the result to stdout -format-file FILENAME Format the SQL in FILENAME , writing the result to stdout -init FILENAME Run the script in FILENAME upon startup (instead... - https://man.mtool.work/duckdb/2.0/clients/cli/arguments.html
- Importing Data 9889
- The first step to using a database system is to insert data into that system. DuckDB can directly connect to many popular data sources and offers several data ingestion methods that allow you to easily and efficiently fill up the database.
...ore details, see the page on CSV loading . Parquet Loading Parquet files can be efficiently loaded and queried using their filename: SELECT * FROM 'test.parquet'; Alternatively, use the read_parquet function : SELECT * FROM read_parquet('test.parquet');...more details, see the page on Parquet loading . JSON Loading JSON files can be efficiently loaded and queried using their filename: SELECT * FROM 'test.json'; Alternatively, use the read_json function : SELECT * FROM read_json('test.json'); Or use the C...OPY statement : COPY tbl FROM 'test.json'; For more details, see the page on JSON loading . Returning the Filename Since DuckDB 1.3.0, the CSV, JSON and Parquet readers support the filename virtual column: COPY (FROM (VALUES (42), (43))... - https://man.mtool.work/duckdb/2.0/data/overview.html
- Partitioned Writes 9794
- Examples Write a table to a Hive partitioned dataset of Parquet files: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)); Write a table to a Hive partitioned dataset of CSV files, allowing overwrites: COPY orders TO 'orde
...g data into many small partitions is expensive. It is generally recommended to have at least 100 MB of data per partition. Filename Pattern By default, files will be named data_0.parquet or data_0.csv . With the flag FILENAME_PATTERN a pattern with {i} o...r {uuid} can be defined to create specific filenames: {i} will be replaced by an index. {uuid} will be replaced by a 128 bits long UUID. Write a table to a Hive partitioned d...ataset of .parquet files, with an index in the filename: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE, FILENAME_PATTERN 'orders_{i}');... - https://man.mtool.work/duckdb/2.0/data/partitioning/partitioned_writes.html
- S3 API Support 9715
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...e/that/uses/a/gcs/secret⟩.parquet'); Reading Reading files from S3 is now as simple as: SELECT * FROM 's3://⟨your-bucket⟩/⟨filename⟩.⟨extension⟩'; Partial Reading The httpfs extension supports partial reading from S3 buckets. Pinning Object Versions By d...e; Reading Multiple Files Multiple files are also possible, for example: SELECT * FROM read_parquet([ 's3://⟨your-bucket⟩/⟨filename-1⟩.parquet', 's3://⟨your-bucket⟩/⟨filename-2⟩.parquet' ]); Globbing File globbing is implemented using the ListObjectsV2 A...ECT count(*) FROM read_parquet('s3://⟨your-bucket⟩/folder*/100?/t[0-9].parquet'); A useful feature when using globs is the filename option, which adds a column named filename that encodes the file that a particular row originated from: SELECT * FROM read... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Files Created by DuckDB 8954
- DuckDB creates several files and directories on disk. This page lists both the global and the local ones. Global Files and Directories DuckDB creates the following global files and directories in the user's home directory (denoted with ~ ):
...r in-memory connections) or relative to the database file (for persistent connections): Name Description Example ⟨database_filename⟩ Database file. Only created in on-disk mode. The file can have any extension with typical extensions being .duckdb , .db...and .ddb . weather.duckdb .tmp/ Temporary directory. Only created in in-memory mode. .tmp/ ⟨database_filename⟩.tmp/ Temporary directory. Only created in on-disk mode. weather.duckdb.tmp/ ⟨database_filename⟩.wal Write-ahead log file.... - https://man.mtool.work/duckdb/2.0/operations_manual/footprint_of_duckdb/files_cr...
- Avro Extension 8273
- The avro extension enables DuckDB to read Apache Avro files. The avro extension was released as a community extension in late 2024 and became a core extension in early 2025. The read_avro Function The extension adds a single DuckDB function
...les to the functions: FROM read_avro('some_file_*.avro'); FROM read_avro(['some_file_1.avro', 'some_file_2.avro']); If the filenames somehow contain valuable information (as is unfortunately all-too-common), you can pass the filename argument to read_avr...o : FROM read_avro('some_file_*.avro', filename=true); This will result in an additional column in the result set that contains the actual filename of the Avro file. Sche... - https://man.mtool.work/duckdb/2.0/core_extensions/avro.html
- S3 Parquet Export 8178
- To write a Parquet file to S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL command: LOAD htt
...rectly configured, Parquet files can be written to S3 using the following command: COPY ⟨table_name⟩ TO 's3://⟨s3-bucket⟩/⟨filename⟩.parquet'; Similarly, Google Cloud Storage (GCS) is supported through the Interoperability API. You need to create HMAC ke...xRfiCYEXAMPLEKEY⟩' ); After setting up the GCS credentials, you can export using: COPY ⟨table_name⟩ TO 'gs://⟨gcs_bucket⟩/⟨filename⟩.parquet';... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_export.html
DuckDB 2.0 Documentation / man.mtool.work