Results of 1 - 10 of about 16 for folder (0.001 sec.)
- SQL on Apache Arrow 14192
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
....Table.from_pydict({'i': [1, 2, 3, 4], 'j': ["one", "two", "three", "four"]}) # create example Parquet files and save in a folder parquet_folder = tempfile.mkdtemp() pq.write_to_dataset(my_arrow_table, parquet_folder) # link to Parquet files using an A...rrow Dataset my_arrow_dataset = ds.dataset(parquet_folder) # query the Apache Arrow Dataset "my_arrow_dataset" and return as an Arrow Table results = con.sql("SELECT * FROM my_arro....Table.from_pydict({'i': [1, 2, 3, 4], 'j': ["one", "two", "three", "four"]}) # create example Parquet files and save in a folder parquet_folder = tempfile.mkdtemp() pq.write_to_dataset(my_arrow_table, parquet_folder) # link to Parquet files using an A... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Data Ingestion 10951
- This page contains examples for data ingestion to Python using DuckDB. First, import the DuckDB package: import duckdb Then, proceed with any of the following sections. CSV Files CSV files can be read using the read_csv function, called eit
...d file. Read from a file using fully auto-detected settings: duckdb.read_csv("example.csv") Read multiple CSV files from a folder: duckdb.read_csv("folder/*.csv") Specify options on how the CSV is formatted internally: duckdb.read_csv("example.csv", he...rom within SQL. Read from a single Parquet file: duckdb.read_parquet("example.parquet") Read multiple Parquet files from a folder: duckdb.read_parquet("folder/*.parquet") Read a Parquet file over https : duckdb.read_parquet("https://some.url/some_file....tored within the JSON file. Read from a single JSON file: duckdb.read_json("example.json") Read multiple JSON files from a folder: duckdb.read_json("folder/*.json") Directly read a JSON file from within SQL: duckdb.sql("SELECT * FROM 'example.json'") C... - https://man.mtool.work/duckdb/2.0/clients/python/data_ingestion.html
- S3 Iceberg Import 9513
- Prerequisites Loading an Iceberg file from S3 requires both the httpfs and iceberg extensions. Install them using the INSTALL SQL command. You only need to install extensions once. INSTALL httpfs; INSTALL iceberg; To load the extensions, us
..., Iceberg tables can be read from S3 using the following command: SELECT * FROM iceberg_scan('s3://⟨bucket⟩/⟨iceberg_table_folder⟩/metadata/⟨id⟩.metadata.json'); If you pass the table folder instead of the metadata file, DuckDB looks for a version-hint... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_iceberg_import...
- Reading Multiple Files 9270
- DuckDB can read multiple files of different types (CSV, Parquet, JSON files) at the same time using either the glob syntax, or by providing a list of files to read. See the combining schemas page for tips on reading files with different sch
...emas. CSV Read all files with a name ending in .csv in the folder dir : SELECT * FROM 'dir/*.csv'; Read all files with a name ending in .csv , two directories deep: SELECT * FROM '*/*/*.cs...v'; Read all files with a name ending in .csv , at any depth in the folder dir : SELECT * FROM 'dir/**/*.csv'; Read the CSV files flights1.csv and flights2.csv : SELECT * FROM read_csv(['flights1.c...SELECT * FROM read_parquet(['file1.parquet', 'file2.parquet', 'file3.parquet']); Read all Parquet files from two specific folders: SELECT * FROM read_parquet(['folder1/*.parquet', 'folder2/*.parquet']); Read all Parquet files that match the glob patte... - https://man.mtool.work/duckdb/2.0/data/multiple_files/overview.html
- Storage Versions and Format 9149
- Compatibility Backward Compatibility Backward compatibility refers to the ability of a newer DuckDB version to read storage files created by an older DuckDB version. Version 0.10 is the first release of DuckDB that supports backward compati
...un the SQL statement EXPORT DATABASE 'tmp' . This allows you to save the whole state of the current database in use inside folder tmp . The content of the tmp folder will be overridden, so choose an empty/non yet existing location. Then, start the newe...r DuckDB and execute IMPORT DATABASE 'tmp' (pointing to the previously populated folder) to load the database, which can be then saved to the file you pointed DuckDB to. A Bash script to achieve this (to be ada...old format, mydata.new.db will contain the same data but in a format accessible by the more recent DuckDB version, and the folder tmp will hold the same data in a universal format as different files. Check EXPORT documentation for more details on the s... - https://man.mtool.work/duckdb/2.0/internals/storage.html
- Hive Partitioning 8924
- Examples Read data from a Hive partitioned dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); Write a table to a Hive partitioned dataset: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, mont
...itioning strategy that is used to split a table into multiple files based on partition keys . The files are organized into folders. Within each folder, the partition key has a value that is determined by the name of the folder. Below is an example of a...ierarchy. And if so, the hive_partitioning flag is enabled automatically. The auto-detection will look at the names of the folders and search for a 'key' = 'value' pattern. This behavior can be overridden by using the hive_partitioning option of the re... - https://man.mtool.work/duckdb/2.0/data/partitioning/hive_partitioning.html
- DBine SQL Editor 8196
- DBine is a free, open-source (Apache 2.0) desktop database manager for Windows, macOS, and Linux that works with 130+ databases and cloud services. Besides a SQL editor, it includes notebooks, charts built from query results, and schema com
...ist of engines. Give the connection a name and enter the path to your DuckDB database file in Database file , or click the folder button to pick it. Leave the field empty or enter :memory: to open an in-memory database. Click Test connection to confirm... - https://man.mtool.work/duckdb/2.0/guides/sql_editors/dbine.html
- Troubleshooting of Extensions 7832
- You might be visiting this page directed via a DuckDB error message, similar to: INSTALL non_existing; HTTP Error: Failed to download extension "non_existing" at URL "http://extensions.duckdb.org/v2.0.0/osx_arm64/non_existing.duckdb_extensi
...regardless of the transport protocol. If successful, this will download and unpack the extension to the default Downloads folder, so that from SQL you can run: INSTALL '~/Downloads/spatial.duckdb_extension'; -- or FORCE INSTALL '~/Downloads/spatial.du... - https://man.mtool.work/duckdb/2.0/extensions/troubleshooting.html
- Writing Tests 7711
- Development and Testing It is crucial that any new features that get added have correct tests that not only test the “happy path”, but also test edge cases and incorrect usage of the feature. In this section, we describe how DuckDB tests ar
...structured and how to make new tests for DuckDB. The tests can be run by running the unittest program located in the test folder. For the default compilations this is located in either build/release/test/unittest (release) or build/debug/test/unittest... - https://man.mtool.work/duckdb/2.0/dev/sqllogictest/writing_tests.html
- Partitioned Writes 7607
- Examples Write a table to a Hive partitioned dataset of Parquet files: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)); Write a table to a Hive partitioned dataset of CSV files, allowing overwrites: COPY orders TO 'orde
...oned Writes When the PARTITION_BY clause is specified for the COPY statement , the files are written in a Hive partitioned folder hierarchy. The target is the name of the root directory (in the example above: orders ). The files are written in-order in... - https://man.mtool.work/duckdb/2.0/data/partitioning/partitioned_writes.html
DuckDB 2.0 Documentation / man.mtool.work