Results of 1 - 10 of about 21 for ://⟨ (0.002 sec.)
- S3 API Support 12499
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...your-region⟩.amazonaws.com' . Now, to query using the above secret, simply query any s3:// prefixed file: SELECT * FROM 's3://⟨your-bucket⟩/⟨your_file⟩.parquet'; credential_chain Provider The credential_chain provider automatically fetches credential...dential_chain, CHAIN config, REGION '⟨eu-west-1⟩', KMS_KEY_ID 'arn:aws:kms:⟨region⟩:⟨account_id⟩:⟨key⟩/⟨key_id⟩', SCOPE 's3://⟨bucket-sub-path⟩' ); R2 Secrets While Cloudflare R2 uses the regular S3 API, DuckDB has a special Secret type, R2 , to make...y. Finally, R2 secrets are only available when using URLs starting with r2:// , for example: SELECT * FROM read_parquet('r2://⟨some-file-that-uses-an-r2-secret⟩.parquet'); GCS Secrets While Google Cloud Storage is accessed by DuckDB using the S3 API,... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Azure Extension 12414
- The azure extension lets DuckDB read and write files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2. Installing and Loading The azure extension will be transparently autoloaded on first use from the official extension reposit
.... Then you can query Azure storage as follows: Azure Blob Storage Allowed URI schemes: az or azure SELECT count(*) FROM 'az://⟨my_container⟩/⟨path⟩/⟨my_file⟩.⟨parquet_or_csv⟩'; Globs are also supported: SELECT * FROM 'az://⟨my_container⟩/⟨path⟩/*.csv...'; SELECT * FROM 'az://⟨my_container⟩/⟨path⟩/**'; Or with a fully qualified path syntax: SELECT count(*) FROM 'az://⟨my_storage_account⟩.blob.core....windows.net/⟨my_container⟩/⟨path⟩/⟨my_file⟩.⟨parquet_or_csv⟩'; SELECT * FROM 'az://⟨my_storage_account⟩.blob.core.windows.net/⟨my_container⟩/⟨path⟩/*.csv'; Azure Data Lake Storage (ADLS) Allowed URI schemes:... - https://man.mtool.work/duckdb/2.0/core_extensions/azure.html
- Delta Extension 11399
- The delta extension adds support for the Delta Lake open-source storage format . It is built using the Delta Kernel . The extension reads Delta tables, local and remote, and writes them: it can create tables and append to them. For implemen
...SELECT * FROM my_table; Reading from an S3 Bucket To scan a Delta table in an S3 bucket , run: SELECT * FROM delta_scan('s3://⟨my_bucket⟩/⟨path⟩'); To authenticate, create a secret : CREATE SECRET ( TYPE s3, PROVIDER credential_chain ); SELECT * FROM...delta_scan('s3://⟨my_bucket⟩/⟨path⟩'); For a public bucket, you may need to set its region in a secret: CREATE SECRET ( TYPE s3, REGION '⟨my_...region⟩' ); SELECT * FROM delta_scan('s3://⟨my_bucket⟩/⟨path⟩'); Reading from Azure Blob Storage To scan a Delta table in an Azure Blob Storage container , run: SELECT... - https://man.mtool.work/duckdb/2.0/core_extensions/delta.html
- SeaweedFS Import 9657
- Prerequisites For SeaweedFS , the S3-compatible gateway allows you to use DuckDB's S3 support to read and write from SeaweedFS buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This only nee
...S credentials, you can query the data using DuckDB's built-in methods, such as read_csv or read_parquet : SELECT * FROM 's3://⟨your-bucket⟩/⟨file⟩.csv'; SELECT * FROM read_parquet('s3://⟨your-bucket⟩/⟨file⟩.parquet'); Writing works the same way, incl...uding glob reads over the result: COPY (SELECT 42 AS answer) TO 's3://⟨your-bucket⟩/sample/answer.parquet'; SELECT * FROM read_parquet('s3://⟨your-bucket⟩/sample/*.parquet'); SeaweedFS also supp... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/seaweedfs_import....
- Lance Extension 8896
- The lance extension adds support for reading and writing Lance tables. Lance is a modern lakehouse format optimized for ML/AI workloads, with native cloud storage support. Installing and Loading You can install the lance extension from Duck
...storage reference Query a Lance Dataset Local file: SELECT * FROM '⟨path/to/dataset.lance⟩' LIMIT 10; S3: SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10; To access object store URIs (e.g., s3://... ), configure a TYPE lance secret using the...Secrets Manager : CREATE SECRET ( TYPE lance, PROVIDER credential_chain, SCOPE 's3://bucket/' ); SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10; Write a Lance Dataset Use the COPY ... TO ... statement to materialize query results a...PE lance secret for that scope using the Secrets Manager : CREATE SECRET ( TYPE lance, PROVIDER credential_chain, SCOPE 's3://⟨bucket⟩/' ); COPY (SELECT 1 AS id) TO 's3://⟨bucket/path/to/out.lance⟩' (FORMAT lance, MODE 'overwrite'); Create a Lance Da... - https://man.mtool.work/duckdb/2.0/core_extensions/lance.html
- Google Cloud Storage Import 8811
- Prerequisites The Google Cloud Storage (GCS) can be used via the httpfs extension . This can be installed with the INSTALL httpfs SQL command. This only needs to be run once. Credentials and Configuration You need to create HMAC keys and de
...PLEKEY⟩' ); Querying After setting up the GCS credentials, you can query the GCS data using: SELECT * FROM read_parquet('gs://⟨gcs_bucket⟩/⟨file.parquet⟩'); Attaching to a Database You can attach to a database file in read-only mode: LOAD httpfs; ATT...ACH 'gs://⟨gcs_bucket⟩/⟨file.duckdb⟩' AS ⟨duckdb_database⟩ (READ_ONLY); Databases in Google Cloud Storage can only be attached in read... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/gcs_import.html
- Secrets Manager 8727
- The Secrets manager provides a unified user interface for secrets across all backends that use them. Secrets can be scoped, so different storage prefixes can have different secrets, allowing for example to join data across organizations in
...should be used. For example: CREATE SECRET secret1 ( TYPE s3, KEY_ID 'my_secret_key1', SECRET 'my_secret_value1', SCOPE 's3://⟨my-bucket⟩' ); CREATE SECRET secret2 ( TYPE s3, KEY_ID 'my_secret_key2', SECRET 'my_secret_value2', SCOPE 's3://⟨my-other-b...ucket⟩' ); Now, if the user queries something from s3://⟨my-other-bucket⟩/something , secret secret2 will be chosen automatically for that request. To see which secret is being use...d, the which_secret scalar function can be used, which takes a path and a secret type as parameters: FROM which_secret('s3://⟨my-other-bucket⟩/file.parquet', 's3'); Listing Secrets Secrets can be listed using the built-in table-producing function, e... - https://man.mtool.work/duckdb/2.0/configuration/secrets_manager.html
- Fastly Object Storage Import 8473
- Prerequisites For Fastly Object Storage, the S3 Compatibility API allows you to use DuckDB's S3 support to read and write from Fastly buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This o
...entials, you can query the data there using DuckDB's built-in methods, such as read_csv or read_parquet : SELECT * FROM 's3://⟨fastly-bucket-name⟩/(file).csv'; SELECT * FROM read_parquet('s3://⟨fastly-bucket-name⟩/⟨file⟩.parquet');... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/fastly_object_sto...
- Tigris Import 8389
- Prerequisites For Tigris , the S3-compatible API allows you to use DuckDB's S3 support to read and write from Tigris buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This only needs to be r
...s credentials, you can query the data using DuckDB's built-in methods, such as read_csv or read_parquet : SELECT * FROM 's3://⟨tigris-bucket-name⟩/⟨file⟩.csv'; SELECT * FROM read_parquet('s3://⟨tigris-bucket-name⟩/⟨file⟩.parquet');... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/tigris_import.htm...
- S3 Parquet Export 8304
- To write a Parquet file to S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL command: LOAD htt
...dentials are correctly configured, Parquet files can be written to S3 using the following command: COPY ⟨table_name⟩ TO 's3://⟨s3-bucket⟩/⟨filename⟩.parquet'; Similarly, Google Cloud Storage (GCS) is supported through the Interoperability API. You ne...tnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩' ); After setting up the GCS credentials, you can export using: COPY ⟨table_name⟩ TO 'gs://⟨gcs_bucket⟩/⟨filename⟩.parquet';... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_export.html
DuckDB 2.0 Documentation / man.mtool.work