Results of 1 - 10 of about 29 for ⟩' (0.002 sec.)
- Catalogs 14733
- The iceberg extension supports attaching Iceberg REST Catalogs. Attaching a catalog is required for writing to Iceberg . Before attaching an Iceberg REST Catalog, you must install the iceberg extension by following the instructions located
...workflow to store login credentials for the OAuth2 service. CREATE SECRET iceberg_secret ( TYPE iceberg, CLIENT_ID '⟨admin⟩', CLIENT_SECRET '⟨password⟩', OAUTH2_SERVER_URI '⟨http://iceberg_rest_catalog_url.com/v1/oauth/tokens⟩' ); If you already h...you can pass it directly to your CREATE SECRET statement CREATE SECRET iceberg_secret ( TYPE iceberg, TOKEN '⟨bearer_token⟩' ); You can attach the Iceberg catalog with the following ATTACH statement. LOAD httpfs; ATTACH '⟨warehouse⟩' AS iceberg_ca...( TYPE iceberg, SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity URI '⟨https://rest_endpoint.com⟩' ); To see the available tables run SHOW ALL TABLES; A REST Catalog with OAuth2 authorization can also be attached with jus... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/catalogs.html
- Lance Extension 10817
- The lance extension adds support for reading and writing Lance tables. Lance is a modern lakehouse format optimized for ML/AI workloads, with native cloud storage support. Installing and Loading You can install the lance extension from Duck
...e; Usage Full SQL reference Cloud storage reference Query a Lance Dataset Local file: SELECT * FROM '⟨path/to/dataset.lance⟩' LIMIT 10; S3: SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10; To access object store URIs (e.g., s3://... ), con...REATE SECRET ( TYPE lance, PROVIDER credential_chain, SCOPE 's3://bucket/' ); SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10; Write a Lance Dataset Use the COPY ... TO ... statement to materialize query results as a Lance dataset. -- Crea...SELECT 1::BIGINT AS id, 'a'::VARCHAR AS s UNION ALL SELECT 2::BIGINT AS id, 'b'::VARCHAR AS s ) TO '⟨path/to/dataset.lance⟩' ( FORMAT lance, MODE 'overwrite' ); -- Read it back via the replacement scan SELECT count(*) FROM '⟨path/to/dataset.lance⟩... - https://man.mtool.work/duckdb/2.0/core_extensions/lance.html
- Delta Extension 10654
- The delta extension adds support for the Delta Lake open-source storage format . It is built using the Delta Kernel . The extension reads Delta tables, local and remote, and writes them: it can create tables and append to them. For implemen
...d it manually, run: INSTALL AND LOAD delta; Usage To scan a local Delta table, run: SELECT * FROM delta_scan('file:///⟨path⟩'); Attaching Tables A Delta table can also be attached as a database. The database holds exactly one table, with the same n...ame as the database, so my_table and my_table.my_table refer to the same table: ATTACH 'file:///⟨path⟩' AS my_table (TYPE delta); SELECT * FROM my_table; Reading from an S3 Bucket To scan a Delta table in an S3 bucket , run: S...ELECT * FROM delta_scan('s3://⟨my_bucket⟩/⟨path⟩'); To authenticate, create a secret : CREATE SECRET ( TYPE s3, PROVIDER credential_chain ); SELECT * FROM delta_scan('s3://... - https://man.mtool.work/duckdb/2.0/core_extensions/delta.html
- AWS Extension 9609
- The aws extension adds functionality, e.g., authentication, on top of the httpfs extension's S3 capabilities , using the AWS SDK. Installing and Loading The aws extension will be transparently autoloaded on first use from the official exten
...ide the region, run: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN config, REGION '⟨eu-west-1⟩' ); Selecting a Profile To load credentials based on a named profile which is not the default (from the AWS_PROFILE environ...PROFILE parameter: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN config, PROFILE '⟨my_profile⟩' ); This approach is equivalent to the deprecated S3 API 's method load_aws_credentials('⟨my_profile⟩') . Assuming a Role (...LACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN 'sts', ASSUME_ROLE_ARN 'arn:aws:iam::⟨account_id⟩:role/⟨role⟩', EXTERNAL_ID '⟨external_id⟩', REGION '⟨us-east-1⟩' ); The sts chain value requires an ASSUME_ROLE_ARN value. If the select... - https://man.mtool.work/duckdb/2.0/core_extensions/aws.html
- Azure Extension 9299
- The azure extension lets DuckDB read and write files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2. Installing and Loading The azure extension will be transparently autoloaded on first use from the official extension reposit
...re Blob Storage Allowed URI schemes: az or azure SELECT count(*) FROM 'az://⟨my_container⟩/⟨path⟩/⟨my_file⟩.⟨parquet_or_csv⟩'; Globs are also supported: SELECT * FROM 'az://⟨my_container⟩/⟨path⟩/*.csv'; SELECT * FROM 'az://⟨my_container⟩/⟨path⟩/**'...ntax: SELECT count(*) FROM 'az://⟨my_storage_account⟩.blob.core.windows.net/⟨my_container⟩/⟨path⟩/⟨my_file⟩.⟨parquet_or_csv⟩'; SELECT * FROM 'az://⟨my_storage_account⟩.blob.core.windows.net/⟨my_container⟩/⟨path⟩/*.csv'; Azure Data Lake Storage (ADL...mes: abfss or abfs (both connect over HTTPS) SELECT count(*) FROM 'abfss://⟨my_filesystem⟩/⟨path⟩/⟨my_file⟩.⟨parquet_or_csv⟩'; Globs are also supported: SELECT * FROM 'abfss://⟨my_filesystem⟩/⟨path⟩/*.csv'; SELECT * FROM 'abfss://⟨my_filesystem⟩/⟨p... - https://man.mtool.work/duckdb/2.0/core_extensions/azure.html
- HTTP(S) Support 9136
- With the httpfs extension, it is possible to directly query files over the HTTP(S) protocol. This works for all files supported by DuckDB or its various extensions, and provides read-only access. SELECT * FROM 'https://domain.tld/file.exten
...TP(S) endpoint, create an HTTP secret using the Secrets Manager : CREATE SECRET http_auth ( TYPE http, BEARER_TOKEN '⟨token⟩' ); Or: CREATE SECRET http_auth ( TYPE http, EXTRA_HTTP_HEADERS MAP { 'Authorization': 'Bearer ⟨token⟩' } ); HTTP Proxy Duc...es. You can add an HTTP proxy using the Secrets Manager : CREATE SECRET http_proxy ( TYPE http, HTTP_PROXY '⟨http_proxy_url⟩', HTTP_PROXY_USERNAME '⟨username⟩', HTTP_PROXY_PASSWORD '⟨password⟩' ); You can also set the scope for an HTTP proxy using...the SCOPE keyword. CREATE SECRET http_proxy ( TYPE HTTP, SCOPE ['⟨https://duckdb.org⟩', '⟨https://some-other-website.org⟩'], HTTP_PROXY '⟨http_proxy_url⟩', HTTP_PROXY_USERNAME '⟨username⟩', HTTP_PROXY_PASSWORD... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/https.html
- Google Cloud Storage Import 8810
- Prerequisites The Google Cloud Storage (GCS) can be used via the httpfs extension . This can be installed with the INSTALL httpfs SQL command. This only needs to be run once. Credentials and Configuration You need to create HMAC keys and de
...clare them: CREATE SECRET ( TYPE gcs, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩' ); Querying After setting up the GCS credentials, you can query the G...CS data using: SELECT * FROM read_parquet('gs://⟨gcs_bucket⟩/⟨file.parquet⟩'); Attaching to a Database You can attach to a database file in read-only mode: LOAD httpfs; ATTACH 'gs://⟨gcs_bucket⟩/⟨fil...e.duckdb⟩' AS ⟨duckdb_database⟩ (READ_ONLY); Databases in Google Cloud Storage can only be attached in read-only mode.... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/gcs_import.html
- S3 Parquet Export 8337
- To write a Parquet file to S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL command: LOAD htt
...parameter should match the region of the bucket you want to access. CREATE SECRET ( TYPE s3, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', REGION '⟨us-east-1⟩' ); Tip If you get an IO Error ( Connection erro...ou need to create HMAC keys and provide the credentials as follows: CREATE SECRET ( TYPE gcs, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩' ); After setting up the GCS credentials, you can export using: COPY ⟨... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_export.html
- Iceberg Extension 8174
- The iceberg extension implements support for the Apache Iceberg open table format . There are two ways to work with Iceberg in DuckDB: Individual tables are read directly from storage, by pointing at a table's metadata. This requires no cat
...and attach the catalog with ATTACH ... (TYPE iceberg, ...) : CREATE SECRET iceberg_secret ( TYPE iceberg, CLIENT_ID '⟨admin⟩', CLIENT_SECRET '⟨password⟩', OAUTH2_SERVER_URI '⟨https://catalog.example.com/v1/oauth/tokens⟩' ); ATTACH '⟨warehouse⟩' AS...my_catalog ( TYPE iceberg, SECRET iceberg_secret, URI '⟨https://catalog.example.com⟩' ); Once attached, the catalog behaves like any other DuckDB database: reference its tables as ⟨catalog⟩.⟨schema⟩.⟨table⟩ a... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/overview.html
- S3 API Support 8092
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...lly providing a key. For example: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER config, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', REGION '⟨us-east-1⟩' ); Tip If you get an IO Error ( Connection erro...e KMS_KEY_ID option: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN config, REGION '⟨eu-west-1⟩', KMS_KEY_ID 'arn:aws:kms:⟨region⟩:⟨account_id⟩:⟨key⟩/⟨key_id⟩', SCOPE 's3://⟨bucket-sub-path⟩' ); R2 Secrets While Cloudfl...t type, R2 , to make configuring it a bit simpler: CREATE OR REPLACE SECRET secret ( TYPE r2, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', ACCOUNT_ID '⟨my_account_id⟩' ); Note the addition of the ACCOUNT_ID... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
DuckDB 2.0 Documentation / man.mtool.work