Results of 1 - 10 of about 38 for '⟨ (0.002 sec.)
- Catalogs 13852
- The iceberg extension supports attaching Iceberg REST Catalogs. Attaching a catalog is required for writing to Iceberg . Before attaching an Iceberg REST Catalog, you must install the iceberg extension by following the instructions located
...secret workflow to store login credentials for the OAuth2 service. CREATE SECRET iceberg_secret ( TYPE iceberg, CLIENT_ID '⟨admin⟩', CLIENT_SECRET '⟨password⟩', OAUTH2_SERVER_URI '⟨http://iceberg_rest_catalog_url.com/v1/oauth/tokens⟩' ); If you al...Bearer token, you can pass it directly to your CREATE SECRET statement CREATE SECRET iceberg_secret ( TYPE iceberg, TOKEN '⟨bearer_token⟩' ); You can attach the Iceberg catalog with the following ATTACH statement. LOAD httpfs; ATTACH '⟨warehouse⟩'...AS iceberg_catalog ( TYPE iceberg, SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity URI '⟨https://rest_endpoint.com⟩' ); To see the available tables run SHOW ALL TABLES; A REST Catalog with OAuth2 authorization ca... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/catalogs.html
- HTTP(S) Support 11604
- With the httpfs extension, it is possible to directly query files over the HTTP(S) protocol. This works for all files supported by DuckDB or its various extensions, and provides read-only access. SELECT * FROM 'https://domain.tld/file.exten
...r an HTTP(S) endpoint, create an HTTP secret using the Secrets Manager : CREATE SECRET http_auth ( TYPE http, BEARER_TOKEN '⟨token⟩' ); Or: CREATE SECRET http_auth ( TYPE http, EXTRA_HTTP_HEADERS MAP { 'Authorization': 'Bearer ⟨token⟩' } ); HTTP Pr...ports HTTP proxies. You can add an HTTP proxy using the Secrets Manager : CREATE SECRET http_proxy ( TYPE http, HTTP_PROXY '⟨http_proxy_url⟩', HTTP_PROXY_USERNAME '⟨username⟩', HTTP_PROXY_PASSWORD '⟨password⟩' ); You can also set the scope for an H...TTP proxy using the SCOPE keyword. CREATE SECRET http_proxy ( TYPE HTTP, SCOPE ['⟨https://duckdb.org⟩', '⟨https://some-other-website.org⟩'], HTTP_PROXY '⟨http_proxy_url⟩', HTTP_PROXY_USERNAME '⟨username⟩',... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/https.html
- Azure Extension 10126
- The azure extension lets DuckDB read and write files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2. Installing and Loading The azure extension will be transparently autoloaded on first use from the official extension reposit
...orage account using a connection string or anonymously. For example: CREATE SECRET secret1 ( TYPE azure, CONNECTION_STRING '⟨connection_string⟩' ); For anonymous access, give the storage account name: CREATE SECRET secret2 ( TYPE azure, PROVIDER co...nfig, ACCOUNT_NAME '⟨storage_account_name⟩' ); credential_chain Provider The credential_chain provider allows connecting using credentials autom..., not when the secret is created. For example: CREATE SECRET secret3 ( TYPE azure, PROVIDER credential_chain, ACCOUNT_NAME '⟨storage_account_name⟩' ); DuckDB also allows specifying a specific chain using the CHAIN keyword. This takes a semicolon-se... - https://man.mtool.work/duckdb/2.0/core_extensions/azure.html
- Amazon RDS with IAM Authentication 9795
- Managed PostgreSQL databases on Amazon RDS and Aurora can authenticate with short-lived IAM tokens instead of a static password. DuckDB supports this through the postgres and aws extensions: the aws extension generates the token from an AWS
...l chain to mint the IAM token: CREATE SECRET aws_rds_secret ( TYPE rds, PROVIDER credential_chain, CHAIN 'env;sso', REGION '⟨eu-west-1⟩', RDS_USER '⟨postgres⟩', RDS_HOST '⟨instance⟩.⟨identifier⟩.⟨region⟩.rds.amazonaws.com', RDS_PORT '5432' ); Then...token generated by the rds secret is used (and refreshed) automatically: CREATE SECRET pg_rds_secret ( TYPE postgres, HOST '⟨instance⟩.⟨identifier⟩.⟨region⟩.rds.amazonaws.com', PORT '5432', USER '⟨postgres⟩', DATABASE '⟨postgres⟩', SSLMODE require,... - https://man.mtool.work/duckdb/2.0/guides/database_integration/rds_iam.html
- AWS Extension 9638
- The aws extension adds functionality, e.g., authentication, on top of the httpfs extension's S3 capabilities , using the AWS SDK. Installing and Loading The aws extension will be transparently autoloaded on first use from the official exten
...then override the region, run: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN config, REGION '⟨eu-west-1⟩' ); Selecting a Profile To load credentials based on a named profile which is not the default (from the AWS_PROF...e), use the PROFILE parameter: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN config, PROFILE '⟨my_profile⟩' ); This approach is equivalent to the deprecated S3 API 's method load_aws_credentials('⟨my_profile⟩') . Assum...et ( TYPE s3, PROVIDER credential_chain, CHAIN 'sts', ASSUME_ROLE_ARN 'arn:aws:iam::⟨account_id⟩:role/⟨role⟩', EXTERNAL_ID '⟨external_id⟩', REGION '⟨us-east-1⟩' ); The sts chain value requires an ASSUME_ROLE_ARN value. If the selected profile itsel... - https://man.mtool.work/duckdb/2.0/core_extensions/aws.html
- S3 Parquet Export 9261
- To write a Parquet file to S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL command: LOAD htt
.... Note that the region parameter should match the region of the bucket you want to access. CREATE SECRET ( TYPE s3, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', REGION '⟨us-east-1⟩' ); Tip If you get an IO E...nteroperability API. You need to create HMAC keys and provide the credentials as follows: CREATE SECRET ( TYPE gcs, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩' ); After setting up the GCS credentials, you ca... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_export.html
- Lance Extension 8931
- The lance extension adds support for reading and writing Lance tables. Lance is a modern lakehouse format optimized for ML/AI workloads, with native cloud storage support. Installing and Loading You can install the lance extension from Duck
...: INSTALL AND LOAD lance; Usage Full SQL reference Cloud storage reference Query a Lance Dataset Local file: SELECT * FROM '⟨path/to/dataset.lance⟩' LIMIT 10; S3: SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10; To access object store URIs...set from a query COPY ( SELECT 1::BIGINT AS id, 'a'::VARCHAR AS s UNION ALL SELECT 2::BIGINT AS id, 'b'::VARCHAR AS s ) TO '⟨path/to/dataset.lance⟩' ( FORMAT lance, MODE 'overwrite' ); -- Read it back via the replacement scan SELECT count(*) FROM '...⟨path/to/dataset.lance⟩'; -- Append more rows to an existing dataset COPY ( SELECT 3::BIGINT AS id, 'c'::VARCHAR AS s ) TO '⟨path/to/dataset.lance⟩' ( FORMAT lance, MODE 'append' ); -- Optionally create an empty dataset (schema only) COPY ( SELECT... - https://man.mtool.work/duckdb/2.0/core_extensions/lance.html
- Fastly Object Storage Import 8821
- Prerequisites For Fastly Object Storage, the S3 Compatibility API allows you to use DuckDB's S3 support to read and write from Fastly buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This o
...on You will need to generate an S3 auth token and create an S3 secret in DuckDB: CREATE SECRET my_secret ( TYPE s3, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', URL_STYLE 'path', REGION '⟨us-east⟩', ENDPOINT...'⟨us-east⟩.object.fastlystorage.app' -- see note below ); The ENDPOINT needs to point to the Fastly endpoint for the region y... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/fastly_object_sto...
- S3 API Support 8538
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...the S3 bucket by manually providing a key. For example: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER config, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', REGION '⟨us-east-1⟩' ); Tip If you get an IO E...S3 using the KMS_KEY_ID option: CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, CHAIN config, REGION '⟨eu-west-1⟩', KMS_KEY_ID 'arn:aws:kms:⟨region⟩:⟨account_id⟩:⟨key⟩/⟨key_id⟩', SCOPE 's3://⟨bucket-sub-path⟩' ); R2 Secrets Wh...DB has a special Secret type, R2 , to make configuring it a bit simpler: CREATE OR REPLACE SECRET secret ( TYPE r2, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', ACCOUNT_ID '⟨my_account_id⟩' ); Note the addit... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Cloudflare R2 Import 8380
- Prerequisites For Cloudflare R2, the S3 Compatibility API allows you to use DuckDB's S3 support to read and write from R2 buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This only needs to
...onfiguration You will need to generate an S3 auth token and create an R2 secret in DuckDB: CREATE SECRET ( TYPE r2, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', ACCOUNT_ID '⟨your-32-character-hexadecimal-acc... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/cloudflare_r2_imp...
DuckDB 2.0 Documentation / man.mtool.work