Results of 1 - 10 of about 28 for endpoint (0.002 sec.)
- Catalogs 13174
- The iceberg extension supports attaching Iceberg REST Catalogs. Attaching a catalog is required for writing to Iceberg . Before attaching an Iceberg REST Catalog, you must install the iceberg extension by following the instructions located
...berg_catalog ( TYPE iceberg, SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity URI '⟨https://rest_endpoint.com⟩' ); To see the available tables run SHOW ALL TABLES; A REST Catalog with OAuth2 authorization can also be attached wi...he aws extension . Then, connect to the catalog using your S3 Tables ARN (available in the AWS Management Console) and the ENDPOINT_TYPE s3_tables option: ATTACH '⟨s3_tables_arn⟩' AS my_s3_tables_catalog ( TYPE iceberg, ENDPOINT_TYPE s3_tables ); Warning...ENDPOINT_TYPE s3_tables always builds an endpoint of the form s3tables.⟨region⟩.amazonaws.com/iceberg . This is incorrect for any r... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/catalogs.html
- Legacy Authentication Scheme for S3 API 11552
- Prior to version 0.10.0, DuckDB did not have a Secrets manager . Hence, the configuration of and authentication to S3 endpoints was handled via variables. This page documents the legacy authentication scheme for the S3 API. Warning This pag
...values). Therefore, avoid using these methods for storing secrets. The recommended way to configure and authenticate of S3 endpoints is to use secrets . Legacy Authentication Scheme To be able to read or write from S3, the correct region should be set: S...ET s3_region = 'us-east-1'; Optionally, the endpoint can be configured in case a non-AWS object storage server is used: SET s3_endpoint = '⟨domain⟩.⟨tld⟩:⟨port⟩'; If the endpo...e and vhost-style URLs is possible using: SET s3_url_style = 'path'; However, note that this may also require updating the endpoint. For example for AWS S3 it is required to change the endpoint to s3.⟨region⟩.amazonaws.com . After configuring the correct... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api_legacy_authentica...
- Iceberg Options 11159
- This page lists the options of the iceberg extension: the parameters accepted by the Iceberg functions , the options of the ATTACH and CREATE SECRET statements used to connect to a catalog , and the global settings. ATTACH Options To make a
...ded to the ATTACH are divided into categories: Parameter Type Default Description URI VARCHAR NULL URL of the REST Catalog endpoint. In DuckDB 2.0, URI is the canonical option name; ENDPOINT is accepted as a deprecated alias (do not supply both). DEFAULT...o not support staged table creation. DISABLE_MULTI_TABLE_COMMIT BOOLEAN false Disables the multi-table transactions/commit endpoint. Enable for catalogs that reject this endpoint. SKIP_CREATE_TABLE_METADATA_UPDATES BOOLEAN false Skips follow-up metadata...rs, see the list of associated additional parameters below this table. Parameter Type Default Accepted options Description ENDPOINT_TYPE VARCHAR NULL S3_TABLES , GLUE A common identifier of an Iceberg Catalog type, sets certain default parameters. Cannot... - https://man.mtool.work/duckdb/2.0/core_extensions/iceberg/iceberg_options.html
- Fastly Object Storage Import 10389
- Prerequisites For Fastly Object Storage, the S3 Compatibility API allows you to use DuckDB's S3 support to read and write from Fastly buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This o
...EY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩', URL_STYLE 'path', REGION '⟨us-east⟩', ENDPOINT '⟨us-east⟩.object.fastlystorage.app' -- see note below ); The ENDPOINT needs to point to the Fastly endpoint for the regio...n you want to use (e.g., eu-central.object.fastlystorage.app ). REGION must use the same region mentioned in ENDPOINT . URL_STYLE needs to use path . Querying After setting up the Fastly Object Storage credentials, you can query the data th... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/fastly_object_sto...
- Azure Extension 9700
- The azure extension lets DuckDB read and write files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2. Installing and Loading The azure extension will be transparently autoloaded on first use from the official extension reposit
...ob.core.windows.net/' ). With fully qualified paths, ACCOUNT_NAME is optional. For paths that are not fully qualified, the ENDPOINT attribute overrides the default endpoint ( blob.core.windows.net for az:// and azure:// , dfs.core.windows.net for abfss:/...e extension will attempt to automatically detect credentials (not used if you pass the connection string). VARCHAR - azure_endpoint Override the Azure endpoint for when the Azure credential providers are used. VARCHAR blob.core.windows.net azure_credenti...le⟩.parquet'; Difference between ADLS and Blob Storage For glob patterns over many directories, prefer abfss:// : the ADLS endpoint lists only the directories that match, while the Blob endpoint lists every file under the prefix and filters afterwards. T... - https://man.mtool.work/duckdb/2.0/core_extensions/azure.html
- S3 API Support 9160
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...uidance on which one to use, see the S3 and AWS Authentication page. The preferred way to configure and authenticate to S3 endpoints is to use secrets . Multiple secret providers are available. To migrate from the deprecated S3 API , use a defined secret...bPxRfiCYEXAMPLEKEY⟩', REGION '⟨us-east-1⟩' ); Tip If you get an IO Error ( Connection error for HTTP HEAD ), configure the endpoint explicitly via ENDPOINT 's3.⟨your-region⟩.amazonaws.com' . Now, to query using the above secret, simply query any s3:// pr...orted parameters that can be used for both the config and credential_chain providers: Name Description Secret Type Default ENDPOINT Specify a custom S3 endpoint S3 , GCS , R2 STRING s3.amazonaws.com for S3 , KEY_ID The ID of the key to use S3 , GCS , R2... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- SeaweedFS Import 8619
- Prerequisites For SeaweedFS , the S3-compatible gateway allows you to use DuckDB's S3 support to read and write from SeaweedFS buckets. This requires the httpfs extension , which can be installed using the INSTALL SQL command. This only nee
...fig.json echo "s3.bucket.create -name ⟨your-bucket⟩" | \ docker exec -i seaweedfs weed shell -master=localhost:9333 The S3 endpoint listens on port 8333. Credentials and Configuration Create an S3 secret with the credentials from the SeaweedFS S3 configu...ration: CREATE SECRET my_secret ( TYPE s3, KEY_ID '⟨your_access_key_id⟩', SECRET '⟨your_secret_access_key⟩', ENDPOINT '⟨seaweedfs-host⟩:8333', URL_STYLE 'path', USE_SSL false ); SeaweedFS serves path-style requests, so set URL_STYLE to path...wildcard DNS is needed. SeaweedFS does not require a region, so REGION can be omitted. Set USE_SSL false for a plain-HTTP endpoint such as the local setup above; omit it when the gateway runs behind TLS. Querying After setting up the SeaweedFS credentia... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/seaweedfs_import....
- Sharing Macros 8619
- DuckDB has a powerful macro mechanism that allows creating shorthands for common tasks. Sharing a Scalar Macro First, we define a macro that pretty-prints a non-negative integer as a short string with thousands, millions, and billions (with
...──────┘ As one would expect, the macro gets persisted in the database. But this also means that we can host it on an HTTPS endpoint and share it with anyone! We have published this macro on blobs.duckdb.org . You can try it from DuckDB: duckdb Make sure...that the httpfs extension is installed: INSTALL httpfs; You can now attach to the remote endpoint and use the macro: ATTACH 'https://blobs.duckdb.org/data/pretty_print_integer_macro.duckdb' AS pretty_print_macro_db; SELE...ble(table_name); To use it, make sure that the httpfs extension is installed: INSTALL httpfs; You can attach to the remote endpoint and use the macro: ATTACH 'https://blobs.duckdb.org/data/compute_table_checksum.duckdb' AS compute_table_checksum_db; CREA... - https://man.mtool.work/duckdb/2.0/guides/snippets/sharing_macros.html
- Unity Catalog Extension 8455
- The unity_catalog extension attaches a Unity Catalog catalog, on Databricks or Unity Catalog OSS, as a DuckDB database. You can list, read, insert into and create its Delta Lake tables. It uses the delta extension to access the tables. For
...LOAD unity_catalog; Usage To connect to Unity Catalog on Databricks or Unity Catalog OSS, create a secret with your token, endpoint and region, then attach a catalog: CREATE SECRET ( TYPE unity_catalog, TOKEN '⟨token⟩', ENDPOINT '⟨endpoint⟩', AWS_REGION...'⟨region⟩' ); ATTACH '⟨catalog_name⟩' AS my_catalog (TYPE unity_catalog, DEFAULT_SCHEMA 'main'); ENDPOINT is the Unity Catalog REST API endpoint. For Databricks, it is your Workspace URL , typically https://⟨instance⟩.cloud.data... - https://man.mtool.work/duckdb/2.0/core_extensions/unity_catalog.html
- S3 Parquet Import 8307
- Prerequisites To load a Parquet file from S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL co
...bPxRfiCYEXAMPLEKEY⟩', REGION '⟨us-east-1⟩' ); Tip If you get an IO Error ( Connection error for HTTP HEAD ), configure the endpoint explicitly via ENDPOINT 's3.⟨your-region⟩.amazonaws.com' . To fetch credentials automatically instead — or to use a named... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_import.html
DuckDB 2.0 Documentation / man.mtool.work