Results of 1 - 10 of about 12 for gcs (0.002 sec.)
- S3 API Support 12912
- The httpfs extension supports reading/writing/ globbing files on object storage servers using the S3 API. S3 offers a standard API to read and write to remote files (while regular http servers, predating S3, do not offer a common write API)
...the config and credential_chain providers: Name Description Secret Type Default ENDPOINT Specify a custom S3 endpoint S3 , GCS , R2 STRING s3.amazonaws.com for S3 , KEY_ID The ID of the key to use S3 , GCS , R2 STRING - REGION The region for which t...o authenticate (should match the region of the bucket to query) S3 , GCS , R2 STRING us-east-1 SECRET The secret of the key to use S3 , GCS , R2 STRING - SESSION_TOKEN Optionally, a session token...can be passed to use temporary credentials S3 , GCS , R2 STRING - URL_COMPATIBILITY_MODE Can help when URLs contain problematic characters S3 , GCS , R2 BOOLEAN false URL_STY... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/s3api.html
- Google Cloud Storage Import 12063
- Prerequisites The Google Cloud Storage (GCS) can be used via the httpfs extension . This can be installed with the INSTALL httpfs SQL command. This only needs to be run once. Credentials and Configuration You need to create HMAC keys and de
...clare them: CREATE SECRET ( TYPE gcs, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩' ); Querying After setting up the GCS...credentials, you can query the GCS data using: SELECT * FROM read_parquet('gs://⟨gcs_bucket⟩/⟨file.parquet⟩'); Attaching to a Database You can attach to a da...tabase file in read-only mode: LOAD httpfs; ATTACH 'gs://⟨gcs_bucket⟩/⟨file.duckdb⟩' AS ⟨duckdb_database⟩ (READ_ONLY); Databases in Google Cloud Storage can only be attached in read-on... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/gcs_import.html
- S3 Parquet Export 9834
- To write a Parquet file to S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL command: LOAD htt
...using the following command: COPY ⟨table_name⟩ TO 's3://⟨s3-bucket⟩/⟨filename⟩.parquet'; Similarly, Google Cloud Storage (GCS) is supported through the Interoperability API. You need to create HMAC keys and provide the credentials as follows: CREAT...E SECRET ( TYPE gcs, KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩', SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩' ); After setting up the GCS credenti...als, you can export using: COPY ⟨table_name⟩ TO 'gs://⟨gcs_bucket⟩/⟨filename⟩.parquet';... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_export.html
- Using fsspec Filesystems 8702
- DuckDB support for fsspec filesystems allows querying data in filesystems that DuckDB's httpfs extension does not support. fsspec has a large number of inbuilt filesystems , and there are also many external implementations . This capability
...e API). Firstly, you must install duckdb and fsspec , and a filesystem interface of your choice. pip install duckdb fsspec gcsfs Then, you can register whichever filesystem you'd like to query: import duckdb from fsspec import filesystem # this line...will throw an exception if the appropriate filesystem interface is not installed duckdb.register_filesystem(filesystem('gcs')) duckdb.sql("SELECT * FROM read_csv('gcs:///bucket/file.csv')") These filesystems are not implemented in C++, hence, the... - https://man.mtool.work/duckdb/2.0/guides/python/filesystems.html
- S3 Parquet Import 8649
- Prerequisites To load a Parquet file from S3, the httpfs extension is required. This can be installed using the INSTALL SQL command. This only needs to be run once. INSTALL httpfs; To load the httpfs extension for usage, use the LOAD SQL co
...can be read from S3 using the following command: SELECT * FROM read_parquet('s3://⟨bucket⟩/⟨file⟩'); Google Cloud Storage (GCS) and Cloudflare R2 DuckDB can also handle Google Cloud Storage (GCS) and Cloudflare R2 via the S3 API. See the relevant gu... - https://man.mtool.work/duckdb/2.0/guides/network_cloud_storage/s3_import.html
- Secrets Manager 8012
- The Secrets manager provides a unified user interface for secrets across all backends that use them. Secrets can be scoped, so different storage prefixes can have different secrets, allowing for example to join data across organizations in
...et types are available: Secret type Service / protocol Extension azure Azure Blob Storage azure ducklake DuckLake ducklake gcs Google Cloud Storage httpfs http HTTP and HTTPS httpfs huggingface Hugging Face httpfs iceberg Iceberg REST Catalog httpfs...r needs to be used. A Secret Provider is a mechanism through which a secret is generated. To illustrate this, for the S3 , GCS , R2 , and AZURE secret types, DuckDB currently supports two providers: CONFIG and credential_chain . The CONFIG provider... - https://man.mtool.work/duckdb/2.0/configuration/secrets_manager.html
- Delta Extension 7959
- The delta extension adds support for the Delta Lake open-source storage format . It is built using the Delta Kernel . The extension reads Delta tables, local and remote, and writes them: it can create tables and append to them. For implemen
...⟩' ); SELECT * FROM delta_scan('az://⟨my_container⟩/⟨path⟩'); Reading from Google Cloud Storage To scan a Delta table in a GCS bucket through its S3 API compatibility , use HMAC keys and create a secret: CREATE SECRET ( TYPE gcs, KEY_ID '⟨hmac_key_i...INTO ), with file statistics for data skipping idempotent appends checkpoints ( CHECKPOINT ) cloud storage (AWS S3, Azure, GCS) with secrets Limitations CREATE OR REPLACE TABLE and CREATE TABLE ... AS SELECT are not supported. Create the table, then... - https://man.mtool.work/duckdb/2.0/core_extensions/delta.html
- Guides 7694
- The guides section contains compact how-to guides that are focused on achieving a single goal. For API references and examples, see the rest of the documentation. Note that there are many tools using DuckDB, which are not covered in the off
...How to export a Parquet file to S3 How to load a Parquet file from S3 Express One How to load a Parquet file directly from GCS How to load a Parquet file directly from Cloudflare R2 How to load a Parquet file directly from Fastly Object Storage How... - https://man.mtool.work/duckdb/2.0/guides/overview.html
- Connect 7588
- Overview Every DuckDB feature reachable from Java starts with a Connection . This page covers the JDBC URL forms the driver accepts, how DuckDB and driver options are passed and which of them win, how connections come to share (or not share
...mple jdbc:duckdb:ducklake:metadata.ducklake;jdbc_pin_db=false . Object Storage Credentials When the data files live on S3, GCS, or another remote file system, the credentials are provided through a secret , created after the connection is opened: tr...any local files, a setup often called a “frozen” lake. This works for a DuckDB or SQLite catalog file served over s3:// , gcs:// , or https:// . The catalog file is given as the ATTACH target with TYPE ducklake , and READ_ONLY is required because t... - https://man.mtool.work/duckdb/2.0/clients/java/connecting.html
- Configuration 7588
- DuckDB has a number of configuration options that can be used to change the behavior of the system. The configuration options can be set using either the SET statement or the PRAGMA statement . They can be reset to their original values usi
...oes nothing. BOOLEAN false GLOBAL extension_directory Deprecated setting. Use extension_directories instead VARCHAR GLOBAL gcs_user_project GCS billing project for Requester Pays VARCHAR NULL GLOBAL lambda_syntax Deprecated setting. Configures the u... - https://man.mtool.work/duckdb/2.0/configuration/overview.html
DuckDB 2.0 Documentation / man.mtool.work