Results of 1 - 10 of about 19 for efficient (0.001 sec.)
- Variant Type 13660
- The VARIANT type stores typed, binary data where each row is self-contained with its own type information. This differs from the JSON type , which is physically stored as text. Because type metadata is embedded per-value, VARIANT provides b
...em, i.e., stores the fields that share a type as separate, typed columns. Shredded data compresses well and can be scanned efficiently. Shredding is only applied to row groups with at least variant_minimum_shredding_size rows (by default, 30,000 rows, rou...riting VARIANT types from Parquet files , including shredding, a technique that stores nested data as flat values for more efficient access. Writing VARIANT to Parquet When writing VARIANT columns to Parquet, DuckDB can automatically shred (decompose) the...on the structure of the first row group. This auto-shredding improves read performance by enabling predicate pushdown and efficient column access. To explicitly provide a schema for shredding, use the SHREDDING copy option: COPY events TO 'events.parquet... - https://man.mtool.work/duckdb/2.0/sql/data_types/variant.html
- Python Function API 9989
- You can create a DuckDB user-defined function (UDF) from a Python function so it can be used in SQL queries. Similarly to regular functions , they need to have a name, a return type and parameter types. Creating Functions To register a Pyth
...ve' is assumed, but you can specify type = 'arrow' to use PyArrow Arrays. In general, using an Arrow UDF will be much more efficient than native because it will be able to operate in batches. null_handling (optional): By default, NULL values are automatic...ect an array of the same amount of rows to be returned from the function. In general, using an Arrow UDF will be much more efficient than native because it will be able to operate in batches. import duckdb import pyarrow as pa from duckdb.sqltypes import... - https://man.mtool.work/duckdb/2.0/clients/python/function.html
- EXPLAIN: Inspect Query Plans 9989
- EXPLAIN SELECT * FROM tbl; The EXPLAIN statement displays the physical plan, i.e., the query plan that will get executed, and is enabled by prepending the query with EXPLAIN . The physical plan is a tree of operators that are executed in a
...specific order to produce the result of the query. To generate an efficient physical plan, the query optimizer transforms the existing physical plan into a better physical plan. To demonstrate, see... - https://man.mtool.work/duckdb/2.0/guides/meta/explain.html
- Integration with Polars 9989
- Polars is a DataFrames library built in Rust with bindings for Python and Node.js. It uses Apache Arrow's columnar format as its memory model. DuckDB can read Polars DataFrames and convert query results to Polars DataFrames. It does this in
...ternally using the efficient Apache Arrow integration. Note that the pyarrow library must be installed for the integration to work. Installation pip in... - https://man.mtool.work/duckdb/2.0/guides/python/polars.html
- JSON Overview 9459
- DuckDB supports SQL functions that are useful for reading values from existing JSON and creating new JSON data. JSON is supported with the json extension which is shipped with most DuckDB distributions and is auto-loaded on first use. If yo
...ransmit data objects consisting of attribute–value pairs and arrays (or other serializable values). While it is not a very efficient format for tabular data, it is very commonly used, especially as a data interchange format. JSONPath and JSON Pointer Synt... - https://man.mtool.work/duckdb/2.0/data/json/overview.html
- Reading and Writing Parquet Files 9459
- Examples Read a single Parquet file: SELECT * FROM 'test.parquet'; Figure out which columns/types are in a Parquet file: DESCRIBE SELECT * FROM 'test.parquet'; Create a table from a Parquet file: CREATE TABLE test AS SELECT * FROM 'test.par
...t: EXPORT DATABASE 'target_directory' (FORMAT parquet); Parquet Files Parquet files are compressed columnar files that are efficient to load and process. DuckDB provides support for both reading and writing Parquet files in an efficient manner, as well as... - https://man.mtool.work/duckdb/2.0/data/parquet/overview.html
- Hugging Face Support 8947
- The httpfs extension introduces support for the hf:// protocol to access datasets hosted in Hugging Face repositories. See the announcement blog post for details. Usage Hugging Face repositories can be queried using the following URL patter
...le: SELECT count(*) AS count FROM 'hf://datasets/cais/mmlu/astronomy/*.parquet'; count 173 By using glob patterns, you can efficiently handle large datasets and perform comprehensive queries across multiple files, simplifying your data inspections and pro...revision. This is a special branch where Hugging Face automatically generates the Parquet files of every dataset to enable efficient scanning. Authentication Configure your Hugging Face Token in the DuckDB Secrets Manager to access private or gated datase... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/hugging_face.html
- Guides 8947
- The guides section contains compact how-to guides that are focused on achieving a single goal. For API references and examples, see the rest of the documentation. Note that there are many tools using DuckDB, which are not covered in the off
...ide) How to design the schema for optimal performance What is the ideal hardware environment for DuckDB How to import data efficiently How to use indexes and zonemaps How to make join operations efficient What performance implications do Parquet files and... - https://man.mtool.work/duckdb/2.0/guides/overview.html
- Data Chunks 8418
- Data chunks represent a horizontal slice of a table. They hold a number of vectors , that can each hold up to the VECTOR_SIZE rows. The vector size can be obtained through the duckdb_vector_size function and is configurable, but is usually
...d vectors are what DuckDB uses natively to store and represent data. For this reason, the data chunk interface is the most efficient way of interfacing with DuckDB. Be aware, however, that correctly interfacing with DuckDB using the data chunk API does re... - https://man.mtool.work/duckdb/2.0/clients/c/data_chunk.html
- Appender 8418
- The Appender can be used to load bulk data into a DuckDB database. It is available in the C, C++, Go, Java, Julia, Node.js, Python, and Rust clients. This page describes how the Appender behaves across clients and shows a minimal example in
...TAMP)"); Appender appender(con, "dates"); // construct the values using the Date/Time/Timestamp types // (this is the most efficient approach) appender.AppendRow( Date::FromDate(1992, 1, 1), Time::FromTime(1, 1, 1, 0), Timestamp::FromDatetime(Date::FromDa... - https://man.mtool.work/duckdb/2.0/data/appender.html
DuckDB 2.0 Documentation / man.mtool.work