Results of 1 - 10 of about 36 for Dataset (0.001 sec.)
- SQL on Apache Arrow 13455
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
..." and return as an Arrow Table results = con.sql("SELECT * FROM my_arrow_table WHERE i = 2").to_arrow_table() Apache Arrow Datasets Arrow Datasets stored as variables can also be queried as if they were regular tables. Datasets are useful to point towar...ds directories of Parquet files to analyze large datasets. DuckDB will push column selections and the row filters it can translate down into the dataset scan operation so that onl...data is pulled into memory. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds # connect to an in-memory database con = duckdb.connect() my_arrow_table = pa.Table.from_pydict({'i': [1, 2, 3, 4],... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Lance Extension 13408
- The lance extension adds support for reading and writing Lance tables. Lance is a modern lakehouse format optimized for ML/AI workloads, with native cloud storage support. Installing and Loading You can install the lance extension from Duck
...ad it using the following commands: INSTALL AND LOAD lance; Usage Full SQL reference Cloud storage reference Query a Lance Dataset Local file: SELECT * FROM '⟨path/to/dataset.lance⟩' LIMIT 10; S3: SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10...PROVIDER credential_chain, SCOPE 's3://bucket/' ); SELECT * FROM 's3://⟨bucket/path/to/out.lance⟩' LIMIT 10; Write a Lance Dataset Use the COPY ... TO ... statement to materialize query results as a Lance dataset. -- Create/overwrite a Lance dataset fro...query COPY ( SELECT 1::BIGINT AS id, 'a'::VARCHAR AS s UNION ALL SELECT 2::BIGINT AS id, 'b'::VARCHAR AS s ) TO '⟨path/to/dataset.lance⟩' ( FORMAT lance, MODE 'overwrite' ); -- Read it back via the replacement scan SELECT count(*) FROM '⟨path/to/datase... - https://man.mtool.work/duckdb/2.0/core_extensions/lance.html
- Integration with PyTorch 10381
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
...sors. Installation pip install -U duckdb numpy pyarrow torch DuckDB to PyTorch This example queries a Parquet-backed Arrow dataset, streams the result as RecordBatch objects and converts each batch into feature and label tensors. import pathlib import t...empfile import duckdb import numpy as np import pyarrow as pa import pyarrow.dataset as ds import pyarrow.parquet as pq import torch base_path = pathlib.Path(tempfile.mkdtemp()) parquet_dir = base_path / "tr...= pa.table( { "feature_0": [0.1, 0.3, 0.5, 0.7], "feature_1": [1.0, 0.0, 1.0, 0.0], "label": [0, 1, 1, 0], } ) pq.write_to_dataset(table, str(parquet_dir)) con = duckdb.connect() train_dataset = ds.dataset(str(parquet_dir)) reader = con.sql(""" SELECT f... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
- UNPIVOT Statement 10032
- The UNPIVOT statement allows multiple columns to be stacked into fewer columns. In the basic case, multiple columns are stacked into two columns: a NAME column (which contains the name of the source column) and a VALUE column (which contain
...s below, but the simplified UNPIVOT syntax can be summarized using spreadsheet pivot table naming conventions as: UNPIVOT ⟨dataset⟩ ON ⟨column(s)⟩ INTO NAME ⟨name_column_name⟩ VALUE ⟨value_column_name(s)⟩ ORDER BY ⟨column(s)_with_order_direction(s)⟩ LIM...IT ⟨number_of_rows⟩; Example Data All examples use the dataset produced by the queries below: CREATE OR REPLACE TABLE monthly_sales (empid INTEGER, dept TEXT, Jan INTEGER, Feb INTEGER,...Expression In many cases, the number of columns to unpivot is not easy to predetermine ahead of time. In the case of this dataset, the query above would have to change each time a new month is added. The COLUMNS expression can be used to select all col... - https://man.mtool.work/duckdb/2.0/sql/statements/unpivot.html
- Hugging Face Support 9778
- The httpfs extension introduces support for the hf:// protocol to access datasets hosted in Hugging Face repositories. See the announcement blog post for details. Usage Hugging Face repositories can be queried using the following URL patter
...n: hf://datasets/⟨my_username⟩/⟨my_dataset⟩/⟨path_to_file⟩ For example, to read a CSV file, you can use the following query: SELECT * FROM...'hf://datasets/datasets-examples/doc-formats-csv-1/data.csv'; Where: datasets-examples is the name of the user/organization doc-formats-...csv-1 is the name of the dataset repository data.csv is the file path in the repository The result of the query is: kind sound dog woof cat meow pokemon pi... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/hugging_face.html
- Partitioned Writes 8970
- Examples Write a table to a Hive partitioned dataset of Parquet files: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)); Write a table to a Hive partitioned dataset of CSV files, allowing overwrites: COPY orders TO 'orde
...rs' (FORMAT csv, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE); Write a table to a Hive partitioned dataset of GZIP-compressed CSV files, setting explicit data files' extension: COPY orders TO 'orders' (FORMAT csv, PARTITION_BY (y...to write a larger number of partitions as many files will be created. The ideal partition count depends on how large your dataset is. To limit the maximum number of files the system can keep open before flushing to disk when writing using PARTITION_BY...es: {i} will be replaced by an index. {uuid} will be replaced by a 128 bits long UUID. Write a table to a Hive partitioned dataset of .parquet files, with an index in the filename: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, month), OVE... - https://man.mtool.work/duckdb/2.0/data/partitioning/partitioned_writes.html
- Hive Partitioning 8162
- Examples Read data from a Hive partitioned dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true); Write a table to a Hive partitioned dataset: COPY orders TO 'orders' (FORMAT parquet, PARTITION_BY (year, mont
...ystem skips reading files that are not necessary to answer a query. For example, consider the following query on the above dataset: SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true) WHERE year = 2022 AND month = 11; When execu... - https://man.mtool.work/duckdb/2.0/data/partitioning/hive_partitioning.html
- R Client 8003
- Installation To use the DuckDB R client, visit the R installation page . The latest stable version of the DuckDB R client is 2.0.0. Installation duckdb : R Client The DuckDB R client can be installed using the following command: install.pac
...ng the dplyr::tbl function. # Establish a CSV for the sake of this example write.csv(mtcars, "mtcars.csv") # Summarize the dataset in DuckDB to avoid reading the entire CSV into R's memory tbl(con, "mtcars.csv") |> group_by(cyl) |> summarise(across(disp...:wt, .fns = mean)) |> collect() # Establish a set of Parquet files dbExecute(con, "COPY flights TO 'dataset' (FORMAT parquet, PARTITION_BY (year, month))") # Summarize the dataset in DuckDB to avoid reading 12 Parquet files into R...'s memory tbl(con, "read_parquet('dataset/**/*.parquet', hive_partitioning = true)") |> filter(month == "3") |> summarise(delay = mean(dep_time, na.rm = TRUE)) |> c... - https://man.mtool.work/duckdb/2.0/clients/r.html
- GDAL Integration 7797
- The spatial extension integrates the GDAL translator library to read and write spatial data from a variety of geospatial vector file formats. See the documentation for the st_read table function for how to make use of this in practice. In o
...Use parenthesis to pass a list of options. See the GDAL docs for the driver you are using for a list of available options. DATASET_CREATION_OPTIONS : list of dataset creation options to pass to the GDAL driver. Use parenthesis to pass a list of options.... - https://man.mtool.work/duckdb/2.0/core_extensions/spatial/gdal.html
- Samples 7750
- Samples are used to randomly select a subset of a dataset. Examples Select a sample of exactly 5 rows from tbl using reservoir sampling: SELECT * FROM tbl USING SAMPLE 5; Select a sample of approximately 10% of the table using system sampli
...ROM tbl, tbl2 WHERE tbl.i = tbl2.i USING SAMPLE reservoir(20%); Syntax Samples allow you to randomly extract a subset of a dataset. Samples are useful for exploring a dataset faster, as often you might not be interested in the exact answers to queries,...ecified percentage of the table, but the variance is vectorSize times higher. As such, system sampling is not suitable for datasets with fewer than ~10k rows, where it can happen that all rows will be filtered out, or all the data will be included, even... - https://man.mtool.work/duckdb/2.0/sql/samples.html
DuckDB 2.0 Documentation / man.mtool.work