Results of 1 - 10 of about 29 for datasets (0.002 sec.)
- Hugging Face Support 13917
- The httpfs extension introduces support for the hf:// protocol to access datasets hosted in Hugging Face repositories. See the announcement blog post for details. Usage Hugging Face repositories can be queried using the following URL patter
...n: hf://datasets/⟨my_username⟩/⟨my_dataset⟩/⟨path_to_file⟩ For example, to read a CSV file, you can use the following query: SELECT * FROM...'hf://datasets/datasets-examples/doc-formats-csv-1/data.csv'; Where: datasets-examples is the name of the user/organization doc-formats-c...the query is: kind sound dog woof cat meow pokemon pika human hello To read a JSONL file, you can run: SELECT * FROM 'hf://datasets/datasets-examples/doc-formats-jsonl-1/data.jsonl'; Finally, for reading a Parquet file, use the following query: SELECT *... - https://man.mtool.work/duckdb/2.0/core_extensions/httpfs/hugging_face.html
- Dutch Railway Datasets 13754
- Examples in this documentation often use datasets based on the Dutch Railway datasets . These high-quality datasets are maintained by the team behind the Rijden de Treinen (Are the trains running?) application . This page contains download
...links to our mirrors to the datasets. In 2024, we have published a blog post on the analysis of these datasets . Loading the Datasets You can load the datasets...ZONE YES NULL NULL NULL Stop:Departure delay BIGINT YES NULL NULL NULL Stop:Departure cancelled BOOLEAN YES NULL NULL NULL Datasets 80-Month Datasets 2019-01 to 2025-08 : 80 months as uncompressed CSVs in a single zip Yearly Datasets The yearly datasets...are about 350 MB each. 2019 2020 2021 2022 2023 2024 2025 Monthly Datasets The monthly datasets are about 30 MB each. 2024-01 2024-02 2024-03 2024-04 2024-05 2024-06 2024-07 2024-08 2024-09 2024-10... - https://man.mtool.work/duckdb/2.0/guides/snippets/dutch_railway_datasets.html
- Benchmarks 9854
- For several of the recommendations in our performance guide, we use microbenchmarks to back up our claims. For these benchmarks, we use datasets from the TPC-H benchmark and the LDBC Social Network Benchmark's BI workload . Datasets We use
...when decompressed into .csv.gz files), while others use the same table's creationDate column (4 GB .parquet file). The TPC datasets used in the benchmark are generated with the DuckDB tpch extension . A Note on Benchmarks Running fair benchmarks is diffi...ark results presented in this guide do not constitute official TPC or LDBC benchmark results. Instead, they merely use the datasets of and some queries provided by the TPC-H and the LDBC BI benchmark frameworks, and omit other parts of the workloads such... - https://man.mtool.work/duckdb/2.0/guides/performance/benchmarks.html
- SQL on Apache Arrow 8696
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
..." and return as an Arrow Table results = con.sql("SELECT * FROM my_arrow_table WHERE i = 2").to_arrow_table() Apache Arrow Datasets Arrow Datasets stored as variables can also be queried as if they were regular tables. Datasets are useful to point toward...s directories of Parquet files to analyze large datasets. DuckDB will push column selections and the row filters it can translate down into the dataset scan operation so that only... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Create Synthetic Data 8435
- DuckDB lets you quickly generate synthetic datasets by combining a few building blocks: functions and clauses that generate rows , functions that generate values to fill those rows, and extensions that generate whole benchmark datasets. Thi
...──┴─────────────────────┴───────────────────────────────────────────────┴─────────┘ 10 rows 5 columns Generating Benchmark Datasets with Extensions To generate large, well-defined datasets, DuckDB ships with the TPC-H and TPC-DS extensions. Each provides... - https://man.mtool.work/duckdb/2.0/guides/snippets/create_synthetic_data.html
- Jupyter Notebooks 8076
- DuckDB's Python client can be used directly in Jupyter notebooks with no additional configuration, as the first section shows. Additional libraries can simplify SQL query development; the rest of this guide uses the JupySQL package. See oth
...=true first. %sql output_df << SELECT sum(i) AS total_i FROM input_df; Visualizing DuckDB Data The most common way to plot datasets in Python is to load them using Pandas and then use matplotlib or seaborn for plotting. This approach requires loading all...gement to the engine and ensures that intermediate computations do not keep eating up memory, efficiently plotting massive datasets. Boxplot & Histogram To create a boxplot, call %sqlplot boxplot , passing the name of the table and the column to plot. In...y You now have the ability to alternate between SQL and Pandas in a simple and highly performant way! You can plot massive datasets directly through the engine (avoiding both the download of the entire file and loading all of it into Pandas in memory). D... - https://man.mtool.work/duckdb/2.0/guides/python/jupyter.html
- TPC-H Extension 8027
- The tpch extension implements the data generator and queries for the TPC-H benchmark . Installing and Loading The tpch extension is shipped by default in some DuckDB builds, otherwise it will be transparently autoloaded on first use. If you
...- 1. Must be defined when the children arguments is defined suffix VARCHAR Append the suffix to table names Pre-Generated Datasets Pre-generated DuckDB databases for TPC-H are available for download: tpch-sf1.db (250 MB) tpch-sf3.db (754 MB) tpch-sf10.d...enerated by the tpch extension have small differences from the official TPC-H specification. To obtain fully TPC-compliant datasets, use the tpchgen-cli project . Limitation The tpch(⟨query_id⟩) function runs a fixed TPC-H query with pre-defined bind par... - https://man.mtool.work/duckdb/2.0/core_extensions/tpch.html
- Tuning Workloads 7929
- DuckDB aims to perform well out of the box, so most workloads do not require any tuning. When tuning is worthwhile, there is no single configuration or magic number that is optimal for every workload: the right settings depend on what you a
...uration reference for the full list of available settings. The preserve_insertion_order Option When importing or exporting datasets (from/to the Parquet or CSV formats), which are much larger than the available memory, an out of memory error may occur: O...(Out-of-Core Processing) A key strength of DuckDB is support for larger-than-memory workloads, i.e., it is able to process datasets that are larger than the available system memory (also known as out-of-core processing ). It can also run queries where th...some complex intermediate aggregate states to disk, these functions can cause an out-of-memory exception when run on large datasets. Profiling If your queries are not performing as well as expected, it's worth studying their query plans: Use EXPLAIN to p... - https://man.mtool.work/duckdb/2.0/guides/performance/how_to_tune_workloads.html
- sqllogictest Introduction 7880
- For testing plain SQL, we use an extended version of the SQL logic test suite, adopted from SQLite . Every test is a single self-contained file located in the test/sql directory. To run tests located outside of the default test directory, s
...ally run using cross products instead. Because of this, query verification can be very slow to do when working with larger datasets. It is therefore recommended to turn on query verification for all unit tests, except those involving larger datasets (mor... - https://man.mtool.work/duckdb/2.0/dev/sqllogictest/intro.html
- Data Ingestion 7684
- This page contains examples for data ingestion to Python using DuckDB. First, import the DuckDB package: import duckdb Then, proceed with any of the following sections. CSV Files CSV files can be read using the read_csv function, called eit
...kDB supports querying multiple types of Apache Arrow objects including tables , record batches, arrays and chunked arrays, datasets , RecordBatchReaders and scanners . See the Python guides for more examples. The filters of a query are pushed into Arrow... - https://man.mtool.work/duckdb/2.0/clients/python/data_ingestion.html
DuckDB 2.0 Documentation / man.mtool.work