Results of 1 - 4 of about 4 for pq (0.003 sec.)
- SQL on Apache Arrow 11835
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
...nly the necessary data is pulled into memory. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds # connect to an in-memory database con = duckdb.connect() my_arrow_table = pa.Table.from_pydi...["one", "two", "three", "four"]}) # create example Parquet files and save in a folder parquet_folder = tempfile.mkdtemp() pq.write_to_dataset(my_arrow_table, parquet_folder) # link to Parquet files using an Arrow Dataset my_arrow_dataset = ds.data...use asynchronous IO to quickly access files. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds import pyarrow.compute as pc # connect to an in-memory database con = duckdb.connect() my_arr... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Integration with PyTorch 10133
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
...port tempfile import duckdb import numpy as np import pyarrow as pa import pyarrow.dataset as ds import pyarrow.parquet as pq import torch base_path = pathlib.Path(tempfile.mkdtemp()) parquet_dir = base_path / "train" table = pa.table( { "feature_0...": [0.1, 0.3, 0.5, 0.7], "feature_1": [1.0, 0.0, 1.0, 0.0], "label": [0, 1, 1, 0], } ) pq.write_to_dataset(table, str(parquet_dir)) con = duckdb.connect() train_dataset = ds.dataset(str(parquet_dir)) reader = con... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
- Relational API 7489
- The Relational API builds a query one step at a time from Python. A relation, a DuckDBPyRelation , is a piece of SQL text: the methods of the Transformation and Functions sections return a new relation whose text extends the one they were c
...ignored: a Parquet file records its own compression. Example import duckdb import pyarrow as pa import pyarrow.parquet as pq ids = pa.array([1], type=pa.int8()) texts = pa.array(['a'], type=pa.string()) example_table = pa.table([ids, texts], names...=["id", "text"]) pq.write_table(example_table, "code_example.parquet") duckdb_conn = duckdb.connect() rel = duckdb_conn.read_parquet("code_exa... - https://man.mtool.work/duckdb/2.0/clients/python/relational_api.html
- COPY Statement 7489
- Examples Read a CSV file into the lineitem table, using auto-detected CSV options: COPY lineitem FROM 'lineitem.csv'; Read a CSV file into the lineitem table, using manually specified CSV options: COPY lineitem FROM 'lineitem.csv' (DELIMITE
...R '|'); Read a Parquet file into the lineitem table: COPY lineitem FROM 'lineitem.pq' (FORMAT parquet); Read a JSON file into the lineitem table, using auto-detected options: COPY lineitem FROM 'lineitem.jso... - https://man.mtool.work/duckdb/2.0/sql/statements/copy.html
PREV
1
NEXT
DuckDB 2.0 Documentation / man.mtool.work