Results of 1 - 2 of about 2 for tempfile (0.001 sec.)
- SQL on Apache Arrow 11228
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
...he dataset scan operation so that only the necessary data is pulled into memory. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds # connect to an in-memory database con = duckdb.connect() my_arr...[1, 2, 3, 4], 'j': ["one", "two", "three", "four"]}) # create example Parquet files and save in a folder parquet_folder = tempfile.mkdtemp() pq.write_to_dataset(my_arrow_table, parquet_folder) # link to Parquet files using an Arrow Dataset my_arrow_data...mpute operations instead. Arrow can use asynchronous IO to quickly access files. import duckdb import pyarrow as pa import tempfile import pyarrow.parquet as pq import pyarrow.dataset as ds import pyarrow.compute as pc # connect to an in-memory database... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Integration with PyTorch 7485
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
...t, streams the result as RecordBatch objects and converts each batch into feature and label tensors. import pathlib import tempfile import duckdb import numpy as np import pyarrow as pa import pyarrow.dataset as ds import pyarrow.parquet as pq import tor...ch base_path = pathlib.Path(tempfile.mkdtemp()) parquet_dir = base_path / "train" table = pa.table( { "feature_0": [0.1, 0.3, 0.5, 0.7], "feature_1": [1.0, 0.0... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
PREV
1
NEXT
DuckDB 2.0 Documentation / man.mtool.work