Results of 1 - 10 of about 10 for pa (0.003 sec.)
- SQL on Apache Arrow 13604
- DuckDB can query multiple different types of Apache Arrow objects. DuckDB pushes the filters of a query that it can translate, such as comparisons, into the scan of an Arrow dataset. Arrow tables, record batches, scanners and RecordBatchRea
...s, so an aggregate that depends on row order, such as first(i) without ORDER BY , can return a different row on each run. Apache Arrow Tables Arrow Tables stored in local variables can be queried as if they are regular tables within DuckDB. import...duckdb import pyarrow as pa # connect to an in-memory database con = duckdb.connect() my_arrow_table = pa.Table.from_pydict({'i': [1, 2, 3, 4], 'j': [..."one", "two", "three", "four"]}) # query the Apache Arrow Table "my_arrow_table" and return as an Arrow Table results = con.sql("SELECT * FROM my_arrow_table WHERE i = 2")... - https://man.mtool.work/duckdb/2.0/guides/python/sql_on_arrow.html
- Export to Apache Arrow 9779
- All results of a query can be exported to an Apache Arrow Table using the to_arrow_table function. Alternatively, results can be returned as a RecordBatchReader using the to_arrow_reader function and results can be read one batch at a time.
....execute() result they first copy all rows and warn with a UserWarning , and the reader does not stream. While a reader is partly read, other queries on its connection raise a ResourceInUseException . Read the reader to the end or close it first, o...sults of con.execute() and the duckdb module no longer have them. Export to an Arrow Table import duckdb import pyarrow as pa my_arrow_table = pa.Table.from_pydict({'i': [1, 2, 3, 4], 'j': ["one", "two", "three", "four"]}) # query the Apache Arrow...n arrow_batch_reader = res.to_arrow_reader() for batch in arrow_batch_reader: # Process a single chunk here print(batch.to_pandas()) Export through the Arrow PyCapsule Interface A relation implements __arrow_c_stream__ of the Arrow PyCapsule interf... - https://man.mtool.work/duckdb/2.0/guides/python/export_arrow.html
- Python Function API 9671
- You can create a DuckDB user-defined function (UDF) from a Python function so it can be used in SQL queries. Similarly to regular functions , they need to have a name, a return type and parameter types. Creating Functions To register a Pyth
...te_function("add_one", add_one) print(con.sql("SELECT add_one(41)").fetchall()) [(42,)] UDFs need NumPy. Without the numpy package, create_function raises InvalidInputException: 'numpy' is required for this operation, but it wasn't installed . The...signature is create_function(name, function, parameters=None, return_type=None, *, type=..., null_handling=..., exception_handling=..., side_effects=False) . The last fou...r parameters are keyword only, and a value they do not accept raises a TypeError : name A string representing the unique name o... - https://man.mtool.work/duckdb/2.0/clients/python/function.html
- Import from Apache Arrow 9457
- CREATE TABLE AS and INSERT INTO can be used to create a table from any query. We can then create tables or insert into existing tables by referring to the Apache Arrow object in the query. This example imports from an Arrow Table , but Duck
...DB can query different Apache Arrow formats as seen in the SQL on Arrow guide . import duckdb import pyarrow as pa # connect to an in-memory database...my_arrow = pa.Table.from_pydict({'a': [42]}) # create the table "my_table" from the DataFrame "my_arrow" duckdb.sql("CREATE TABLE my_tab... - https://man.mtool.work/duckdb/2.0/guides/python/import_arrow.html
- Integration with PyTorch 8402
- PyTorch tensors can be built from DuckDB query results by exporting each result batch as Apache Arrow . This pattern is useful when training or serving models on Parquet data because DuckDB can filter and project the input before the batche
...are converted to tensors. Installation pip install -U duckdb numpy pyarrow torch DuckDB to PyTorch This example queries a Parquet-backed Arrow dataset, streams the result as RecordBatch objects and converts each batch into feature and label tensor...s. import pathlib import tempfile import duckdb import numpy as np import pyarrow as pa import pyarrow.dataset as ds import pyarrow.par...quet as pq import torch base_path = pathlib.Path(tempfile.mkdtemp()) parquet_dir = base_path / "train" table = pa.table( { "feature_0": [0.1, 0.3, 0.5, 0.... - https://man.mtool.work/duckdb/2.0/guides/python/pytorch.html
- Relational API 8187
- The Relational API builds a query one step at a time from Python. A relation, a DuckDBPyRelation , is a piece of SQL text: the methods of the Transformation and Functions sections return a new relation whose text extends the one they were c
...│ │ 17 │ 3 │ └───────┴────────┘ Only a statement that reads, such as a SELECT , gives a lazy relation. Any other statement passed to sql() , such as CREATE TABLE or INSERT , runs at the call, and the relation holds what it returned. The params list...from an Arrow object. from_csv_auto Alias of read_csv() . from_df Create a relation object from the DataFrame in df. from_parquet Alias of read_parquet() . from_query Alias of sql() . query Alias of sql() . read_csv Create a relation object from t...he CSV file in 'name'. read_json Create a relation object from the JSON file in 'name'. read_parquet Create a relation object from the Parquet path(s) or file-like object(s) in 'path_or_buffer'. sql Run a SQL query. A... - https://man.mtool.work/duckdb/2.0/clients/python/relational_api.html
- Delta Extension 7651
- The delta extension adds support for the Delta Lake open-source storage format . It is built using the Delta Kernel . The extension reads Delta tables, local and remote, and writes them: it can create tables and append to them. For implemen
...connect to Unity Catalog, DuckDB has the unity_catalog extension . Installing and Loading The delta extension will be transparently autoloaded on first use from the official extension repository. If you would like to install and load it manually, r...un: INSTALL AND LOAD delta; Usage To scan a local Delta table, run: SELECT * FROM delta_scan('file:///⟨path⟩'); Attaching Tables A Delta table can also be attached as a database. The database holds exactly one table, with the sa...me name as the database, so my_table and my_table.my_table refer to the same table: ATTACH 'file:///⟨path⟩' AS my_table (TYPE delta); SELECT * FROM my_table; Reading from an S3 Bucket To scan a Delta table in an S3 bucket , ru... - https://man.mtool.work/duckdb/2.0/core_extensions/delta.html
- Collations 7651
- Collations provide rules for how text should be sorted or compared in the execution engine. Collations are useful for localization, as the rules for how text should be ordered are different for different languages or for different countries
.... These orderings are often incompatible with one another. For example, in English the letter y comes between x and z . However, in Lithuanian the letter y co..., different collations are supported. The user must choose which collation they want to use when performing sorting and comparison operations. By default, the BINARY collation is used. That means that strings are ordered and compared based only on...ot make much sense for special unicode characters. It is, however, by far the fastest method of performing ordering and comparisons. Hence it is recommended to stick with the BINARY collation unless required otherwise. The BINARY collation is also... - https://man.mtool.work/duckdb/2.0/sql/expressions/collations.html
- Data Ingestion 7437
- This page contains examples for data ingestion to Python using DuckDB. First, import the DuckDB package: import duckdb Then, proceed with any of the following sections. CSV Files CSV files can be read using the read_csv function, called eit
...ad_csv("example.csv", filename=True) Read a file-like object, such as an io.BytesIO or an open file; this needs the fsspec package: import io duckdb.read_csv(io.BytesIO(b"a,b\n1,2\n")) See the CSV Import page for more information. To write a relati...on to a CSV file, use write_csv . If you pass overwrite=False then it refuses to replace a file that exists. Parquet Files Parquet files can be read using the read_pa...rquet function, called either from within Python or directly from within SQL. Read from a single Parquet file: duckdb.read_parquet("example.parquet") Read multiple Parquet files from a folder: duckdb.read_parquet("folder/*... - https://man.mtool.work/duckdb/2.0/clients/python/data_ingestion.html
- Python API 7437
- Installation To use the DuckDB Python client, visit the Python installation page . The latest stable version of the DuckDB Python client is 2.0.0. Installation The DuckDB Python API can be installed using pip : pip install duckdb . Please s
...ee the installation page for details. It is also possible to install DuckDB using conda : conda install python-duckdb -c conda-forge . Python ver...ow() Data Input DuckDB can ingest data from a wide variety of formats – both on-disk and in-memory. See the data ingestion page for more information. import duckdb duckdb.read_csv("example.csv") # read a CSV file into a Relation duckdb.read_parquet...("example.parquet") # read a Parquet file into a Relation duckdb.read_json("example.json") # read a JSON file into a Relation duckdb.sq... - https://man.mtool.work/duckdb/2.0/clients/python/overview.html
PREV
1
NEXT
DuckDB 2.0 Documentation / man.mtool.work