Buckets:
| # Table Classes | |
| Each `Dataset` object is backed by a PyArrow Table. | |
| A Table can be loaded from either the disk (memory mapped) or in memory. | |
| Several Table types are available, and they all inherit from [table.Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table). | |
| ## Table[[datasets.table.Table]] | |
| Wraps a pyarrow Table by using composition. | |
| This is the base class for `InMemoryTable`, `MemoryMappedTable` and `ConcatenationTable`. | |
| It implements all the basic attributes/methods of the pyarrow Table class except | |
| the Table transforms: `slice, filter, flatten, combine_chunks, cast, add_column, | |
| append_column, remove_column, set_column, rename_columns` and `drop`. | |
| The implementation of these methods differs for the subclasses. | |
| - **full** (`bool`, defaults to `False`) -- | |
| If `True`, run expensive checks, otherwise cheap checks only.- ``pa.lib.ArrowInvalid`` -- if validation fails</raises><raisederrors>``pa.lib.ArrowInvalid`` | |
| Perform validation checks. An exception is raised if validation fails. | |
| By default only cheap validation checks are run. Pass `full=True` | |
| for thorough validation checks (potentially `O(n)`). | |
| - **other** ([Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)) -- | |
| Table to compare against. | |
| - **check_metadata** `bool`, defaults to `False`) -- | |
| Whether schema metadata equality should be checked as well.`bool` | |
| Check if contents of two tables are equal. | |
| - **max_chunksize** (`int`, defaults to `None`) -- | |
| Maximum size for `RecordBatch` chunks. Individual chunks may be | |
| smaller depending on the chunk layout of individual columns.`List[pyarrow.RecordBatch]` | |
| Convert Table to list of (contiguous) `RecordBatch` objects. | |
| `dict` | |
| Convert the Table to a `dict` or `OrderedDict`. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| Arrow MemoryPool to use for allocations. Uses the default memory | |
| pool is not passed. | |
| - **strings_to_categorical** (`bool`, defaults to `False`) -- | |
| Encode string (UTF8) and binary types to `pandas.Categorical`. | |
| - **categories** (`list`, defaults to `empty`) -- | |
| List of fields that should be returned as `pandas.Categorical`. Only | |
| applies to table-like data structures. | |
| - **zero_copy_only** (`bool`, defaults to `False`) -- | |
| Raise an `ArrowException` if this function call would require copying | |
| the underlying data. | |
| - **integer_object_nulls** (`bool`, defaults to `False`) -- | |
| Cast integers with nulls to objects. | |
| - **date_as_object** (`bool`, defaults to `True`) -- | |
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype. | |
| - **timestamp_as_object** (`bool`, defaults to `False`) -- | |
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is | |
| useful if you have timestamps that don't fit in the normal date | |
| range of nanosecond timestamps (1678 CE-2262 CE). | |
| If `False`, all timestamps are converted to `datetime64[ns]` dtype. | |
| - **use_threads** (`bool`, defaults to `True`) -- | |
| Whether to parallelize the conversion using multiple threads. | |
| - **deduplicate_objects** (`bool`, defaults to `False`) -- | |
| Do not create multiple copies Python objects when created, to save | |
| on memory use. Conversion will be slower. | |
| - **ignore_metadata** (`bool`, defaults to `False`) -- | |
| If `True`, do not use the 'pandas' metadata to reconstruct the | |
| DataFrame index, if present. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| For certain data types, a cast is needed in order to store the | |
| data in a pandas DataFrame or Series (e.g. timestamps are always | |
| stored as nanoseconds in pandas). This option controls whether it | |
| is a safe cast or not. | |
| - **split_blocks** (`bool`, defaults to `False`) -- | |
| If `True`, generate one internal "block" for each column when | |
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this | |
| can temporarily reduce memory note that various pandas operations | |
| can trigger "consolidation" which may balloon memory use. | |
| - **self_destruct** (`bool`, defaults to `False`) -- | |
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow | |
| memory while converting the Arrow object to pandas. If you use the | |
| object after calling `to_pandas` with this option it will crash your | |
| program. | |
| - **types_mapper** (`function`, defaults to `None`) -- | |
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`. | |
| This can be used to override the default pandas type for conversion | |
| of built-in pyarrow types or in absence of `pandas_metadata` in the | |
| Table schema. The function receives a pyarrow DataType and is | |
| expected to return a pandas `ExtensionDtype` or `None` if the | |
| default conversion should be used for that type. If you have | |
| a dictionary mapping, you can pass `dict.get` as function.`pandas.Series` or `pandas.DataFrame``pandas.Series` or `pandas.DataFrame` depending on type of object | |
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the field to retrieve.`pyarrow.Field` | |
| Select a schema field by its column name or numeric index. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the column to retrieve.`pyarrow.ChunkedArray` | |
| Select a column by its column name, or numeric index. | |
| `pyarrow.ChunkedArray` | |
| Iterator over all columns in their numerical order. | |
| `pyarrow.Schema` | |
| Schema of the table and its columns. | |
| `List[pa.ChunkedArray]` | |
| List of all columns in numerical order. | |
| int | |
| Number of columns in this table. | |
| int | |
| Number of rows in this table. | |
| Due to the definition of a table, all columns have the same number of | |
| rows. | |
| `(int, int)`Number of rows and number of columns. | |
| Dimensions of the table: (#rows, #columns). | |
| Total number of bytes consumed by the elements of the table. | |
| ## InMemoryTable[[datasets.table.InMemoryTable]] | |
| The table is said in-memory when it is loaded into the user's RAM. | |
| Pickling it does copy all the data using memory. | |
| Its implementation is simple and uses the underlying pyarrow Table methods directly. | |
| This is different from the `MemoryMapped` table, for which pickling doesn't copy all the | |
| data in memory. For a `MemoryMapped`, unpickling instead reloads the table from the disk. | |
| `InMemoryTable` must be used when data fit in memory, while `MemoryMapped` are reserved for | |
| data bigger than memory or when you want the memory footprint of your application to | |
| stay low. | |
| - **full** (`bool`, defaults to `False`) -- | |
| If `True`, run expensive checks, otherwise cheap checks only.- ``pa.lib.ArrowInvalid`` -- if validation fails</raises><raisederrors>``pa.lib.ArrowInvalid`` | |
| Perform validation checks. An exception is raised if validation fails. | |
| By default only cheap validation checks are run. Pass `full=True` | |
| for thorough validation checks (potentially `O(n)`). | |
| - **other** ([Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)) -- | |
| Table to compare against. | |
| - **check_metadata** `bool`, defaults to `False`) -- | |
| Whether schema metadata equality should be checked as well.`bool` | |
| Check if contents of two tables are equal. | |
| - **max_chunksize** (`int`, defaults to `None`) -- | |
| Maximum size for `RecordBatch` chunks. Individual chunks may be | |
| smaller depending on the chunk layout of individual columns.`List[pyarrow.RecordBatch]` | |
| Convert Table to list of (contiguous) `RecordBatch` objects. | |
| `dict` | |
| Convert the Table to a `dict` or `OrderedDict`. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| Arrow MemoryPool to use for allocations. Uses the default memory | |
| pool is not passed. | |
| - **strings_to_categorical** (`bool`, defaults to `False`) -- | |
| Encode string (UTF8) and binary types to `pandas.Categorical`. | |
| - **categories** (`list`, defaults to `empty`) -- | |
| List of fields that should be returned as `pandas.Categorical`. Only | |
| applies to table-like data structures. | |
| - **zero_copy_only** (`bool`, defaults to `False`) -- | |
| Raise an `ArrowException` if this function call would require copying | |
| the underlying data. | |
| - **integer_object_nulls** (`bool`, defaults to `False`) -- | |
| Cast integers with nulls to objects. | |
| - **date_as_object** (`bool`, defaults to `True`) -- | |
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype. | |
| - **timestamp_as_object** (`bool`, defaults to `False`) -- | |
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is | |
| useful if you have timestamps that don't fit in the normal date | |
| range of nanosecond timestamps (1678 CE-2262 CE). | |
| If `False`, all timestamps are converted to `datetime64[ns]` dtype. | |
| - **use_threads** (`bool`, defaults to `True`) -- | |
| Whether to parallelize the conversion using multiple threads. | |
| - **deduplicate_objects** (`bool`, defaults to `False`) -- | |
| Do not create multiple copies Python objects when created, to save | |
| on memory use. Conversion will be slower. | |
| - **ignore_metadata** (`bool`, defaults to `False`) -- | |
| If `True`, do not use the 'pandas' metadata to reconstruct the | |
| DataFrame index, if present. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| For certain data types, a cast is needed in order to store the | |
| data in a pandas DataFrame or Series (e.g. timestamps are always | |
| stored as nanoseconds in pandas). This option controls whether it | |
| is a safe cast or not. | |
| - **split_blocks** (`bool`, defaults to `False`) -- | |
| If `True`, generate one internal "block" for each column when | |
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this | |
| can temporarily reduce memory note that various pandas operations | |
| can trigger "consolidation" which may balloon memory use. | |
| - **self_destruct** (`bool`, defaults to `False`) -- | |
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow | |
| memory while converting the Arrow object to pandas. If you use the | |
| object after calling `to_pandas` with this option it will crash your | |
| program. | |
| - **types_mapper** (`function`, defaults to `None`) -- | |
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`. | |
| This can be used to override the default pandas type for conversion | |
| of built-in pyarrow types or in absence of `pandas_metadata` in the | |
| Table schema. The function receives a pyarrow DataType and is | |
| expected to return a pandas `ExtensionDtype` or `None` if the | |
| default conversion should be used for that type. If you have | |
| a dictionary mapping, you can pass `dict.get` as function.`pandas.Series` or `pandas.DataFrame``pandas.Series` or `pandas.DataFrame` depending on type of object | |
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the field to retrieve.`pyarrow.Field` | |
| Select a schema field by its column name or numeric index. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the column to retrieve.`pyarrow.ChunkedArray` | |
| Select a column by its column name, or numeric index. | |
| `pyarrow.ChunkedArray` | |
| Iterator over all columns in their numerical order. | |
| `pyarrow.Schema` | |
| Schema of the table and its columns. | |
| `List[pa.ChunkedArray]` | |
| List of all columns in numerical order. | |
| int | |
| Number of columns in this table. | |
| int | |
| Number of rows in this table. | |
| Due to the definition of a table, all columns have the same number of | |
| rows. | |
| `(int, int)`Number of rows and number of columns. | |
| Dimensions of the table: (#rows, #columns). | |
| Total number of bytes consumed by the elements of the table. | |
| Names of the table's columns. | |
| - **offset** (`int`, defaults to `0`) -- | |
| Offset from start of table to slice. | |
| - **length** (`int`, defaults to `None`) -- | |
| Length of slice (default is until end of table starting from | |
| offset).`datasets.table.Table` | |
| Compute zero-copy slice of this Table. | |
| Select records from a Table. See `pyarrow.compute.filter` for full usage. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| For memory allocations, if required, otherwise use default pool.`datasets.table.Table` | |
| Flatten this Table. Each column with a struct type is flattened | |
| into one column per struct field. Other columns are left unchanged. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| For memory allocations, if required, otherwise use default pool.`datasets.table.Table` | |
| Make a new table by combining the chunks this table has. | |
| All the underlying chunks in the `ChunkedArray` of each column are | |
| concatenated into zero or one chunk. | |
| - **target_schema** (`Schema`) -- | |
| Schema to cast to, the names and order of fields must match. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| Check for overflows or other unsafe conversions.`datasets.table.Table` | |
| Cast table values to another schema. | |
| - **metadata** (`dict`, defaults to `None`) --`datasets.table.Table`shallow_copy | |
| EXPERIMENTAL: Create shallow copy of table by replacing schema | |
| key-value metadata with the indicated new metadata (which may be `None`, | |
| which deletes any existing metadata). | |
| - **i** (`int`) -- | |
| Index to place the column at. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column added. | |
| Add column to Table at position. | |
| A new table is returned with the column added, the original table | |
| object is left unchanged. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column added. | |
| Append column at end of columns. | |
| - **i** (`int`) -- | |
| Index of column to remove.`datasets.table.Table`New table without the column. | |
| Create new Table with the indicated column removed. | |
| - **i** (`int`) -- | |
| Index to place the column at. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column set. | |
| Replace column in Table at position. | |
| Create new table with columns renamed to provided names. | |
| - **columns** (`Union[List[str], List[int]]`) -- | |
| The column names or integer indices to select.[datasets.table.Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)New table with the specified columns, and metadata preserved. | |
| Select columns of the table. | |
| Returns a new table with the specified columns, and metadata preserved. | |
| - **columns** (`List[str]`) -- | |
| List of field names referencing existing columns.`datasets.table.Table`New table without the columns.- ``KeyError`` -- : if any of the passed columns name are not existing.</raises><raisederrors>``KeyError`` | |
| Drop one or more columns and return a new table. | |
| - **df** (`pandas.DataFrame`) -- | |
| - **schema** (`pyarrow.Schema`, *optional*) -- | |
| The expected schema of the Arrow Table. This can be used to | |
| indicate the type of columns if we cannot infer it automatically. | |
| If passed, the output will have exactly this schema. Columns | |
| specified in the schema that are not found in the DataFrame columns | |
| or its index will raise an error. Additional columns or index | |
| levels in the DataFrame which are not specified in the schema will | |
| be ignored. | |
| - **preserve_index** (`bool`, *optional*) -- | |
| Whether to store the index as an additional column in the resulting | |
| `Table`. The default of None will store the index as a column, | |
| except for RangeIndex which is stored as metadata only. Use | |
| `preserve_index=True` to force it to be stored as a column. | |
| - **nthreads** (`int`, defaults to `None` (may use up to system CPU count threads)) -- | |
| If greater than 1, convert columns to Arrow in parallel using | |
| indicated number of threads. | |
| - **columns** (`List[str]`, *optional*) -- | |
| List of column to be converted. If `None`, use all columns. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| Check for overflows or other unsafe conversions,`datasets.table.Table` | |
| Convert pandas.DataFrame to an Arrow Table. | |
| The column types in the resulting Arrow Table are inferred from the | |
| dtypes of the pandas.Series in the DataFrame. In the case of non-object | |
| Series, the NumPy dtype is translated to its Arrow equivalent. In the | |
| case of `object`, we need to guess the datatype by looking at the | |
| Python objects in this Series. | |
| Be aware that Series of the `object` dtype don't carry enough | |
| information to always lead to a meaningful Arrow type. In the case that | |
| we cannot infer a type, e.g. because the DataFrame is of length 0 or | |
| the Series only contains `None/nan` objects, the type is set to | |
| null. This behavior can be avoided by constructing an explicit schema | |
| and passing it to this function. | |
| Examples: | |
| ```python | |
| >>> import pandas as pd | |
| >>> import pyarrow as pa | |
| >>> df = pd.DataFrame({ | |
| ... 'int': [1, 2], | |
| ... 'str': ['a', 'b'] | |
| ... }) | |
| >>> pa.Table.from_pandas(df) | |
| <pyarrow.lib.Table object at 0x7f05d1fb1b40> | |
| ``` | |
| - **arrays** (`List[Union[pyarrow.Array, pyarrow.ChunkedArray]]`) -- | |
| Equal-length arrays that should form the table. | |
| - **names** (`List[str]`, *optional*) -- | |
| Names for the table columns. If not passed, schema must be passed. | |
| - **schema** (`Schema`, defaults to `None`) -- | |
| Schema for the created table. If not passed, names must be passed. | |
| - **metadata** (`Union[dict, Mapping]`, defaults to `None`) -- | |
| Optional metadata for the schema (if inferred).`datasets.table.Table` | |
| Construct a Table from Arrow arrays. | |
| - **mapping** (`Union[dict, Mapping]`) -- | |
| A mapping of strings to Arrays or Python lists. | |
| - **schema** (`Schema`, defaults to `None`) -- | |
| If not passed, will be inferred from the Mapping values | |
| - **metadata** (`Union[dict, Mapping]`, defaults to `None`) -- | |
| Optional metadata for the schema (if inferred).`datasets.table.Table` | |
| Construct a Table from Arrow arrays or columns. | |
| - **batches** (`Union[Sequence[pyarrow.RecordBatch], Iterator[pyarrow.RecordBatch]]`) -- | |
| Sequence of `RecordBatch` to be converted, all schemas must be equal. | |
| - **schema** (`Schema`, defaults to `None`) -- | |
| If not passed, will be inferred from the first `RecordBatch`.`datasets.table.Table` | |
| Construct a Table from a sequence or iterator of Arrow `RecordBatches`. | |
| ## MemoryMappedTable[[datasets.table.MemoryMappedTable]] | |
| The table is said memory mapped when it doesn't use the user's RAM but loads the data | |
| from the disk instead. | |
| Pickling it doesn't copy the data into memory. | |
| Instead, only the path to the memory mapped arrow file is pickled, as well as the list | |
| of transforms to "replay" when reloading the table from the disk. | |
| Its implementation requires to store an history of all the transforms that were applied | |
| to the underlying pyarrow Table, so that they can be "replayed" when reloading the Table | |
| from the disk. | |
| This is different from the `InMemoryTable` table, for which pickling does copy all the | |
| data in memory. | |
| `InMemoryTable` must be used when data fit in memory, while `MemoryMapped` are reserved for | |
| data bigger than memory or when you want the memory footprint of your application to | |
| stay low. | |
| - **full** (`bool`, defaults to `False`) -- | |
| If `True`, run expensive checks, otherwise cheap checks only.- ``pa.lib.ArrowInvalid`` -- if validation fails</raises><raisederrors>``pa.lib.ArrowInvalid`` | |
| Perform validation checks. An exception is raised if validation fails. | |
| By default only cheap validation checks are run. Pass `full=True` | |
| for thorough validation checks (potentially `O(n)`). | |
| - **other** ([Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)) -- | |
| Table to compare against. | |
| - **check_metadata** `bool`, defaults to `False`) -- | |
| Whether schema metadata equality should be checked as well.`bool` | |
| Check if contents of two tables are equal. | |
| - **max_chunksize** (`int`, defaults to `None`) -- | |
| Maximum size for `RecordBatch` chunks. Individual chunks may be | |
| smaller depending on the chunk layout of individual columns.`List[pyarrow.RecordBatch]` | |
| Convert Table to list of (contiguous) `RecordBatch` objects. | |
| `dict` | |
| Convert the Table to a `dict` or `OrderedDict`. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| Arrow MemoryPool to use for allocations. Uses the default memory | |
| pool is not passed. | |
| - **strings_to_categorical** (`bool`, defaults to `False`) -- | |
| Encode string (UTF8) and binary types to `pandas.Categorical`. | |
| - **categories** (`list`, defaults to `empty`) -- | |
| List of fields that should be returned as `pandas.Categorical`. Only | |
| applies to table-like data structures. | |
| - **zero_copy_only** (`bool`, defaults to `False`) -- | |
| Raise an `ArrowException` if this function call would require copying | |
| the underlying data. | |
| - **integer_object_nulls** (`bool`, defaults to `False`) -- | |
| Cast integers with nulls to objects. | |
| - **date_as_object** (`bool`, defaults to `True`) -- | |
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype. | |
| - **timestamp_as_object** (`bool`, defaults to `False`) -- | |
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is | |
| useful if you have timestamps that don't fit in the normal date | |
| range of nanosecond timestamps (1678 CE-2262 CE). | |
| If `False`, all timestamps are converted to `datetime64[ns]` dtype. | |
| - **use_threads** (`bool`, defaults to `True`) -- | |
| Whether to parallelize the conversion using multiple threads. | |
| - **deduplicate_objects** (`bool`, defaults to `False`) -- | |
| Do not create multiple copies Python objects when created, to save | |
| on memory use. Conversion will be slower. | |
| - **ignore_metadata** (`bool`, defaults to `False`) -- | |
| If `True`, do not use the 'pandas' metadata to reconstruct the | |
| DataFrame index, if present. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| For certain data types, a cast is needed in order to store the | |
| data in a pandas DataFrame or Series (e.g. timestamps are always | |
| stored as nanoseconds in pandas). This option controls whether it | |
| is a safe cast or not. | |
| - **split_blocks** (`bool`, defaults to `False`) -- | |
| If `True`, generate one internal "block" for each column when | |
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this | |
| can temporarily reduce memory note that various pandas operations | |
| can trigger "consolidation" which may balloon memory use. | |
| - **self_destruct** (`bool`, defaults to `False`) -- | |
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow | |
| memory while converting the Arrow object to pandas. If you use the | |
| object after calling `to_pandas` with this option it will crash your | |
| program. | |
| - **types_mapper** (`function`, defaults to `None`) -- | |
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`. | |
| This can be used to override the default pandas type for conversion | |
| of built-in pyarrow types or in absence of `pandas_metadata` in the | |
| Table schema. The function receives a pyarrow DataType and is | |
| expected to return a pandas `ExtensionDtype` or `None` if the | |
| default conversion should be used for that type. If you have | |
| a dictionary mapping, you can pass `dict.get` as function.`pandas.Series` or `pandas.DataFrame``pandas.Series` or `pandas.DataFrame` depending on type of object | |
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the field to retrieve.`pyarrow.Field` | |
| Select a schema field by its column name or numeric index. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the column to retrieve.`pyarrow.ChunkedArray` | |
| Select a column by its column name, or numeric index. | |
| `pyarrow.ChunkedArray` | |
| Iterator over all columns in their numerical order. | |
| `pyarrow.Schema` | |
| Schema of the table and its columns. | |
| `List[pa.ChunkedArray]` | |
| List of all columns in numerical order. | |
| int | |
| Number of columns in this table. | |
| int | |
| Number of rows in this table. | |
| Due to the definition of a table, all columns have the same number of | |
| rows. | |
| `(int, int)`Number of rows and number of columns. | |
| Dimensions of the table: (#rows, #columns). | |
| Total number of bytes consumed by the elements of the table. | |
| Names of the table's columns. | |
| - **offset** (`int`, defaults to `0`) -- | |
| Offset from start of table to slice. | |
| - **length** (`int`, defaults to `None`) -- | |
| Length of slice (default is until end of table starting from | |
| offset).`datasets.table.Table` | |
| Compute zero-copy slice of this Table. | |
| Select records from a Table. See `pyarrow.compute.filter` for full usage. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| For memory allocations, if required, otherwise use default pool.`datasets.table.Table` | |
| Flatten this Table. Each column with a struct type is flattened | |
| into one column per struct field. Other columns are left unchanged. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| For memory allocations, if required, otherwise use default pool.`datasets.table.Table` | |
| Make a new table by combining the chunks this table has. | |
| All the underlying chunks in the ChunkedArray of each column are | |
| concatenated into zero or one chunk. | |
| - **target_schema** (`Schema`) -- | |
| Schema to cast to, the names and order of fields must match. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| Check for overflows or other unsafe conversions.`datasets.table.Table` | |
| Cast table values to another schema | |
| - **metadata** (`dict`, defaults to `None`) --`datasets.table.Table`shallow_copy | |
| EXPERIMENTAL: Create shallow copy of table by replacing schema | |
| key-value metadata with the indicated new metadata (which may be None, | |
| which deletes any existing metadata. | |
| - **i** (`int`) -- | |
| Index to place the column at. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column added. | |
| Add column to Table at position. | |
| A new table is returned with the column added, the original table | |
| object is left unchanged. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column added. | |
| Append column at end of columns. | |
| - **i** (`int`) -- | |
| Index of column to remove.`datasets.table.Table`New table without the column. | |
| Create new Table with the indicated column removed. | |
| - **i** (`int`) -- | |
| Index to place the column at. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column set. | |
| Replace column in Table at position. | |
| Create new table with columns renamed to provided names. | |
| - **columns** (`Union[List[str], List[int]]`) -- | |
| The column names or integer indices to select.[datasets.table.Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)New table with the specified columns, and metadata preserved. | |
| Select columns of the table. | |
| Returns a new table with the specified columns, and metadata preserved. | |
| - **columns** (`List[str]`) -- | |
| List of field names referencing existing columns.`datasets.table.Table`New table without the columns.- ``KeyError`` -- : if any of the passed columns name are not existing.</raises><raisederrors>``KeyError`` | |
| Drop one or more columns and return a new table. | |
| ## ConcatenationTable[[datasets.table.ConcatenationTable]] | |
| The table comes from the concatenation of several tables called blocks. | |
| It enables concatenation on both axis 0 (append rows) and axis 1 (append columns). | |
| The underlying tables are called "blocks" and can be either `InMemoryTable` | |
| or `MemoryMappedTable` objects. | |
| This allows to combine tables that come from memory or that are memory mapped. | |
| When a `ConcatenationTable` is pickled, then each block is pickled: | |
| - the `InMemoryTable` objects are pickled by copying all the data in memory. | |
| - the MemoryMappedTable objects are pickled without copying the data into memory. | |
| Instead, only the path to the memory mapped arrow file is pickled, as well as the list | |
| of transforms to "replays" when reloading the table from the disk. | |
| Its implementation requires to store each block separately. | |
| The `blocks` attributes stores a list of list of blocks. | |
| The first axis concatenates the tables along the axis 0 (it appends rows), | |
| while the second axis concatenates tables along the axis 1 (it appends columns). | |
| If some columns are missing when concatenating on axis 0, they are filled with null values. | |
| This is done using `pyarrow.concat_tables(tables, promote=True)`. | |
| You can access the fully combined table by accessing the `ConcatenationTable.table` attribute, | |
| and the blocks by accessing the `ConcatenationTable.blocks` attribute. | |
| - **full** (`bool`, defaults to `False`) -- | |
| If `True`, run expensive checks, otherwise cheap checks only.- ``pa.lib.ArrowInvalid`` -- if validation fails</raises><raisederrors>``pa.lib.ArrowInvalid`` | |
| Perform validation checks. An exception is raised if validation fails. | |
| By default only cheap validation checks are run. Pass `full=True` | |
| for thorough validation checks (potentially `O(n)`). | |
| - **other** ([Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)) -- | |
| Table to compare against. | |
| - **check_metadata** `bool`, defaults to `False`) -- | |
| Whether schema metadata equality should be checked as well.`bool` | |
| Check if contents of two tables are equal. | |
| - **max_chunksize** (`int`, defaults to `None`) -- | |
| Maximum size for `RecordBatch` chunks. Individual chunks may be | |
| smaller depending on the chunk layout of individual columns.`List[pyarrow.RecordBatch]` | |
| Convert Table to list of (contiguous) `RecordBatch` objects. | |
| `dict` | |
| Convert the Table to a `dict` or `OrderedDict`. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| Arrow MemoryPool to use for allocations. Uses the default memory | |
| pool is not passed. | |
| - **strings_to_categorical** (`bool`, defaults to `False`) -- | |
| Encode string (UTF8) and binary types to `pandas.Categorical`. | |
| - **categories** (`list`, defaults to `empty`) -- | |
| List of fields that should be returned as `pandas.Categorical`. Only | |
| applies to table-like data structures. | |
| - **zero_copy_only** (`bool`, defaults to `False`) -- | |
| Raise an `ArrowException` if this function call would require copying | |
| the underlying data. | |
| - **integer_object_nulls** (`bool`, defaults to `False`) -- | |
| Cast integers with nulls to objects. | |
| - **date_as_object** (`bool`, defaults to `True`) -- | |
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype. | |
| - **timestamp_as_object** (`bool`, defaults to `False`) -- | |
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is | |
| useful if you have timestamps that don't fit in the normal date | |
| range of nanosecond timestamps (1678 CE-2262 CE). | |
| If `False`, all timestamps are converted to `datetime64[ns]` dtype. | |
| - **use_threads** (`bool`, defaults to `True`) -- | |
| Whether to parallelize the conversion using multiple threads. | |
| - **deduplicate_objects** (`bool`, defaults to `False`) -- | |
| Do not create multiple copies Python objects when created, to save | |
| on memory use. Conversion will be slower. | |
| - **ignore_metadata** (`bool`, defaults to `False`) -- | |
| If `True`, do not use the 'pandas' metadata to reconstruct the | |
| DataFrame index, if present. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| For certain data types, a cast is needed in order to store the | |
| data in a pandas DataFrame or Series (e.g. timestamps are always | |
| stored as nanoseconds in pandas). This option controls whether it | |
| is a safe cast or not. | |
| - **split_blocks** (`bool`, defaults to `False`) -- | |
| If `True`, generate one internal "block" for each column when | |
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this | |
| can temporarily reduce memory note that various pandas operations | |
| can trigger "consolidation" which may balloon memory use. | |
| - **self_destruct** (`bool`, defaults to `False`) -- | |
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow | |
| memory while converting the Arrow object to pandas. If you use the | |
| object after calling `to_pandas` with this option it will crash your | |
| program. | |
| - **types_mapper** (`function`, defaults to `None`) -- | |
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`. | |
| This can be used to override the default pandas type for conversion | |
| of built-in pyarrow types or in absence of `pandas_metadata` in the | |
| Table schema. The function receives a pyarrow DataType and is | |
| expected to return a pandas `ExtensionDtype` or `None` if the | |
| default conversion should be used for that type. If you have | |
| a dictionary mapping, you can pass `dict.get` as function.`pandas.Series` or `pandas.DataFrame``pandas.Series` or `pandas.DataFrame` depending on type of object | |
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the field to retrieve.`pyarrow.Field` | |
| Select a schema field by its column name or numeric index. | |
| - **i** (`Union[int, str]`) -- | |
| The index or name of the column to retrieve.`pyarrow.ChunkedArray` | |
| Select a column by its column name, or numeric index. | |
| `pyarrow.ChunkedArray` | |
| Iterator over all columns in their numerical order. | |
| `pyarrow.Schema` | |
| Schema of the table and its columns. | |
| `List[pa.ChunkedArray]` | |
| List of all columns in numerical order. | |
| int | |
| Number of columns in this table. | |
| int | |
| Number of rows in this table. | |
| Due to the definition of a table, all columns have the same number of | |
| rows. | |
| `(int, int)`Number of rows and number of columns. | |
| Dimensions of the table: (#rows, #columns). | |
| Total number of bytes consumed by the elements of the table. | |
| Names of the table's columns. | |
| - **offset** (`int`, defaults to `0`) -- | |
| Offset from start of table to slice. | |
| - **length** (`int`, defaults to `None`) -- | |
| Length of slice (default is until end of table starting from | |
| offset).`datasets.table.Table` | |
| Compute zero-copy slice of this Table. | |
| Select records from a Table. See `pyarrow.compute.filter` for full usage. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| For memory allocations, if required, otherwise use default pool.`datasets.table.Table` | |
| Flatten this Table. Each column with a struct type is flattened | |
| into one column per struct field. Other columns are left unchanged. | |
| - **memory_pool** (`MemoryPool`, defaults to `None`) -- | |
| For memory allocations, if required, otherwise use default pool.`datasets.table.Table` | |
| Make a new table by combining the chunks this table has. | |
| All the underlying chunks in the `ChunkedArray` of each column are | |
| concatenated into zero or one chunk. | |
| - **target_schema** (`Schema`) -- | |
| Schema to cast to, the names and order of fields must match. | |
| - **safe** (`bool`, defaults to `True`) -- | |
| Check for overflows or other unsafe conversions.`datasets.table.Table` | |
| Cast table values to another schema. | |
| - **metadata** (`dict`, defaults to `None`) --`datasets.table.Table`shallow_copy | |
| EXPERIMENTAL: Create shallow copy of table by replacing schema | |
| key-value metadata with the indicated new metadata (which may be `None`, | |
| which deletes any existing metadata). | |
| - **i** (`int`) -- | |
| Index to place the column at. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column added. | |
| Add column to Table at position. | |
| A new table is returned with the column added, the original table | |
| object is left unchanged. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column added. | |
| Append column at end of columns. | |
| - **i** (`int`) -- | |
| Index of column to remove.`datasets.table.Table`New table without the column. | |
| Create new Table with the indicated column removed. | |
| - **i** (`int`) -- | |
| Index to place the column at. | |
| - **field_** (`Union[str, pyarrow.Field]`) -- | |
| If a string is passed then the type is deduced from the column | |
| data. | |
| - **column** (`Union[pyarrow.Array, List[pyarrow.Array]]`) -- | |
| Column data.`datasets.table.Table`New table with the passed column set. | |
| Replace column in Table at position. | |
| Create new table with columns renamed to provided names. | |
| - **columns** (`Union[List[str], List[int]]`) -- | |
| The column names or integer indices to select.[datasets.table.Table](/docs/datasets/pr_8281/en/package_reference/table_classes#datasets.table.Table)New table with the specified columns, and metadata preserved. | |
| Select columns of the table. | |
| Returns a new table with the specified columns, and metadata preserved. | |
| - **columns** (`List[str]`) -- | |
| List of field names referencing existing columns.`datasets.table.Table`New table without the columns.- ``KeyError`` -- : if any of the passed columns name are not existing.</raises><raisederrors>``KeyError`` | |
| Drop one or more columns and return a new table. | |
| - **tables** (list of `Table` or list of `pyarrow.Table`) -- | |
| List of tables. | |
| - **axis** (`{0, 1}`, defaults to `0`, meaning over rows) -- | |
| Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns | |
| (horizontally). | |
| Create `ConcatenationTable` from list of tables. | |
| ## Utils[[datasets.table.concat_tables]] | |
| - **tables** (list of `Table`) -- | |
| List of tables to be concatenated. | |
| - **axis** (`{0, 1}`, defaults to `0`, meaning over rows) -- | |
| Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns | |
| (horizontally). | |
| `datasets.table.Table`If the number of input tables is > 1, then the returned table is a `datasets.table.ConcatenationTable`. | |
| Otherwise if there's only one table, it is returned as is. | |
| Concatenate tables. | |
| `List[str]`A list of paths to the cache files loaded by the table. | |
| Get the cache files that are loaded by the table. | |
| Cache file are used when parts of the table come from the disk via memory mapping. | |
Xet Storage Details
- Size:
- 38.1 kB
- Xet hash:
- 79814c382bd6330236a21ccbd1cb845f6b8f4c3b77d15d0be8bb5571d4446f4e
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.