# Filesystem API

The `HfFileSystem` class provides a pythonic file interface to the Hugging Face Hub based on [`fsspec`](https://filesystem-spec.readthedocs.io/en/latest/).

## HfFileSystem[[huggingface_hub.HfFileSystem]]

`HfFileSystem` is based on [fsspec](https://filesystem-spec.readthedocs.io/en/latest/), so it is compatible with most of the APIs that it offers. For more details, check out [our guide](../guides/hf_file_system) and fsspec's [API Reference](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem).

#### huggingface_hub.HfFileSystem[[huggingface_hub.HfFileSystem]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L140)

Access a remote Hugging Face Hub repository as if were a local file system.

> [!WARNING]
> [HfFileSystem](/docs/huggingface_hub/v1.12.1/en/package_reference/hf_file_system#huggingface_hub.HfFileSystem) provides fsspec compatibility, which is useful for libraries that require it (e.g., reading
>     Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility
>     layer. For better performance and reliability, it's recommended to use `HfApi` methods when possible.

The file system supports paths for the `hf://` protocol, which follows those URL schemes:

* Models, Datasets and Spaces repositories:

```
hf://[@]/
hf://datasets/[@]/
hf://spaces/[@]/
```

* Buckets (generic storage):

```
hf://buckets//
```

Note: when using the [HfFileSystem](/docs/huggingface_hub/v1.12.1/en/package_reference/hf_file_system#huggingface_hub.HfFileSystem) directly, passing the `hf://` protocol prefix is optional in paths.

Usage:

```python
>>> from huggingface_hub import hffs

>>> # List files
>>> hffs.glob("my-username/my-model/*.bin")
['my-username/my-model/pytorch_model.bin']
>>> hffs.ls("datasets/my-username/my-dataset", detail=False)
['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json']

>>> # Read/write files
>>> with hffs.open("my-username/my-model/pytorch_model.bin") as f:
...     data = f.read()
>>> with hffs.open("my-username/my-model/pytorch_model.bin", "wb") as f:
...     f.write(data)
```

Specify a token for authentication:
```python
>>> from huggingface_hub import HfFileSystem
>>> hffs = HfFileSystem(token=token)
```

cp_filehuggingface_hub.HfFileSystem.cp_filehttps://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L814[{"name": "path1", "val": ": str"}, {"name": "path2", "val": ": str"}, {"name": "revision", "val": ": str | None = None"}, {"name": "**kwargs", "val": ""}]- **path1** (`str`) --
  Source path to copy from.
- **path2** (`str`) --
  Destination path to copy to.
- **revision** (`str`, *optional*) --
  The git revision to copy from.0

Copy a file within or between repositories.

> [!WARNING]
> Note: When possible, use `HfApi.upload_file()` for better performance.

**Parameters:**

endpoint (`str`, *optional*) : Endpoint of the Hub. Defaults to .

token (`bool` or `str`, *optional*) : A valid user access token (string). Defaults to the locally saved token, which is the recommended method for authentication (see https://huggingface.co/docs/huggingface_hub/quick-start#authentication). To disable authentication, pass `False`.

block_size (`int`, *optional*) : Block size for reading and writing files.

expand_info (`bool`, *optional*) : Whether to expand the information of the files.

- ****storage_options** (`dict`, *optional*) : Additional options for the filesystem. See [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.__init__).
#### exists[[huggingface_hub.HfFileSystem.exists]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L1009)

Check if a file exists.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.exists).

> [!WARNING]
> Note: When possible, use `HfApi.file_exists()` for better performance.

**Parameters:**

path (`str`) : Path to check.

**Returns:**

``bool``

True if file exists, False otherwise.
#### find[[huggingface_hub.HfFileSystem.find]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L753)

List all files below path.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.find).

**Parameters:**

path (`str`) : Root path to list files from.

maxdepth (`int`, *optional*) : Maximum depth to descend into subdirectories.

withdirs (`bool`, *optional*) : Include directory paths in the output. Defaults to False.

detail (`bool`, *optional*) : If True, returns a dict mapping paths to file information. Defaults to False.

refresh (`bool`, *optional*) : If True, bypass the cache and fetch the latest data. Defaults to False.

revision (`str`, *optional*) : The git revision to list from.

**Returns:**

``Union[list[str], dict[str, dict[str, Any]]]``

List of paths or dict of file information.
#### get_file[[huggingface_hub.HfFileSystem.get_file]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L1096)

Copy single remote file to local.

> [!WARNING]
> Note: When possible, use `HfApi.hf_hub_download()` or `HfApi.download_bucket_files` for better performance.

**Parameters:**

rpath (`str`) : Remote path to download from.

lpath (`str`) : Local path to download to.

callback (`Callback`, *optional*) : Optional callback to track download progress. Defaults to no callback.

outfile (`IO`, *optional*) : Optional file-like object to write to. If provided, `lpath` is ignored.
#### glob[[huggingface_hub.HfFileSystem.glob]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L737)

Find files by glob-matching.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob).

**Parameters:**

path (`str`) : Path pattern to match.

**Returns:**

``list[str]``

List of paths matching the pattern.
#### info[[huggingface_hub.HfFileSystem.info]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L899)

Get information about a file or directory.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.info).

> [!WARNING]
> Note: When possible, use `HfApi.get_paths_info()` or `HfApi.repo_info()`  for better performance
> (or `HfApi.get_bucket_paths_info()` or `HfApi.bucket_info()` for buckets)

**Parameters:**

path (`str`) : Path to get info for.

refresh (`bool`, *optional*) : If True, bypass the cache and fetch the latest data. Defaults to False.

revision (`str`, *optional*) : The git revision to get info from.

**Returns:**

``dict[str, Any]``

Dictionary containing file information (type, size, commit info, etc.).
#### invalidate_cache[[huggingface_hub.HfFileSystem.invalidate_cache]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L380)

Clear the cache for a given path.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.invalidate_cache).

**Parameters:**

path (`str`, *optional*) : Path to clear from cache. If not provided, clear the entire cache.
#### isdir[[huggingface_hub.HfFileSystem.isdir]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L1034)

Check if a path is a directory.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isdir).

**Parameters:**

path (`str`) : Path to check.

**Returns:**

``bool``

True if path is a directory, False otherwise.
#### isfile[[huggingface_hub.HfFileSystem.isfile]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L1052)

Check if a path is a file.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isfile).

**Parameters:**

path (`str`) : Path to check.

**Returns:**

``bool``

True if path is a file, False otherwise.
#### ls[[huggingface_hub.HfFileSystem.ls]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L497)

List the contents of a directory.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.ls).

> [!WARNING]
> Note: When possible, use `HfApi.list_repo_tree()` for better performance.

**Parameters:**

path (`str`) : Path to the directory.

detail (`bool`, *optional*) : If True, returns a list of dictionaries containing file information. If False, returns a list of file paths. Defaults to True.

refresh (`bool`, *optional*) : If True, bypass the cache and fetch the latest data. Defaults to False.

revision (`str`, *optional*) : The git revision to list from.

**Returns:**

``list[Union[str, dict[str, Any]]]``

List of file paths (if detail=False) or list of file information
dictionaries (if detail=True).
#### modified[[huggingface_hub.HfFileSystem.modified]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L874)

Get the last modified time of a file.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.modified).

**Parameters:**

path (`str`) : Path to the file.

**Returns:**

``datetime``

Last modified time of the file.
#### resolve_path[[huggingface_hub.HfFileSystem.resolve_path]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L276)

Resolve a Hugging Face file system path into its components.

**Parameters:**

path (`str`) : Path to resolve.

revision (`str`, *optional*) : The revision of the repo to resolve. Defaults to the revision specified in the path.

**Returns:**

``HfFileSystemResolvedPath``

Resolved path information containing `repo_type`, `repo_id`, `revision` and `path_in_repo`.
#### rm[[huggingface_hub.HfFileSystem.rm]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L447)

Delete files from a repository.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.rm).

> [!WARNING]
> Note: When possible, use `HfApi.delete_file()` for better performance.

**Parameters:**

path (`str`) : Path to delete.

recursive (`bool`, *optional*) : If True, delete directory and all its contents. Defaults to False.

maxdepth (`int`, *optional*) : Maximum number of subdirectories to visit when deleting recursively.

revision (`str`, *optional*) : The git revision to delete from.
#### url[[huggingface_hub.HfFileSystem.url]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L1070)

Get the HTTP URL of the given path.

**Parameters:**

path (`str`) : Path to get URL for.

**Returns:**

``str``

HTTP URL to access the file or directory on the Hub.
#### walk[[huggingface_hub.HfFileSystem.walk]]

[Source](https://github.com/huggingface/huggingface_hub/blob/v1.12.1/src/huggingface_hub/hf_file_system.py#L721)

Return all files below the given path.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk).

**Parameters:**

path (`str`) : Root path to list files from.

**Returns:**

``Iterator[tuple[str, list[str], list[str]]]``

An iterator of (path, list of directory names, list of file names) tuples.