models/Megatron/docs/source/api-guide/tensor_parallel.rst

tensor\_parallel package
========================

This package contains an implementation for tensor parallelism in transformer
models (see `Megatron-LM: Training Multi-Billion Parameter Language Models
Using Model Parallelism <https://arxiv.org/abs/1909.08053>`_ and `Reducing
Activation Recomputation in Large Transformer Models <https://arxiv.org/abs/2205.05198>`_
for details).

Submodules
----------

tensor\_parallel.cross\_entropy module
--------------------------------------

.. automodule:: core.tensor_parallel.cross_entropy
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.data module
----------------------------

.. automodule:: core.tensor_parallel.data
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.layers module
------------------------------

.. automodule:: core.tensor_parallel.layers
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.mappings module
--------------------------------

.. automodule:: core.tensor_parallel.mappings
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.random module
------------------------------

.. automodule:: core.tensor_parallel.random
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.utils module
-----------------------------

.. automodule:: core.tensor_parallel.utils
   :members:
   :undoc-members:
   :show-inheritance:

Module contents
---------------

.. automodule:: core.tensor_parallel
   :members:
   :undoc-members:
   :show-inheritance: