Integration of the Input System Package with ML-Agents

Overview

One area we are always trying to improve is getting developers up and running with ML-Agents. With this in mind, we have implemented an InputActuatorComponent. This component integrates with the Input System Package to set up an action space for your Agent based on an InputActionAsset that is referenced by the IInputActionAssetProvider interface, or the PlayerInput component that may be living on your player controlled Agent. This means that if you have code outside of your agent that handles input, you will not need to implement the Heuristic function in agent as well. The InputActuatorComponent will handle this for you. You can now train and run inference on Agents with an action space defined by an InputActionAsset.

This implementation includes:

C# InputActuatorComponent you can attach to your Agent.
Implement the IInputActionAssetProvider in the Componenet where you handle player input.
An example environment where the input handling code is not in the Heuristic function of the Agent subclass.

Feedback

We have only implemented a small subset of InputControl types that we thought would cover a large portion of what most developers would use. Please let us know if you want more control types implemented by posting in the ML-Agents forum.

We would also like your feedback on the workflow of integrating this into your games. If you run into workflow issues please let us know in the ML-Agents forums, or if you've discovered a bug, please file a bug on our GitHub page.

Getting started

The C# code for the InputActuatorComponent exists inside of the extensions package (com.unity.ml-agents.extensions). A good first step would be to familiarize with the extensions package by reading the document here. The second step would be to take a look at how we have implemented the C# code in the example Input Integration scene (located under ML-Agents-Input-Example/Assets/ML-Agents/Examples/PushBlock/). Once you have some familiarity, then the next step would be to add the InputActuatorComponent to your player Agent. The example we have implemented uses C# Events to send information from the Input System.

Additionally, see below for additional technical specifications on the C# code for the InputActuatorComponent.

Technical specifications for the InputActuatorComponent

`IInputActionsAssetProvider` Interface

The InputActuatorComponent searches for a Component that implements IInputActionAssetProvider on the GameObject they both are attached to. It is important to note that if multiple Components on your GameObject need to access an InputActionAsset to handle events, they will need to share the same instance of the InputActionAsset that is returned from the IInputActionAssetProvider.

`InputActuatorComponent` class

The InputActuatorComponent is the bridge between ML-Agents and the Input System.. It allows ML-Agents to

create an ActionSpec for your Agent based on an InputActionAsset that comes from an IInputActionAssetProvider.
send simulated input from a training process or a neural network
let developers keep their input handling code in one place

This is accomplished by adding the InputActuatorComponenet to an Agent which already has the PlayerInput component attached.

Setting up a scene using the `InputActuatorComponent`

Add the com.unity.inputsystem version 1.1.0-preview.3 or later to your project via the Package Manager window.
If you have already setup an InputActionAsset skip to Step 3, otherwise follow these sub steps:
1. Create an InputActionAsset to allow your Agent to be controlled by the Input System.
2. Handle the events from the Input System where you normally would (i.e. a script external to your Agent class).
Add the InputSystemActuatorComponent to the GameObject that has the PlayerInput and Agent components attached.