Getting Started with Unity ML-Agents
A tutorial on Unity ML-Agents for non-Unity developers.
In this article, you will learn how to start using Unity ML-Agents and train your own AI agents. This tutorial will go through the installation process, explaining the basics of the Unity interface, and of the ML-Agents Toolkit.
What is Unity and the Unity ML-Agents Toolkit
Unity is a game engine, a platform designed to facilitate people to build games. Besides game development, Unity is also utilized in other areas, such as simulation, architecture, animation, etc.
Unity ML-Agents is an open-source project that enables games and simulations to serve as environments for training intelligent agents.
Installation
Download Unity
Go to Unity’s website create an account and download Unity Hub.
On Unity Hub go to install and click on add to download a Unity version.
Any version later than 2018.4 should work. For this tutorial, I’ll be using 2020.3.0f1, which is the latest Long-Term Support version. If you are unable to add a new Unity version because of license follow this article to get your license activated.
Create a new Unity Project
Go to the project tab and click on NEW.
Select the 3D template, give the project a name and select the location where you want to save it.
Download Python
Go to the python website and download and install python. The version recommended in the ML-Agents documentation is either 3.6 or 3.7.
Create Virtual Environment and Install Required Packages
In order to not cause with the python packages necessary, we are going to create a virtual environment. On the directory of the Unity project, create a virtual environment
python -m venv venvActivate the virtual environment, and install the following python packages:
Pytorch version 1.7.1 (This might change in the future, check the ML-Agents documentation to be sure of what version to use).
pip install torch~=1.7.1 -f https://download.pytorch.org/whl/torch_stable.htmlML-Agents
pip install ml-agentsInstall ML-Agents Unity package
The last thing that we need is to install the Unity ML-Agents
Open the Unity project you created previously, you should see something like this:
On the top, click on Window, then Package Manager. In the Package Manager, set it to “Packages: Unity Registry” on the top left, then look for the ML-Agents package.
Before installing, click on the gear on the top right, open the project setting, go to Package Manager and check the option “Enable Preview Packages.” This will allow you to download later versions of ML-Agents. This is necessary because some of the examples provided utilize features from the preview versions.
Back to the package manager, you can now see the Preview versions of ML-Agents. Go ahead and install the latest one by clicking on Install on the bottom right.
Getting the Examples from ML-Agents
The last thing needed is to get the examples. Clone or download the ML-Agents repository. Inside Project/Assets copy the ML-Agents folder and add it to the Assets folder in your Unity project. You can simply drag it to the Project window in the Unity editor
In the console, there might be some error messages. This is because some of the examples utilize an extension of ML-Agents that you haven’t installed. You can delete the examples that are not functioning by locating them in the Examples folder that is inside the ML-Agents folder you just added to the project. To find what examples are not working, you can look at the error messages because they show the directory. In my case, Match3 and Sorter are missing the extension, so I’ll delete them for now.
Loading the 3D Ball example
The last thing you need to do before starting is to load a Scene to the Unity project. On the Project window, open folders in the following order Assets > ML-Agents > Examples > 3DBall > Scenes. You should see three icons with Unity’s logo. Double-click on the Icon name 3D-Ball, and you will load the scene
What you need to know about Unity
Now that you have everything installed, let’s start explaining some basic concepts of Unity.
Game Objects
Every object in Unity is a Game Object. Game objects are containers that can contain components, these components are responsible for the behavior of the game object.
The Scene
The scene contains all objects that will be in the environment. You can use the scene view to manipulate the position and scale of objects in the 3D-space using the tools that are on the top left.
The Hierarchy
The game objects that are in the scene can also be seen from the Hierarchy on the left. The hierarchy is basically a list of the game objects that are present in the scene.
Some objects can have a parent-child relationship, one example above is that “Ball” and “Agent” are both children of the “3DBall” object.
The hierarchy also shows what objects are “Prefabs.” Prefabs are objects that are a copy of another object that is saved in a folder instead of in the scene. You can make changes to the prefab and apply these changes to all objects that are in the scene, which were copied from this prefab. If you go to Examples\3DBall\Prefabs you should be able to find the 3DBall prefab.
The Inspector
As mentioned previously, game objects contain components. To see the components of a particular game object, you can click on it on the Hierarchy. The components of the game object will show on the right, in the Inspector window.
On the Hierarchy, expand one of the 3DBall game object, and click on the Agent game object. After you select it, the inspector will show
In the example above, Transform, Box Collider, Behaviour Parameters, Ball 3D Agent, Decision Requester, and Model Overrider are all components of the Agent GameObject.
Transform and Box Collider are built-in Unity components. Transform is what determines the position and scale of the Game Object in the Scene. Box Collider is what makes the GameObject solid, allowing the ball to collide into it and not move through it. The other components are C# Scripts, which are not built-in Unity features, but part of the ML-Agents framework.
Playing the Game
On the top, there is a play button.
In this example, after hitting play, you should see the ball falling on the agent’s head, and the agents start to move to try to balance the ball on their heads. This is happening because there is already a trained model which provides the agents their behavior.
Training the agent
Before start training an agent, let’s go back to the inspector of the agent on the scene. There are two components that are important, the Behavior Parameters and the Ball 3D Agent Script.
Behavior parameters
You can see there is Vector Observation. A vector is an array of floats. The space size parameter is the size of this array. In this case, its space size is 8 because each observation consists of the rotation of the agent on the X and Z axis, which are two float values, the position of the ball (X, Y, Z), and the velocity of the ball in each direction. The Stacked Vectors parameter is used if you want to stack the observations. If you stack observations, you wait until a number of observations are observed, then you send them all together to the neural network.
There are also Actions. In this case, the action is represented by two values, the amount of rotation there is on the X-axis and on the Z-axis.
Next, you have the Model and the Inference Device. There is already a trained model provided, but you can click on the circle and change the model to “None” if you want to train your own model. The inference device is what hardware is going to perform the computations.
Then there is the Behavior type. There are two behavior types, Heuristic Only, and Inference Only. The Heuristic Behavior is the behavior hard-coded by programmers. The programmer tells exactly what the agent should do at each state. The Inference Behavior is the behavior learned by the AI using the neural network. By setting this to default, it will try to first utilize the Inference Behavior. If no model is specified, it will then try to use Heuristics.
Ball 3D Agent Script
The other important component is the Ball 3D Agent Script. You can open the script by clicking on the three dots on the right and going to edit the script.
If you want to change your code editor click on Edit on the top left. Open preferences, on the External Tools tab you can change the External Script Editor to the one you prefer.
The first thing you should look at is that the Ball3DAgent is inheriting from an Agent class. The Agent class is a class from ML-Agents that contain all the methods necessary for creating and training a new agent. What we do is override these methods to make the Agent suitable for the task we want to train it to.
Initialize method
Then we can look at the Initialize method. This method is called once when the environment starts running. Here we are finding a reference to the rigid body component from the ball of that agent, and setting some environment parameters
Collect Observations method
This is where the agent receives the information about the state space. It receives the rotation on the X-axis and Z-axis, the position of the ball relative to the agent, and the velocity of the ball. This constitutes the vector of size 8 mentioned previously on the Behavior Parameters.
On Action Received method
The OnActionReceived method is where you turn the action value into concrete action. It is taking the action values and changing the rotation of the Agent. It is also where the Agent receives a reward for their action and where the episode might end if the ball falls off the head of the agent.
On Episode Begin method
The OnEpisodeBegin method resets the environment by changing the rotation of the Agent on the X and Z-axis to a random value, resetting the velocity of the ball to zero, and placing the ball in a random position on top of the agent.
Beginning the training process
To train an agent open a command prompt. Set the directory to one of your Unity project and activate the environment. You can start training by simply running the following command:
mlagents-learnA message saying that you can start training by pressing the Play Button in the Unity Editor will show up. Go to the Unity Editor and click on play.
On the Unity Game Window, you can see all the agents training in real-time. Over some time, you will notice how they start to keep the ball on top of their heads for a longer time.
Here is a video of the 3D Ball Agents training: