Deep RL Framework

There are two components:

1. The client is the learning algorithms and runs in Python.
2. The soccer server serves as an interface between the client and the soccer simulator. This is the code contained in the tools/learning dir.

Installation

Client (Python - training algorithms)

• Update algorithms submodule baselines. The master branch should be up to date. If this is the first time, run:

git submodule update --init

• Follow repo installation in the README. Recommend installing it a virtual environment (conda). This will install tensorflow, etc...

OBS: As future work, it's a good idea to refactor soccer_env interface to integrate with other RL repositories, such as rllab and spinning-up-rl. It is also important to update baselines submodule instead of using this obsolete version.

Server (C++)

• Since the server uses the ITAndroids source code, you must first install all dependencies from the ITAndroids source code. For installation information, check itandroids-soccer3d/documentation/ and follow the guides:

• GuiaBasico_ITAndroids3d.pdf
• IntegrandoTensorFlow.md

(Optional) Setup - Cluster

• Firstly, create an account in colfax cluster (Intel DevCloud) and follow the steps to learn how to connect via ssh.
• Then, copy the content from this folder to your account home.
• Follow the Setup documentation to pull repositories and create the conda environment. IMPORTANT: The conda environment is located in ~/.conda/envs/rlearning-3d(if you need to change any hyperparameter in the algorithm, you need to change inside the folder lib/python3.5/site-packages/baselines)
• Copy distributed_devcloud folder with all scripts at HOME directory.

Building

Server - Local

• mkdir build && cd build && cmake .. && make -j 4 SoccerAgentServer_Main
• After build, you just need to run as detailed earlier.

Server - Cluster

The difference is that we need to ensure a completely static binary to avoid undefined references within the cluster environment.

• Use the rl-server-clusterbranch that has no dependency of libtensorflow_cc
• mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=Release .. && make -j 4 SoccerAgentServer_Main
• Then, just copy to the right folder inside DevCloud using scp command.
• Cluster folder structure (this structure ensures the scripts for running distributed works correctly):
  • ddpg-humanoid
  • distributed-devcloud
  • soccer3d
    • binaries --- SoccerAgentServer_Main

Running

Server (C++)

• The most stable version from the RL server is on branch rl-server and rl-server-cluster. The first one has the purpose of running in a local configuration. The second one, on the other side, is intended to run on Intel DevCloud.

  • OBS: We need to use two branches because of libtensorflow_cc. Is it a dynamic lib so we can't link statically to a binary that need to be deployed in the cluster. This is other problem to be solved. Until now, there is no static version from this lib.
  • Other way to solve this is to change CMAKE source to desconsider this lib during linking and the deploy the lib together with server's binary.

• Run build_proto.sh. This generates the grpc and protobuf classes inside codegen

• Build the SoccerAgentServer_Main target

• To run the server interface (from binaries folder):

./SoccerAgentServer_Main --server-port=3100 --monitor-port=3200

• If the environment you are trying to learn requires another agent simply build and run the other agent as well.

Client (Python - training algorithms)

• Run python run_codegen.py to generate the grpc and protobuf classes in python.
• After running the server interface, choose which RL algorithm you would like to learn
• Run the run_soccer.py script for the appropriate algorithm. For example, if you would like to run ppo1 algorithm, run the following command from any directory inside the baselines dir:

python -m baselines.ppo1.run_soccer

You should now see the agent appear on field.

Running Distributed

• Inside the cluster environment, you have the following scripts to run distributed:
  • delete_jobs.sh --- Delete all jobs running in the account.
  • remove_logs.sh --- Remove the logs generated by jobs in that folder.
  • start_distributed_training.sh <n_nodes> <agent_per_node> --- Run the distributed training using <n_nodes> worker nodes and <agent_per_node> agents in each node.

Simspark, agent and RoboViz ports

• Simspark

  rcssserver3d --agent-port 3101 --server-port 3201 rcssserver3d --agent-port 3100 --server-port 3200

• Agents:

  ./SoccerServer_Main --server-port=3101 --monitor-port=3201 ./SoccerServer_Main --server-port=3100 --monitor-port=3200

• RoboViz: ./roboviz.sh --serverPort=3201 ./roboviz.sh --serverPort=3200

  • All prints will run in the first RoboViz instance, unless you change the roboviz port in the config file, fixed to 32769.

Tensorboard

• To debug training metrics, use the tensorboard.
• tensorboard --logdir=<LOG_DIR> --port=<PORT>, where LOG_DIR is the log directory generated by the training procedure.

Additional information

• Check the oficial baseline repo for some more instructions.
• There are a few rl agents (each for a different task) that you could try running. To change the rl task, change the agent unique_ptr in the SoccerAgentService.cpp to the one you are interested in running.
• For more information about the usage of this code, check Luckeciano and Muzio Bachelor's Thesis.