How to Install Device Lab and XGen on Multiple Machines

A typical scenario is that XGen runs on a remote server (e.g., a machine in a data center) and the testing devices are local. XGen and device lab can be installed on different machines. This part explains the installation in such a scenario.

Network && firewall configuration before installation

There are many ports are used internally by XGen components. Use command below to check if your firewall is enabled:

sudo ufw status

If your firewall is enabled, please ensure following ports are allowed in the firewall rules: 22 80 28000 38000 19000 12379 6379 55672 45672 4369 5671 5672 15671 25672 27017 19001-19200. And please ensure these domains are also accessible *.cocopie.ai *.cocopie.io *.bitbucket.org *.aliyuncs.com *.amazoncs.com.

How to Install Device Lab Controller

Requirements

• GNU/Linux x86_64 with kernel version > 3.10 (Ubuntu or CentOS or WSL2)
• Docker
• Memory size: 16GB or more
• Disk size: 100GB or more
• Network: Important. The controller needs to be accessible from the agent and XGen. Ports 80, 19000, 28000, 55672 should be open and reserved for the controller. If you want to use Device Lab across the networks, please configure your firewall or security group to allow traffic on these ports.

Step 1: Install Docker

Please follow the official Docker instruction to make sure that Docker Engine has been installed. Instructions: Install Docker Engine

Make sure that the current user is in the docker group, if not, please run the following command:

 sudo groupadd docker  # if docker group does not exist
 sudo usermod -aG docker $USER

Important: re-login to make the change take effect.

 docker ps

if the above command can run successfully, the user has been granted the docker permission.

Install the following commands if they are not installed yet:

 sudo apt install unzip curl git bc

Step 2: Install Device Lab Controller

Install Device Lab Controller with Dashboard with the install script.

bash xgen_controller_deploy.sh

Usage: bash xgen_controller_deploy.sh [OPTIONS]
A script for fetching and setting up the docker image of CoCoPIE XGen controller.
Options:
  -h, --help                    Show this help message.
  --region string               US or HK (default US; HK is faster for mainland China).
  --path string                 Home directory of XGen Controller (default $HOME/XGenController).
Examples:
  bash xgen_controller_deploy.sh
  bash xgen_controller_deploy.sh --region HK
  bash xgen_controller_deploy.sh --region HK --path ~/XGenController

For most of the cases, the default options are good enough. So just run the following command:

bash xgen_controller_deploy.sh --region HK

# or

bash xgen_controller_deploy.sh --region US

If following output is shown, then the installation is successful.

Congratulations! XGen Controller has been successfully installed in the directory /home/ubuntu/XGenController (In the current example, the user is using Ubuntu. If your user is not using Ubuntu, that is okay. In any case, the installation will be performed in the directory specified by the $HOME variable)..

Next, you can access the XGen Controller by typing the command "run_controller". This will take you to the controller container environment.

For the initial Controller setup, kindly consult the documentation and configure the Controller's public IP address on the dashboard webpage.

After the installation, command run_controller will be written into .bashrc. Please run command source ~/.bashrc to make this command available in your shell.

Now for configuring the controller, please run command run_controller to start controller container and dependencies.

Output like this will be presented:

Kindly consult the documentation (https://xgen.cocopie.ai) and set up the public IP address of the controller on the dashboard webpage.
Once the configuration is finalized, proceed to execute the run_controller command once more.

• By default, the URL of dashboard is the address of your controller host machine and listening port is 80 and opened to 127.0.0.1. You can visit the dashboard by http://127.0.0.1

• If the user wants to visit dashboard in other machine instead of controller, the user should visit the dashboard by LAN (internal) IP address of controller host machine.

• If the firewall has been configured correctly, the user can also visit dashboard by WAN (public) IP address of controller host machine.

tips: If there are firewalls or other network restrictions, you can use an SSH tunnel to map port 80 on the deployed XGen machine to a port on your local machine, such as 8080. This way, you can access the remote machine's dashboard by visiting http://127.0.0.1:8080.

SSH tunnel port mapping example: ssh -L 8080:127.0.0.1:80 <username>@<host_ip>

At the first time, you'll be redirected to the login page. The default account and password are both admin. Then you'll be redirected to change password page to change the default password for security.

After changing your password, go to Config page by clicking the Config button on the top right corner. Then go to Controller tab to configure controller's external IP address. This address is used by other components (e.g., XGen, agents) to connect to the controller.

Please note, the IP address should be the LAN IP address of controller host machine. If the firewall is right configured, it can also be the WAN IP address of the controller hosting machine. Which address to use depends on the network environment and the way for the Device Lab to offer the service.

Click save to save the changes.

Then go back to the controller, enter the controller by running run_controller. (Alternatively, you can use the docker exec -it xgen_controller bash command, which has the same effect.) Then run xgenctl status to check the status of controller and its dependencies.

xgenctl status

Settings check passed.
Redis check passed.
MQ check passed.
Supervisor services check passed.

If any error is shown, please fix it before going to next step by following the error message.

You can use the following command to exit the Controller container, but it will not stop the Controller service:

$ exit

How to Upgrade Device Lab Controller

Just simply run upgrade_controller. Controller and dashboard will be upgraded to the latest version.

How to Uninstall Device Lab Controller

Just simply run uninstall_controller. The controller, dashboard, and related containers, images will be uninstalled.

How to Install Device Lab Agent

Agent should be installed after the controller is installed and configured.

Requirements

• GNU/Linux x86_64 with kernel version > 3.10 (Ubuntu or CentOS or WSL2)
• Docker
• Memory size: 16GB or more
• Disk size: 100GB or more
• Network: The agent needs to be accessible from the controller.
• Devices: The agent needs to be able to connect to the devices via USB. The default suggested devices are 64-bit devices (phones/tablets) that run Android 9 and iOS 11 or above. Support of other devices can be added separately.

Step 1: Install Docker

Please follow the official Docker instruction to make sure that Docker Engine has been installed. Instructions: Install Docker Engine

Make sure that the user is in the docker group, if not, please run the following command:

 sudo groupadd docker  # if docker group does not exist
 sudo usermod -aG docker $USER

Important: re-login to make the change take effect.

 docker ps

if the above command can run successfully, the user has been granted the docker permission.

Install the following commands if they are not installed yet:

 sudo apt install unzip curl git bc

Step 2: Install Device Lab Agent

Install Device Lab Agent with the install script.

Before installation, please ensure that usbmuxd service is not running on the host machine. Here we need to explain that we use usbmuxd for communication with iOS devices, similar to how adb-server works for Android devices. In the previous version, usbmuxd was installed as a system service on the host machine. In the current version, we have included it within the Docker container. Therefore, if usbmuxd is present on your machine, it will cause the agent container to fail to start. Therefore, when running the agent container, please stop the usbmuxd service and remove it to avoid conflicts.

How to stop usbmuxd service on UBUNTU20.04:

sudo systemctl stop usbmuxd

And how to delete usbmuxd service on UBUNTU20.04:

sudo apt remove usbmuxd -y

Install Device Lab Agent with the agent deploy script.

bash xgen_agent_deploy.sh

Usage: bash xgen_agent_deploy.sh [OPTIONS]
A script for fetching and setting up the docker image of CoCoPIE XGen agent.
Options:
  -h, --help                    Show this help message.
  --region string               US or HK (default US; HK is faster for mainland China).
  --path string                 Home directory of XGen (default $HOME/XGenAgent).
Examples:
  bash xgen_agent_deploy.sh
  bash xgen_agent_deploy.sh --region HK
  bash xgen_agent_deploy.sh --region HK --path ~/XGenAgent

For most of the cases, the default options are good enough. So just run the following command:

bash xgen_agent_deploy.sh --region HK

# or

bash xgen_agent_deploy.sh --region US

Output like this will be presented:

Congratulations! XGen Agent is successfully installed under directory /home/ubuntu/XGenAgent.(If your user is not using Ubuntu, that is okay. In any case, the installation will be performed in the directory specified by the $HOME variable).
Next, you can access XGen Agent by typing command 'run_agent'. It will lead you to the Agent container environment.

For the initial execution of the Agent, please access the xgen_agent container and provide the xgen-controller connection details in the configuration file '/root/.xgen_agent/agent.conf'.
Once the agent.conf is configured, do not forget to run 'xgenctl restart' to apply the settings, and use 'xgenctl status' to verify the proper functioning of all services.

After the installation, command run_agent will be written into .bashrc. Please run command source ~/.bashrc to make this command available in your shell.

Running command run_agent to enter the agent. To make the agent accessible from the controller, there are some configurations to be done.

• Configure the /root/.xgen_agent/agent.conf file of the agent container like the following

  xgen_controller_host=<your_controller_ip>
  xgen_controller_port=28000 # do not change this
  

  The 'xgen_controller_host' filled in here should be the public IP address of the controller, which is the same as the one entered in the dashboard configuration page during the controller installation.

• Then restart all background services by running the following command:

  xgenctl restart
  

  To check the status of the agent, run the following command:

  xgenctl status
  
  Settings check passed.
  Redis check passed.
  MQ check passed.
  Supervisor services check passed.
  

• One more thing. Check out the .agent_id at /root/.xgen_agent/.agent_id of the agent container

  cat /root/.xgen_agent/.agent_id
  

Log into the dashboard, and then go to the "config -> agent" page. Add the agent with the .agent_id above like the following: The purpose of this step is to add the agent ID to the whitelist of the controller, informing the controller that this is a legitimate agent.

• Before connecting any devices, please read the How to connect Devices part below for more information.

• Connect your devices via USB to the agent host machine, and run the following command to check the status of the devices:

  xgen_devices list
  
  Platform    UUID                       Brand    Model             Task Status    Connection Status
  ----------  -------------------------  -------  ----------------  -------------  -------------------
  Android     3131612233002SC            vivo     V2072A            idle           available
  Android     6c74106e                   Xiaomi   MI 9              idle           available
  iOS         00008101-0009245134F9001E  Apple    iPhone 12 16.3.1  idle           available
  

• You can use the following command to exit agent container, but it will not stop the agent service:

  $ exit
  

The user may connect multiple agents to a controller by simply repeating this process. If everything is OK, the agents will show in the dashboard:

How to Upgrade Device Lab Agent

Just simply run upgrade_agent

How to Uninstall Device Lab Agent

Just simply run uninstall_agent. The Agent and related containers, images will be uninstalled.

How to Install XGen

Requirements

• GNU/Linux x86_64 with kernel version > 3.10 (Ubuntu or CentOS or WSL2)
• NVIDIA GPU with Architecture >= Kepler (or compute capability 3.0)
• Memory size: 16GB or more

• Disk size: 500GB or more XGen by itself takes less than 20GB. But more space would be needed to hold AI models and datasets. The amount depends on the models and datasets the user uses. The AI models included in XGen's common AI are 500MB or less. The sizes of the default datasets included in XGen's common AI are as follows (downloadable on demand):

  • ImageNet: 144.02 GB

  • UCF101: 75.74 GB

  • CoCo2017: 25.01 GB

Step 1: Install Nvidia-Driver, Docker and Nvidia Container Toolkit

If the XGen machine is running the Ubuntu Linux operating system:

1. Please make sure that nvidia-dirver (version >= 510) has been installed. Instructions: Nvidia-Driver Installation Guide (CUDA installation is not mandatory on the host, because the Nvidia Container Toolkit in the later step includes CUDA support.) There is another solution as well if there is no specific requirement for the driver:

   # install ubuntu-devices-common to check devices version
   sudo apt install ubuntu-devices-common -y
   # check devices version and determine CUDA driver version needed
   sudo ubuntu-drivers devices # this command will list all available drivers and related apt package name
   # output example:
   vendor   : NVIDIA Corporation
   driver   : nvidia-driver-470 - distro non-free
   driver   : nvidia-driver-535-open - distro non-free
   driver   : nvidia-driver-525 - distro non-free
   driver   : nvidia-driver-535 - distro non-free recommended
   driver   : nvidia-driver-525-server - distro non-free
   driver   : nvidia-driver-535-server - distro non-free
   driver   : nvidia-driver-535-server-open - distro non-free
   driver   : nvidia-driver-470-server - distro non-free
   driver   : nvidia-driver-525-open - distro non-free
   driver   : xserver-xorg-video-nouveau - distro free builtin
   
   # install the recommended driver or the driver you need
   sudo apt install nvidia-driver-535 -y
   

   2. Please make sure that Docker Engine has been installed. Instructions: Install Docker Engine

   curl -fsSL https://get.docker.com -o get-docker.sh
   sudo bash ./get-docker.sh
   

   3. Please make sure Nvidia Container Toolkit is installed. Instructions: Nvidia Container Toolkit Installation Guide

After completing the above steps, the user can verify whether the deployment requirements of XGen Docker Image are met by running the following command in Linux:

 docker run --rm --gpus all nvidia/cuda:11.1.1-base-ubuntu20.04 nvidia-smi

When success, the NVIDIA GPU information will be output to the screen.

Hint: to run docker command without sudo authorization, the user needs to be granted the docker permission. The admin can do that through the following steps:

 sudo groupadd docker  # if docker group does not exist
 sudo usermod -aG docker $USER

Important: re-login to make the change take effect.

 docker ps

if the above command can run successfully, the user has been granted the docker permission.

Install the following commands if they are not installed yet:

 sudo apt install unzip curl git bc

Run command below to avoid memory issue:

 sudo sysctl vm.overcommit_memory=1

Step 2: Install XGen

XGen installation script has many installation options. The user can run the following command to see the options:

bash xgen_deploy.sh -h

Usage: bash xgen_deploy.sh [OPTIONS]
This script is designed to fetch and set up the Docker image for CoCoPIE XGen.
Options:
  -h, --help                    Display this help message.
  --region string               Specify the region (default is US; HK is faster for mainland China).
  --memory bytes                Set the memory limit for XGen (default is 70% of System Memory).
  --xgen-path string            Define the home directory for XGen (default is $HOME/XGen).
  --data string                 Specify a location on the host to store the dataset (default is $xgen-path/Data).
  --output string               Define the output directory for XGen (default is $xgen-path/Output).
Examples:
  bash xgen_deploy.sh
  bash xgen_deploy.sh --region HK
  bash xgen_deploy.sh --xgen-path ~/ML/XGen
  bash xgen_deploy.sh --xgen-path ~/ML/XGen --data ~/MLDataSets

For instance, the following command installs the XGen Docker image with the default setting. For users in mainland China, the following command usually runs faster, otherwise, use '--region US'.

 bash xgen_deploy.sh --region HK

After the command, a folder XGen is created under the user $HOME directory as XGen's home directory, and a command run_xgen is put into the directory. The user can define a folder as XGen's home directory by adding the option --xgen-path /path/to to the command. (If the user has specified a system path as XGen's home directory, the user needs to run install script with sudo; it is however not recommended to install XGen in a system directory).

Update the shell environment:

 source ~/.bashrc

The user can now type the following command to enter XGen shell:

 run_xgen --name <username>

Step3: Configure XGen

XGen is proprietary software. If a user lacks a license, execute xgenctl status and adhere to the instructions for acquiring a license.

The license file must be named license and situated within the /etc/xgen/license directory. Users can verify the license status by executing xgenctl status.

Configure the file /root/.xgen/xgen.conf to your device lab controller IP address and port. like this:

xgen_controller_host = <controller_ip>
xgen_controller_port = 28000

Then run xgenctl restart to restart the daemon services. For checking the status of XGen, run xgenctl status.

xgenctl status

Your license has 959 days left.
License check passed.
Settings check passed.
Redis check passed.
Controller connection check passed.

If there is any problem reported by xgenctl status, please follow the error message to fix it.

Listing Devices

In the XGen container, we also provide a tool to list the devices connected to Device Lab by command xgen_devices. It works for both Android and iOS devices. The usage is as follows:

# enter into the XGen container
run_xgen --name <username>
# to see the usage of xgen_devices
xgen_devices --help

Usage: xgen_devices [OPTIONS]

  List all devices connected to XGen.

Options:
  -avl, --available-only          Only show available devices which are
                                  reachable by XGen
  -p, --platform [android|ios|all]
                                  Filter devices by platform
  -v, --verbose                   Verbose mode
  -w, --watch                     Watch mode
  --help                          Show this message and exit.

# list all the connected devices
xgen_devices

The command returns a list of devices like the following:

Platform    UUID                       Brand    Model             Task Status    Connection Status
----------  -------------------------  -------  ----------------  -------------  -------------------
Android     3131612233002SC            vivo     V2072A            idle           available
Android     6c74106e                   Xiaomi   MI 9              idle           available
Android     5ttcbiduljtkgesg           Redmi    Redmi Note 8 Pro  idle           available
Android     93fdaa69                   Xiaomi   MI 8              idle           available
Android     W8EA7LEANFCIU8IN           OPPO     PACM00            idle           available
iOS         00008101-0009245134F9001E  Apple    iPhone 12 16.3.1  idle           available
iOS         00008020-001A29993CD2002E  Apple    iPhone XS 15.1    idle           available

• Platform is the platform of the device, either Android or iOS.
• UUID is the device ID.
• Brand is the brand of the device.
• Model is the model of the device.
• Task Status is the status of the device, either Idle or Busy.
• Connection Status is the status of the device, either Available or Unavailable.

How to Start XGen

To start XGen, the first step is to start its docker container by running the following command, which resides in the bin folder of XGen:

 run_xgen --name <username>

The command creates and starts an XGen container named xgen_<version>_<username>. Meanwhile, it creates a folder Projects under the XGen home directory. It is for holding all AI models and optimization results, including users' own models to be optimized. That folder is mapped to /root/Projects folder inside the XGen container; the files in that folder are hence accessible both inside and outside the XGen container.

After run_xgen --name <username>, XGen's shell will be started. In that shell, type the following to start XGen:

 XGen

Please follow the instruction to get the license.

Because the XGen container is a docker container, users can use the standard docker commands to manage the container, such as stopping it, checking its status, and so on. But users should avoid using docker commands to directly create XGen containers or rename the created XGen container. Always use run_xgen --name <username> to create XGen containers. Otherwise, there could be unexpected data overwritten and file corruptions. more detail XGen command usage please following XGen Usage.

Sometimes, during an XGen session, users might experience a disconnection from the machine running XGen. Here are some helpful hints.

How to Upgrade XGen

For users who already have version v1.1.0 installed, direct upgrading to the latest v1.3.0 version using the upgrade_xgen command is not supported. A fresh installation is required for v1.3.0 since there have been architectural and module organization optimizations.

For new users installing version v1.3.0, for future upgrades, simply use the upgrade_xgen command to directly upgrade.

For more details about v1.3.0, please see What's New.

After re-installation, the user can start XGen as before through run_xgen --name <username>.

How to Uninstall XGen

The user may uninstall XGen by running the uninstall_xgen command in the bin folder of XGen:

uninstall_xgen

The command will prompt options for users to choose the components (e.g., previously generated models) to keep.

How to Connect Devices

XGen measures the speed of the optimized models directly on connected devices (e.g., Android phones). The devices should be connected to the machine that runs an XGen Agent. This part explains how to connect devices.

Before explaining the detail, it is worth noting that regardless of the method of connection, the following apply:

• For Android devices, the user should turn on the "Developer mode", "USB debugging". Please see instructions.

• On the devices, at the first connection to the machine, an authorization window pops up; please authorize the machine to access this device and trust the machine always.

• The USB cable used for connection must be a data cable rather than a charging cable.
• Running command adb devices inside the agent shell would show the list of Android devices that are accessible by agent.
• Running command sib devices inside the agent shell would show the list of iOS devices that are accessible by agent.

Connect Android devices via USB

Connect the devices with the agent machine via a USB data cable. Some USB cables are only for charging, which won't work. (When the phone is connected with the computer via a data cable, usually the computer prompts to select connection mode, select Transfer Files.) The phone needs to be configured as stated above.

ADB has been installed in Agent container. So after completing the devices connection, then run the following command to check if the devices are connected to the agent container machine:

# enter 
run_agent
# check if your android device is connected
xgen_devices list -nm

Connect iOS devices

1. Install CoCoBench app in TestFlight. For installation, the user needs to have TestFlight installed on the iOS device to be used with XGen, and follow the invitation link in the XGen welcome email to install CoCoBench via TestFlight, and then launch CoCoBench on the iOS device. To guarantee the app runs correctly, here are some tips:

   • Make sure that CoCoBench of the latest version is installed.
   • Make sure that CoCoBench is running foreground.
   • Make sure that the screen of the iOS device is always on, which means that the screen is not locked and the device does not enter the sleep mode.
   • Make sure that there is no password set for the iOS device.
   • Make sure that the auto update of CoCoBench in TestFlight is turned on; otherwise, manually check it to keep the app up to date.
   • By default, CoCoBench expires 90 days after the publish date due to the limitation of TestFlight. The user must update CoCoBench to the latest version in TestFlight before the expiration date; enabling auto update would simplify this.

2. Connect the iOS device to the host machine via a USB data cable (not a charging cable). Then the host machine might pop up a window to ask the user to trust the host machine. Click Trust.

3. Enter Agent container, then run the following command to check if the iOS device is connected to the agent container machine:

   # enter 
   run_agent
   # check if your iOS device is connected
   xgen_devices list -nm
   

4. If the iOS device falls offline due to some reason (e.g., unplugged and replugin), the user needs to restart the usbmuxd services as shown below:

   xgenctl restart -s usbmuxd
   Service usbmuxd restarted.
   

   Alternatively, a simpler approach is to restart all services.

   xgenctl restart 
   

   After that, reconnect the iOS device to the host machine.

Hints

Avoid issues from disconnection Sometimes in the middle of an XGen session, the user may get disconnected from the machine that runs XGen. To ensure that the session is not interrupted by the disconnection and the user can get back to the session after reconnection, the user can use tmux:

• Install tmux by, for instance, the following command (on Ubuntu): sudo apt-get install tmux

• In the host terminal, run the following to start a new session: tmux new -s Some-Session-Name

• In the tmux window, start XGen as before: run_xgen --name <username>

• The session will continue running even if a disconnection happens. If the user is disconnected and then reconnected to the machine, run the following command will bring back the session: tmux attach -t Some-Session-Name.