Skip to content

Chapter 6. Status Monitoring

This chapter explains how users can monitor the status of XGen's executions and Device Lab.

Monitoring XGen's Status

After entering all the configuration from the command line, the XGen console prints the total number of stages/cycles of this run. A stage refers to a main phase in the XGen execution, and one cycle in a stage refers to one iteration of the main loop in that stage. The specific meanings of a stage and a cycle depend on the selected mode of XGen; details are omitted in this manual.

During its execution, XGen prints its status to the screen and also records the info in a log file. Upon normal termination of an XGen execution, the log file is usually put inside the output folder in XGen (i.e., /root/output/ inside XGen docker) as mentioned in the Result section. the location will be printed on the screen right before the termination of XGen.

When there are errors, the user may check that log file for info. When the user files an error report, including that log file will be helpful.

Note: XGen sometimes terminate early before completing all the stages and cycles because it has already found a result that meets the requirement.

current_stage.png

At the end of the XGen run, there is a summary of the results of the run, including but not limited to the following:

  • the statistics of the final results: a table showing the statistics of the finally produced AI model
  • compiled file path: the folder that holds the finally produced AI model and code that are ready to be used in an app
  • onnx model path: the produced AI model in ONNX format
  • historical trained models info file: the csv file recording the info of all the runs of XGen (including this one) that share the same configuration file as this run does:

finish.png

Users can go into the corresponding paths to view the XGen output files.

Monitoring Device Lab's Status

Where to find the daemon services log files

There are many daemon services running in the background. And logs file for each files can be found in a hidden directory under the home directory of the container. Below is where the logs file locate:

  • Device Lab Controller: /root/.xgen_controller/logs
  • Device Lab Agent: /root/.xgen_agent/logs
  • Device Lab Dashboard: /root/.xgen_dashboard/logs
  • XGen: /root/.xgen/logs

How to check the status of the daemon services

The status of the daemon services can be checked by running the following command:

xgenctl status
This tool will help to determine if the daemon services are running properly. If not, it will show the error message.

How to restart the daemon services

The daemon services can be restarted by running the following command:

xgenctl restart
Command above will restart all the services.

Or restart a specific service by running the following command:

xgenctl restart -s SERVICE_NAME

Use xgenctl restart --help to see what services are available.