SDAccel Debug Features

In this chapter, different features of the SDAccel™ environment supporting debugging efforts are examined. This chapter introduces the debugging tools available to analyze the project and perform debugging. The next chapter illustrates debug techniques using the features described here.

Defensive Programming

The SDAccel environment is capable of creating very efficient implementations. In some cases, however, implementation issues can occur. One such case is if a write request is emitted before there is enough data available in the process to complete the write transaction. This can cause deadlock conditions when multiple concurrent kernels are affected by this issue and the write request of a kernel depends on the input read being completed.

To avoid such situations, a conservative mode is available on the adapter. In principle, it delays the write request until it has all of the data necessary to complete the write. This mode is enabled during compilation by applying the following --xp option to the xocc compiler:

--xp param:compiler.axiDeadLockFree=yes

Because enabling this mode can impact performance, you might prefer to use this as a defensive programming technique where this option is inserted during development and testing and then removed during optimization. You might also want to add this option when the accelerator hangs repeatedly.

SDAccel Software Debug

The SDAccel environment supports typical software-like debugging for the host as well as kernel code. This flow is supported during software and hardware emulation and allows the use of break points and the analysis of variables as commonly done during software debugging.

Note: The host code can still be debugged in this mode even when the actual hardware is executed.

IDE Debug Flow

The SDAccel integrated design environment (IDE) flow provides easy access to the debug capabilities. Setting up an executable for debugging requires many steps when performed manually. These steps are handled by the IDE when you use the IDE debug flow.

Note: The SDAccel debug flow relies on shell scripts during debugging. This requires that the setup files such as .bashrc or .cshrc do not interfere with the SDAccel setup, such as the LD_LIBRARY_PATH.

Preparing the executable for debugging requires that you change the build configurations to enable the application of debug flags. You can set these options through the Project Settings in the SDx™ environment. There are two check boxes provided in the Options section for the Active build configuration. One enables host debug builds while the other enables debugging of the kernels. The checkboxes are named Host debug and Kernel debug respectively.

A more intuitive way to set these build options is through the context menu settings. To do this, right-click on the build configuration in the Assistant view and select Settings. Alternatively, you can double-click on the build configuration. The same two checkboxes are presented. While you can enable host debug on all targets, kernel debug is only supported for software emulation and hardware emulation build targets. This completes the setup; cleaning the build directory and rebuilding the application ensure that the project is ready for running in the GDB debug environment.

Running a GDB session from the IDE takes care of all the setup required. It automatically manages the environment setup for hardware or software emulation. It configures the SDAccel runtime to ensure debug support by the runtime environment, and manages the different consoles required for the execution of the kernel model, the host model, and the debug server.

As a result, when initiating the debug session, the SDAccel environment asks to switch into the debug perspective, which presents several windows to manage the different debug consoles and source code windows.

After starting the application, by default the application is stopped right at the beginning of the main function body in the host code. As with any GDB graphical front end, you can now set breakpoints and inspect variables in the host code. The SDAccel environment enables the same capabilities for the accelerated kernel implementation in a transparent way.

Note: In hardware emulation, because the C/C++/OpenCL™ kernel code is translated for efficient implementation, breakpoints cannot be placed on all statements. Mostly, untouched loops and functions are available for breakpoints, and similarly only preserved variables can be accessed.

Command Line Debug Flow

The command line debug flow in the SDAccel environment provides tools to debug the host and kernel application running in all modes: software emulation, hardware emulation, or hardware execution.

Note: The host code can be debugged using this feature in the hardware execution mode only.

There are four steps to debugging in the SDAccel environment using the command line flow:

General environment setup.
Prepare the host code for debug.
Prepare the kernel code for debug.
Launch GDB Standalone to debug.

IMPORTANT: The SDAccel environment supports host program debugging in all modes, but kernel debugging is only supported in the emulation flows with gdb. In addition, more hardware-centric debugging support, such as waveform analysis, is provided for the kernels.

General Environment Setup

Running software or hardware emulation requires first the tool setup followed by the selection of the emulation mode.

To set up the tool environment and run the SDx tool, source the file below so that SDx command settings are in the PATH:
- C Shell: source <SDX_INSTALL_DIR>/settings64.csh
- Bash: source <SDX_INSTALL_DIR>/settings64.sh
To set up the runtime environment reponsible for the interaction between the software and hardware implementation, source the file below:
- C Shell: source /opt/xilinx/xrt/setup.csh
- Bash: source /opt/xilinx/xrt/setup.sh

Table 1. Select Emulation Mode
Environment Variable	Value
`XCL_EMULATION_MODE`	`sw_emu` or `hw_emu` These environment settings are used by the runtime library to correctly execute the desired emulation. This is required in addition to building the executable for the specific emulation flow.

Preparing the Host Code

The host program needs to be compiled with debugging information generated in the executable by adding the -g option to the xcpp command line option, as follows:

xcpp -g ...

TIP: Because xcpp is simply a wrapper around the system compiler (gcc), the -g option enables the compiler to generate debug information.

Preparing the Kernel

Kernel code can be debugged together with the host program in either software emulation or hardware emulation. Debugging information needs to be generated first in the binary container by passing the -g option to the xocc command line executable:

xocc -g -t [sw_emu | hw_emu | hw] ...

The –t (or -target) option is used to specify the compilation target as either software emulation (sw_emu), hardware emulation (hw_emu), or hardware execution (hw).

In the software emulation flow, additional runtime checks can be performed for OpenCL based kernels. The runtime checks include:

Checking out-of-bound access made by kernel interface buffers (option: address)
Checking uninitialized memory access initiated by kernel local to kernel (option: memory)

The options are enabled through the -–xp option and the param:compiler.fsanitize directive, and need to be enabled during the link stage (-l) as shown in the following examples:

xocc -l –t sw_emu --xp param:compiler.fsanitize=address -o bin_kernel.xclbin
xocc -l –t sw_emu --xp param:compiler.fsanitize=memory -o bin_kernel.xclbin
xocc -l –t sw_emu --xp param:compiler.fsanitize=address,memory -o bin_kernel.xclbin

When applied, the emulation run produces a debug log with emulation diagnostic messages such as <project_dir>/Emulation-SW/<proj_name>-Default>/emulation_debug.log.

Launching GDB Host Code Debug

You can launch GDB standalone to debug the host program if the code is built with debug information (built with the -g flag). This flow should also work while using a graphical front-end for GDB, such as the Data Display Debugger (DDD) available from GNU. The following steps are the instructions for launching GDB.

To set up the environment to run the SDx tool, source the file below so that SDx command settings are in the PATH:
- C Shell: source <SDX_INSTALL_DIR>/settings64.csh
- Bash: source <SDX_INSTALL_DIR>/settings64.sh
To set up the runtime environment responsible for the interaction between the software and hardware implementation, source the file below:
- C Shell: source /opt/xilinx/xrt/setup.csh
- Bash: source /opt/xilinx/xrt/setup.sh
Ensure that the environment variable XCL_EMULATION_MODE is set to the correct mode.
The application debug feature must be enabled at runtime using an attribute in the sdaccel.ini file. Create an sdaccel.ini file in the same directory as your host executable, and include the following lines:
```
[Debug]
app_debug=true
```
This informs the runtime library that the kernel is debug enabled.
Start gdb through the Xilinx® wrapper:
```
xgdb --args host.exe test.xclbin
```
The xgdb wrapper performs the following setup steps under the hood:
- Launches GDB on the host program:
```
gdb --args host.exe test.xclbin
```
- Sets up the environment variables PYTHONHOME and PYTHONPATH to Python installation. Currently, the gdb in the SDx environment expects Python 2.6 or Python 2.7. For example, if the Python available on the machine is Python 2.6, set the environment as shown (Bash shell shown):
```
export PYTHONHOME=/usr 
export PYTHONPATH=/usr/lib64/python2.6/:/usr/lib64/python2.6/lib-dynload/
```
- Sources the Python script in the GDB console to enable the Xilinx GDB extensions:
```
gdb> source ${XILINX_SDX}/scripts/appdebug.py
```

Launching Host and Kernel Debug

In software emulation, to better mimic the hardware being emulated, kernels are spawned off as separate processes. If you are using GDB to debug the host code, breakpoints set on kernel lines are not hit because the kernel code is not run within that process. To support the concurrent debugging of the host code and the kernel code, the SDAccel environment provides a mechanism to attach to spawned kernels through the use of sdx_server.

You must start three different terminals in the command line flow. In the first terminal, start the sdx_server using the following command:
```
${XILINX_VIVADO}/bin/sdx_server --sdx-url
```
In a second terminal, run the host code in xgdb as described in Launching GDB Host Code Debug.
At this point, the first terminal running the sdx_server should provide a GDB listener port NUM on standard out. Keep track of the number returned by the sdx_server as the GDB listener port is used by GDB to debug the kernel process. When the GDB listener port is printed, the spawned kernel process has attached to the sdx_server and is waiting for commands from you. To control this process, you must start a new instance of GDB and connect to the sdx_server.

IMPORTANT: If the sdx_server is running, then all spawned processes compiled for debug connect and wait for control from you. If no GDB ever attaches or provides commands, the kernel code appears to hang.
In a third terminal, run the xgdb command, and at the GDB prompt, run the following commands:
- For software emulation:
```
“file ${XILINX_SDX}/data/emulation/unified/cpu_em/generic_pcie/model/genericpciemodel”
```
- For hardware emulation:
  1. Locate the sdx_server temporary directory:/tmp/sdx/$uid.
  2. Find the sdx_server process id (PID) containing the DWARF file of this debug session.
  3. At the gdb command line, run: file /tmp/sdx/$uid/$pid/NUM.DWARF.
- In either case, connect to the kernel process:
```
target remote :NUM
```
  Where NUM is the number returned by the sdx_server as the GDB listener port.
TIP: When debugging software/hardware emulation kernels in the SDAccel IDE, these steps are handled automatically and the kernel process is automatically attached, providing multiple contexts to debug both the host code and kernel code simultaneously.

After these commands are executed, you can set breakpoints on your kernels as needed, run the continue command, and debug your kernel code. When the all kernel invocations have finished, the host code continues, and the sdx_server connection drops.

For both software and hardware emulation flows, there are restrictions with respect to the accelerated kernel code debug interactions. Because this code is preprocessed in the software emulation flow, and then translated in the hardware emulation flow into a hardware description language (HDL) and simulated during debugging, it is not always possible to set breakpoints at all locations. Especially with hardware emulation, only a limited number of breakpoints such as on preserved loops and functions are supported. Nevertheless, this mode is useful for debugging the kernel/host interface.

Utilities for Hardware Debugging

In some cases, the normal SDAccel IDE and command line debug features are limited in their ability to isolate an issue. This is especially true when the software or hardware appears not to make any progress (hangs). These kinds of system issues are best analyzed with the help of the utilities mentioned in this section.

Using the Linux dmesg Utility

Well-designed Linux kernels and modules report issues through the kernel ring buffer. This is also true for SDAccel environment modules that allow you to debug the interaction with the accelerator board on the lowest Linux level.

Note: This utility intended for use in hardware debug only.

TIP: In most cases, it is sufficient to work with the less verbose xbutil feature to localize a problem. Refer to the SDx Command and Utility Reference Guide for more information on the xbutil command.

The dmesg utility is a Linux tool that lets you read the kernel ring buffer. The kernel ring buffer holds kernel information messages in a circular buffer. A circular buffer of fixed size is used to limit the resource requirements by overwriting the oldest entry with the next incoming message.

In the SDAccel tool, the xocl module and xclmgmt driver modules write informational messages to the ring buffer. Thus, for an application hang or crash, or for that matter any unexpected behavior (like being unable to program the bitstream, and so on), the dmesg tool should be used to check the ring buffer.

The following image shows the layers of the software platform associated with the SDAccel board platform.

To review messages from the Linux tool, you should first clear the ring buffer:

sudo dmesg -c

This flushes all messages from the ring buffer and make it easier to spot messages from the xocl and xclmgmt. After that, start your application and run dmesg in another terminal.

sudo dmesg

The dmesg utility prints a record such as the following module reports:

In the example shown above, the AXI Firewall 2 has tripped, which is better examined using the xbutil utility.

Using the Xilinx xbutil Utility

The Xilinx board utility (xbutil) is a powerful standalone command line utility that can be used to debug lower level hardware/software interaction issues. A full description of this utility can be found in the SDx Command and Utility Reference Guide.

With respect to debugging, the following xbutil options are of special interest:

query: Provides an overall status of an SDAccel environment platform.

program: Downloads a binary (xclbin) to the programmable region of the Xilinx device.

status: Extracts the status of the SDx environment Performance Monitors (spm) and the Lightweight AXI Protocol Checkers (lapc).

Hardware Debugging Using ChipScope

After the final system image (xclbin) is generated and executed on the SDAccel environment platform, the entire system including the host application running on the CPU, and the accelerated kernels on the Xilinx FPGA, can be confirmed to be executing correctly on the actual hardware. At this stage you can validate the functioning of the host code and kernel in the target hardware, and debug any issues found. Some of the conditions that can be looked for or analyzed are listed as follows:

System hangs that could be due to protocol violations:
- These violations can take down the entire system.
- These violations can cause the kernel to get invalid data or to hang.
- It is hard to determine where or when these violations originated.
- To debug this condition, you should use an ILA triggered off of the AXI protocol checker, which needs to be configured on the SDAccel platform in use.
Problems inside the RTL kernel:
- These problems are sometimes caused by the implementation: timing issues, race condition, and bad design constraint.
- Functional bugs that hardware emulation did not show.
Performance problems:
- For example, the frames per second processing is not what you expect.
- You can examine data beats and pipelining.
- Using an ILA with trigger sequencer, you can examine the burst size, pipelining, and data width to locate the bottleneck.

Checking the FPGA Board for Hardware Debug Support

Supporting hardware debugging requires the platform to support several IP components, most notably the Debug Bridge. Talk to your platform designer to determine if these components are included in the platform shell. If a Xilinx platform is used, debug availability can be verified using the platforminfo utility to query the platform. Debug capabilities are listed under the chipscope_debug objects.

For example, to query the a platform for hardware debug support, the following platforminfo command can be used. A response can be seen showing that the platform contains a user and management debug network, and also supports debugging a MicroBlaze™ processor.

$ platforminfo --json="hardwarePlatform.extensions.chipscope_debug" --platform xilinx_u200_xdma_201830_1
{
    "debug_networks": {
        "user": {
            "name": "User Debug Network",
            "pcie_pf": "1",
            "bar_number": "0",
            "axi_baseaddr": "0x000C0000",
            "supports_jtag_fallback": "false",
            "supports_microblaze_debug": "true",
            "is_user_visible": "true"
        },
        "mgmt": {
            "name": "Management Debug Network",
            "pcie_pf": "0",
            "bar_number": "0",
            "axi_baseaddr": "0x001C0000",
            "supports_jtag_fallback": "true",
            "supports_microblaze_debug": "true",
            "is_user_visible": "false"
        }
    }
}

Enabling ChipScope from the SDx IDE

The SDx IDE provides options to enable ChipScope debug on all the interface ports of the compute units in the design. When enabling this option on a compute unit, the SDAccel environment compiler adds a System ILA debug core to monitor the interface ports of the compute unit. This ensures that you can debug the interface signals on the SDAccel environment platform hardware while the kernel is running. You can access this through the Settings command by right-clicking on a kernel in the system build configuration in the Assistant window as shown below.

This brings up the Hardware Function Settings dialog box as shown in the following figure. You can use the Debug and Profiling Settings table in this dialog box to enable the ChipScope Debug checkbox for specific compute units of the kernel, which enables the monitoring of all the interfaces/ports on the compute unit.

TIP: Enabling the ChipScope Debug option on larger designs with multiple kernels and/or compute units can result in overuse of the FPGA device resources. Xilinx recommends using the xocc --dk list_ports option on the command line to determine the number and type of interfaces on the compute units. If you know which ports need to be monitored for debug as the design runs in hardware, the recommended methodology is to use the -–dk option documented in the following topic.

Command Line Flow

The full SDAccel kernel code compilation and linking command line flow can be found in the SDAccel Environment User Guide, Chapter 8. The following section covers the xocc linker options that can be used to list the available kernel ports as well as enable the System Integrated Logic Analyzer core on the selected ports. You should only use this flow if you are already familiar with the steps to build an SDAccel kernel at the command line.

The System Integrated Logic Analyzer debug core provides transaction-level visibility into an accelerated kernel or function running on hardware. AXI traffic of interest can also be captured and viewed using the System ILA core. The ILA core can be instantiated in the overall hardware of an existing RTL IP design to enable debugging features within that design, or it can be inserted automatically by the compiler. The xocc compiler provides the --dk option to attach System ILA cores at the interfaces to the kernels for debugging and performance monitoring purposes.

The -–dk option to enable ILA IP core insertion has the following syntax:

 --dk <[chipscope|list_ports]<:compute_unit_name><:interface_name>>

In general, the <interface_name> is optional. If not specified, all ports are expected to be analyzed. The chipscope option requires the explicit name of the compute unit to be provided for the <compute_unit_name> and <interface_name>. The list_ports option generates a list of valid compute units and port combinations in the current design and must be used after the kernel has been compiled.

Before using the --dk option, the kernel must be compiled into an .xo file. For a complete description of each xocc command line option as well as the complete SDAccel command line build flow, refer to the SDAccel Environment User Guide.

The first command compiles the kernel source files into an .xo file:

xocc -c -k <kernel_name> --platform <platform> -o <kernel_xo_file>.xo <kernel_source_files>

After the kernel has been compiled into an .xo file, --dk list_ports can be added to the command line options used during the xocc linking process. This causes the xocc compiler to print the list of valid compute units and port combinations. See the following example:

xocc -l --platform <platform> --nk <kernel_name>:<compute_units>:<kernel_nameN> --dk list_ports <kernel_xo_file>.xo

Finally, ChipScope debug can be enabled on the desired ports by replacing list_ports with the appropriate --dk chipscope command:

xocc -l --platform <platform> --nk <kernel_name>:<compute_units>:<kernel_nameN> --dk chipscope:<compute_unit_name>:<interface_name> <kernel_xo_file>.xo

Note:

Multiple --dk option switches can be specified in a single command line to additively increase interface monitoring capability.

Refer to the SDx Command and Utility Reference Guide for more information on any xocc option. When the design is built, you can debug the design using the Vivado® hardware manager as described in Vivado Design Suite User Guide: Programming and Debugging (UG908).