45. Debugging & Profiling

45.1. Debugging with GDB

You can use a launcher script generated through the build process. You may run the game server with GDB attached, you may attach a GDB to a running game server process.

For example, you may run like the following. (It assumes that the project is hello and the flavor is game.)

$ ./hello.game-local --gdb

If there is a running game server process, it will automatically attaches to the process. If not, it will run a new game server with attached GDB. For the latter case, you can start your debugging session by typing run command.

 Please type `run` to start debugging.
 If you want to check the arguments, please type `show args`.


 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
 Copyright (C) 2016 Free Software Foundation, Inc.
 ... (skipped)
 Reading symbols from /usr/bin/funapi_runner...(no debugging symbols found)...done.
 (gdb) run

Tip

Due to the security enforcment on linux operating system, you may not allowed to attach a debugger to the process ran by the same user.

You can use the sudo command as a workaround. Or you may change the system configuration by running the following command. It is worth noting that you must not run the following command on a production system.

$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

After rebooting the machine, the configuration will be reset to the default state.

If there are multiple game-server instances of the same flavor, you can choose the process by specifying the process id on the command line.

# Please substitute the {PID} with the process id.
$ ./hello.game-local --gdb --pid {PID}

45.2. Debugging dump files

iFun Engine uses Google Breakpad to generate minidump files (.dmp) in the dump directory when the game server crashes. Minidump files are much smaller than core dump files. You can use these files to easily check the thread stacks that crashed.

45.2.1. Minidump file location

These files are saved in the logs/dumps/ directory in the build directory during development. If the game server is run using a daemon on a live server, dump files are saved in the /var/crash/funapi/{{project-name}}/ directory.

45.2.2. Viewing minidump data

You can see stacks in the minidump files by using the following commands:

Note

The following explanation assumes use of a daemon on a live server. If you are in the development phase, you can change /var/crash/funapi/{{project-name}}/ to logs/dumps/.

$ funapi_stackwalk /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp \
    /usr/share/{{project-name}}/symbols

The following is an example of operation.

 $ funapi_stackwalk /var/crash/funapi/example/08835639-ff4e-e778-6c14c453-5147e752.dmp /usr/share/example/symbols
 OS Linux 0.0.0 Linux 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64
 CPU amd64 family 6 model 42 stepping 7 8
 Crash SIGSEGV 0x0 31

 0 libexample.so: /build/example-source/src/event_handlers.cc:155 (+0x4) - example::OnTick
 1 libexample.so: /usr/include/boost/function/function_template.hpp:112 (+0x14) - ...
 2 funapi_runner: ...
 ...
 7 libboost_thread-mt.so.1.53.0 (+0xd24a)
 8 libpthread-2.17.so (+0x7df5)
 9 libc-2.17.so (+0xf61ad)

45.2.3. Debugging minidumps with GDB

Note

The following explanation assumes use of a daemon on a live server. If you are in the development phase, you can change /var/crash/funapi/{{project-name}}/ to logs/dumps/.

Use the following commands to debug minidump files with GDB.

$ funapi_core_analyze /usr/lib/lib{{project-name}}.so \
    /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp

 $ funapi_core_analyze /usr/lib/libhello.so /var/crash/funapi/hello/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp
 Finding game server .so file
 Converting dumps/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp into coredump
 Launching gdb to inspect generated coredump (hello.23945.core).
 Please press 'y' to load required symbols

 ...

 Failed to read a valid object file image from memory.
 Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.game --app_flavor=game --'.
 #0  0x00007fc18d76c2de in ?? ()
 add symbol table from file "src/libhello.so" at
   .text_addr = 0x7fc18d0ceaf0
 Reading symbols from src/libhello.so...done.
 (gdb) bt
 #0  0x00007fc18d76c2de in hello::TestFilters () at ../src/filter.cc:367
 #1  0x00007fc18d43094f in (anonymous namespace)::HelloServer::Install (arguments=...) at ../src/hello_server.cc:113
 #2  0x00007fc18d430f18 in HelloServer_install (arguments=...) at ../src/hello_server.cc:182
 #3  0x00000000007328d3 in ?? ()
 #4  0x0000000000601b5c in ?? ()
 #5  0x00007fc1962d2ec5 in ?? ()
 #6  0x0000000000000000 in ?? ()
 (gdb) info locals
 x = 0x0
 (gdb)

45.2.4. Generating core dumps instead of minidumps

In order to minimize disk usage, you can only see stack values and directly referenced values in minidumps. If you need more data for debugging, you can save core dumps instead of minidumps and debug with those.

Warning

Since the entire memory footprint of the core dump process is saved on the disk, disk usage may increase when this process uses a lot of memory.

Important

You need to check OS settings first to see whether core dumps are generated.

# if the output of the command is zero,
# coredump will not be available on disk.
$ ulimit -c
0

# set the maximum size of the coredump to unlimited
$ ulimit -c unlimited

Run the launcher with the following settings.

$ ./hello-local --enable_breakpad=false

If OS settings are not changed, core files are generated in the current directory after crashes. You can debug applicable files as follows:

$ gdb /usr/bin/funapi_runner core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
...
Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.default --app_flavor=de'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
71      *x = 0;
[Current thread is 1 (Thread 0x7f287e5ec800 (LWP 21649))]
(gdb) bt
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
#1  0x00007f28715bad8a in HelloServer_start () at ../src/hello_server.cc:83
#2  0x0000000000845c03 in fun::StartComponents() ()
#3  0x000000000065eabe in main ()
(gdb)

45.3. Debug Memory Issues

You can debug various memory problems using Address Sanitizer with iFun Engine. It will detect issues such as a double-free, use-after-free. And it will locate the problematic call stack for the detected problems. For detailed information please refer Address Sanitizer website.

It is worth noting that memory deallocation during shutdown will not be accounted.

Warning

Windows Platform is not supported.

Warning

For Centos 7, memory leak detection does not work due to the low GCC version.

45.3.1. Enable Address Sanitizer

Update the USE_ADDRESS_SANITIZER flag in CMakeLists.txt. (If you cannot find the flag, please add it before include(Funapi).)

# Builds with Address Sanitizer for memory corruption debugging.
set(USE_ADDRESS_SANITIZER true)

After updating the flag, rebuild your project.

45.3.2. Run Tests

For example, add following code to the Start() function of your game server.

int *p = new int;
delete p;
*p = 1;  // Error(Heap use after free)

After rebuilding and executing the server process, ASAN will detect following error:

=================================================================
==73232==ERROR: AddressSanitizer: heap-use-after-free on address 0x60200001feb0 at pc 0x7f5a00b0147b bp 0x7fff6ac9caf0 sp 0x7fff6ac9cae0
WRITE of size 4 at 0x60200001feb0 thread T0
    #0 0x7f5a00b0147a in Start /tmp/test/example-source/src/example_server.cc:83
    #1 0x7f5a00b015b6 in ExampleServer_start /tmp/test/example-source/src/example_server.cc:97
    #2 0xa3e1c4 in fun::StartComponents() (/usr/bin/funapi_runner_asan+0xa3e1c4)
    #3 0x7ea0bb in main (/usr/bin/funapi_runner_asan+0x7ea0bb)
    #4 0x7f5a09df16a2 in __libc_start_main (/lib64/libc.so.6+0x236a2)
    #5 0x825ccd in _start (/usr/bin/funapi_runner_asan+0x825ccd)

0x60200001feb0 is located 0 bytes inside of 4-byte region [0x60200001feb0,0x60200001feb4)
freed by thread T0 here:
    #0 0x7f5a0e7f9d60 in operator delete(void*, unsigned long) (/lib64/libasan.so.5+0xf2d60)
    #1 0x7f5a00b01440 in Start /tmp/test/example-source/src/example_server.cc:82
    #2 0x7f5a00b015b6 in ExampleServer_start /tmp/test/example-source/src/example_server.cc:97
    #3 0xa3e1c4 in fun::StartComponents() (/usr/bin/funapi_runner_asan+0xa3e1c4)
    #4 0x7ea0bb in main (/usr/bin/funapi_runner_asan+0x7ea0bb)
    #5 0x7f5a09df16a2 in __libc_start_main (/lib64/libc.so.6+0x236a2)

previously allocated by thread T0 here:
    #0 0x7f5a0e7f8780 in operator new(unsigned long) (/lib64/libasan.so.5+0xf1780)
    #1 0x7f5a00b01425 in Start /tmp/test/example-source/src/example_server.cc:81
    #2 0x7f5a00b015b6 in ExampleServer_start /tmp/test/example-source/src/example_server.cc:97
    #3 0xa3e1c4 in fun::StartComponents() (/usr/bin/funapi_runner_asan+0xa3e1c4)
    #4 0x7ea0bb in main (/usr/bin/funapi_runner_asan+0x7ea0bb)
    #5 0x7f5a09df16a2 in __libc_start_main (/lib64/libc.so.6+0x236a2)

SUMMARY: AddressSanitizer: heap-use-after-free /tmp/test/example-source/src/example_server.cc:83 in Start

If source files and line number are not printed, install llvm package and try again with the command below.

$ ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer ASAN_OPTIONS=symbolize=1 {project-name}.{flavor-name}-local

45.4. Debug A Memory Leak

Note

Memory leak debugging is supported for C++.

45.4.1. Prerequisites

This method requires a certain feature from recent linux kernels. It may not work with older linux kernels. For Ubuntu 16.04, you need to install packages from iovisor. For Centos 7, you need to install a kernel-ml package, which is a recent mainline kernel release.

  1. Install bcc-tools or bpfcc-toools package, which is provided by your linux distro.

  2. Copy memleak script to your home directory. It is located under /usr/share/bcc/tools or /usr/sbin/ directory, and its name can be either memleak or memleak-bpfcc.

  3. To trace the jemalloc used by the iFunEngine, you need modify certain line from the script. Please change attach_probes("pvalloc") line to attach_probes("pvalloc", can_fail=True).

45.4.2. Trace Memory Leaks

# Find a running game server process.
# You may run following command and use the PID from it.
$ pgrep -a funapi_runner

# Trade the selected game server process. (PID=1234)
$ sudo ./memleak -p 1234 --top=5 --older=10000 --obj=jemalloc

  • The command traces memory allocations of the process with PID=1234.

  • It will periodically print TOP 5 call stacks of currently allocated memory.

  • It will only print memory allocations older than 10 seconds. (--older=10000)

  • It only traces memory allocation/deallcation by the jemalloc allocator.

The trace can be only performed after the memleak program is exectued. It cannot detect any memory leak happened before its launch.

For example, it may print following traces. The output order is arbitrary. You should look for the call stack with most frequent allocation or, call stack with huge allocations.

[15:06:03] Top 5 stacks with outstanding allocations:
  2056 bytes in 1 allocations from stack
    operator new(unsigned long)+0x18 [libstdc++.so.6.0.25]
    [unknown] [funapi_runner]
    fun::UpdateCounter(...)+0x45 [funapi_runner]
    ... (중략)
    boost::detail::thread_data<boost::function<void ()> >::run()+0x36 [funapi_runner]
    [unknown] [libboost_thread.so.1.65.1]
  2504 bytes in 1 allocations from stack
    operator new(unsigned long)+0x18 [libstdc++.so.6.0.25]
    [unknown] [funapi_runner]
    fun::RawRandomGenerator::GenerateUuid(bool)+0x3b [funapi_runner]
    ... (중략)
    fun::ThreadEntryPoint(boost::function<void ()> const&)+0x37 [funapi_runner]
    boost::detail::thread_data<boost::function<void ()> >::run()+0x36 [funapi_runner]
    [unknown] [libboost_thread.so.1.65.1]
  3896 bytes in 158 allocations from stack
    operator new(unsigned long)+0x18 [libstdc++.so.6.0.25]
    [unknown] [funapi_runner]
    ... (중략)
    fun::ThreadEntryPoint(boost::function<void ()> const&)+0x37 [funapi_runner]
    boost::detail::thread_data<boost::function<void ()> >::run()+0x36 [funapi_runner]
    [unknown] [libboost_thread.so.1.65.1]
  22032 bytes in 162 allocations from stack
    operator new(unsigned long)+0x18 [libstdc++.so.6.0.25]
    ... (중략)
    boost::detail::thread_data<boost::function<void ()> >::run()+0x36 [funapi_runner]
    [unknown] [libboost_thread.so.1.65.1]
  1056964608 bytes in 126 allocations from stack
    operator new(unsigned long)+0x18 [libstdc++.so.6.0.25]
    hello::OnTick(unsigned long const&, boost::posix_time::ptime const&)+0x15 [libhello.so]
    boost::detail::function::void_function_invoker2... +0x43 [libhello.so]
    fun::OnAppTimerExpired(boost::shared_ptr<fun::AppTimerExpiredEvent const>)+0x49 [funapi_runner]
    ... (중략)
    boost::detail::thread_data<boost::function<void ()> >::run()+0x36 [funapi_runner]
    [unknown] [libboost_thread.so.1.65.1]

There is a function called OnTick, which allocated total of 1,056,964,608 bytes of memory for its 126 allocations. So you need to look after the OnTick function.

If you traced a release build, callstacks can be inaccurate. You need to guess the callstack.

45.5. CPU profiling

Note

This section deals with general CPU profiling. Please see Event profiling and debugging to learn more about profiling specific to iFun Engine.

Note

C# version profiling will be supported later.

You can find CPU bottlenecks using the following process even without rebuilding the game server.

45.5.1. Installing required programs

45.5.1.1. perf utility

Ubuntu

$ sudo apt-get install linux-tools-generic

Important

You need the linux-tools package when using non-generic kernels. i.e. linux-tools-<kernel name>.

CentOS

$ sudo yum install perf

45.5.1.2. FlameGraph

$ sudo wget -O /usr/local/bin/flamegraph.pl \
    https://raw.githubusercontent.com/brendangregg/FlameGraph/master/flamegraph.pl

$ sudo wget -O /usr/local/bin/stackcollapse-perf.pl \
    https://raw.githubusercontent.com/brendangregg/FlameGraph/master/stackcollapse-perf.pl

$ sudo chmod a+x /usr/local/bin/flamegraph.pl /usr/local/bin/stackcollapse-perf.pl

45.5.2. Get profiling results

Step 1. Run the game server. (Skip if already running)

$ ./hello-local

  1. Run the profiling command as follows:

    $ sudo /usr/bin/funapi_profile {{ProjectName}}

    Tip

    만약 Flavors: Identifying servers according to their role 를 쓰는 경우 아래처럼 flavor 를 지정합니다.

    $ sudo funapi_profile {{ProjectName}} {{FlavorName}}

    Example

    $ sudo /usr/bin/funapi_profile hello
Profiling server for activity (for 60 seconds)
[sudo] password for user:
[ perf record: Woken up 7 times to write data ]
[ perf record: Captured and wrote 1.626 MB perf.23428.data (~71057 samples) ]
Generating flamegraph
[kernel.kallsyms] with build id d7fccc433a6b698638325b755df2772c15636d04 not found, continuing without symbols
Generated: perf.hello_server.default.23428.svg

  2. Open the SVG file on the last output line in a browser. This is perf.hello_server.default.23428.svg in the example above.

    _images/profiling-sample.svg

In this example, when the hello::OnTick() function is the bottleneck, it is displayed horizontally in the flamegraph. You can find and optimize this function.

Tip

Click this function to expand the SVG file to show more details.

Tip

Hover over a function in the SVG file to show the CPU usage of that function and its sub-functions at the lower left.

Tip

Use the Search function in the upper right of the SVG file to easily see which parts of the project code use the CPU the most.

Assuming the project is hello, enter ^hello:: into Search to show only the project functions in purple.