44. 转储文件的调试和CPU性能分析¶

44.1. Debugging with GDB¶

You can use a launcher script generated through the build process. You may run the game server with GDB attached, you may attach a GDB to a running game server process.

For example, you may run like the following. (It assumes that the project is hello and the flavor is game.)

$ ./hello.game-local --gdb

If there is a running game server process, it will automatically attaches to the process. If not, it will run a new game server with attached GDB. For the latter case, you can start your debugging session by typing run command.

 Please type `run` to start debugging.
 If you want to check the arguments, please type `show args`.


 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
 Copyright (C) 2016 Free Software Foundation, Inc.
 ... (skipped)
 Reading symbols from /usr/bin/funapi_runner...(no debugging symbols found)...done.
 (gdb) run

Tip

Due to the security enforcment on linux operating system, you may not allowed to attach a debugger to the process ran by the same user.

You can use the sudo command as a workaround. Or you may change the system configuration by running the following command. It is worth noting that you must not run the following command on a production system.

$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

After rebooting the machine, the configuration will be reset to the default state.

If there are multiple game-server instances of the same flavor, you can choose the process by specifying the process id on the command line.

# Please substitute the {PID} with the process id.
$ ./hello.game-local --gdb --pid {PID}

44.2. 调试转储文件¶

iFun Engine在游戏服务器崩溃时，会运用 Google Breakpad ，在转储目录下生成(.dmp)文件。与核心转储(Core dump)相比，小型转储(Mini dump)会生成容量更小的转储文件。利用该文件，可以简单查看已崩溃的线程的堆栈。

44.2.1. 小型转储的生成位置¶

对于开发阶段，在Build目录的 logs/dumps/ 目录下保存相应文件。对于在live服务器中通过daemon运行游戏服务器时，转储文件保存在 /var/crash/funapi/{{project-name}}/ 目录中。

44.2.2. 查看小型转储文件内容¶

生成的小型转储文件可通过以下命令语句查看堆栈。

Note

以下说明是以在Live服务器中通过Daemon运行时为假设条件的。如果正处于开发阶段，将 /var/crash/funapi/{{project-name}}/ 更改成 logs/dumps/ 即可。

$ funapi_stackwalk /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp \
    /usr/share/{{project-name}}/symbols

以下是执行示例。

 $ funapi_stackwalk /var/crash/funapi/example/08835639-ff4e-e778-6c14c453-5147e752.dmp /usr/share/example/symbols
 OS Linux 0.0.0 Linux 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64
 CPU amd64 family 6 model 42 stepping 7 8
 Crash SIGSEGV 0x0 31

 0 libexample.so: /build/example-source/src/event_handlers.cc:155 (+0x4) - example::OnTick
 1 libexample.so: /usr/include/boost/function/function_template.hpp:112 (+0x14) - ...
 2 funapi_runner: ...
 ...
 7 libboost_thread-mt.so.1.53.0 (+0xd24a)
 8 libpthread-2.17.so (+0x7df5)
 9 libc-2.17.so (+0xf61ad)

44.2.3. 通过GDB调试小型转储¶

Note

以下说明是以在Live服务器中通过Daemon运行时为假设条件的。如果正处于开发阶段，将 /var/crash/funapi/{{project-name}}/ 更改成 logs/dumps/ 即可。

使用以下命令语句，即可通过GDB调试小型转储文件。

$ funapi_core_analyze /usr/lib/lib{{project-name}}.so \
    /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp

 $ funapi_core_analyze /usr/lib/libhello.so /var/crash/funapi/hello/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp
 Finding game server .so file
 Converting dumps/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp into coredump
 Launching gdb to inspect generated coredump (hello.23945.core).
 Please press 'y' to load required symbols

 ...

 Failed to read a valid object file image from memory.
 Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.game --app_flavor=game --'.
 #0  0x00007fc18d76c2de in ?? ()
 add symbol table from file "src/libhello.so" at
   .text_addr = 0x7fc18d0ceaf0
 Reading symbols from src/libhello.so...done.
 (gdb) bt
 #0  0x00007fc18d76c2de in hello::TestFilters () at ../src/filter.cc:367
 #1  0x00007fc18d43094f in (anonymous namespace)::HelloServer::Install (arguments=...) at ../src/hello_server.cc:113
 #2  0x00007fc18d430f18 in HelloServer_install (arguments=...) at ../src/hello_server.cc:182
 #3  0x00000000007328d3 in ?? ()
 #4  0x0000000000601b5c in ?? ()
 #5  0x00007fc1962d2ec5 in ?? ()
 #6  0x0000000000000000 in ?? ()
 (gdb) info locals
 x = 0x0
 (gdb)

44.2.4. 生成核心转储代替小型转储¶

对于小型转储，为了尽量减少硬盘的使用量，可以仅查看堆栈中的值和其中直接参考的值。当需要调试的信息比这还要多时，可保留核心转储来代替小型转储，对其进行调试。

Warning

核心转储是将进程所占用的所有内存全部保存到硬盘上，因此在占用内存较多时，硬盘使用量也随之增加。

Important

首先须要在OS设置中查看是否可以保存核心转储。

# 아래와 같이 0이 나오는 경우 코어덤프가 디스크에 남지 않습니다.
$ ulimit -c
0

# 남길 수 있는 코어덤프 용량 제한을 해제
$ ulimit -c unlimited

通过以下选项来运行launcher。

$ ./hello-local --enable_breakpad=false

当不更改OS设置时，崩溃后 core 文件将在当前目录下生成。该文件可按所下所示进行调试。

$ gdb /usr/bin/funapi_runner core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
...
Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.default --app_flavor=de'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
71      *x = 0;
[Current thread is 1 (Thread 0x7f287e5ec800 (LWP 21649))]
(gdb) bt
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
#1  0x00007f28715bad8a in HelloServer_start () at ../src/hello_server.cc:83
#2  0x0000000000845c03 in fun::StartComponents() ()
#3  0x000000000065eabe in main ()
(gdb)

44.3. CPU性能分析¶

Note

该会话用于处理一般的CPU性能分析。iFun引擎的专业性能分析请参考事件性能分析与调试。

Note

C# 版本的性能分析计划后续提供支持。

即使不重新构建游戏服务器，也可通过以下过程找出CPU 瓶颈。

44.3.1. 安装必要程序¶

44.3.1.1. `perf` Utility¶

Ubuntu

$ sudo apt-get install linux-tools-generic
Important

当使用Kernel，而不是generic时，需要与Kernel 相对应的linux-tools包。即， linux-tools-<Kernel名称> 。

CentOS

$ sudo yum install perf

44.3.1.2. FlameGraph ¶

$ sudo wget -O /usr/local/bin/flamegraph.pl \
    https://raw.githubusercontent.com/brendangregg/FlameGraph/master/flamegraph.pl

$ sudo wget -O /usr/local/bin/stackcollapse-perf.pl \
    https://raw.githubusercontent.com/brendangregg/FlameGraph/master/stackcollapse-perf.pl

$ sudo chmod a+x /usr/local/bin/flamegraph.pl /usr/local/bin/stackcollapse-perf.pl

44.3.2. 提取性能分析结果¶

Step 1.运行游戏服务器。(已运行的情况除外)

$ ./hello-local

2.执行如下性能分析命令。

$ sudo /usr/bin/funapi_profile {{ProjectName}}

Tip

如使用 Flavor: 按不同作用来区分服务器，需按如下所示，指定flavor。

$ sudo funapi_profile {{ProjectName}} {{FlavorName}}

示例

$ sudo /usr/bin/funapi_profile hello
Profiling server for activity (for 60 seconds)
[sudo] password for user:
[ perf record: Woken up 7 times to write data ]
[ perf record: Captured and wrote 1.626 MB perf.23428.data (~71057 samples) ]
Generating flamegraph
[kernel.kallsyms] with build id d7fccc433a6b698638325b755df2772c15636d04 not found, continuing without symbols
Generated: perf.hello_server.default.23428.svg

3.用浏览器打开最后一行中的SVG文件。上述示例中为 perf.hello_server.default.23428.svg 。

本示例属于 hello::OnTick() 函数为瓶颈的情况，在flamegraph中会以较长横向显示。找出该函数，对其进行优化即可。

Tip

SVG文件在点击相应函数后将扩大显示更具体的信息。

Tip

在SVG中，如将鼠标放在函数上，左下方将显示相应函数及子函数的CPU占用时间比率。

Tip

使用SVG文件右上方的 Search 功能，可轻松查看项目已使用的代码中哪个部分的CPU占用量最多。

如果 hello 项目在 Search 中输入 ^hello:: ，将用紫色仅显示项目的函数。

44. 转储文件的调试和CPU性能分析¶

44.1. Debugging with GDB¶

44.2. 调试转储文件¶

44.2.1. 小型转储的生成位置¶

44.2.2. 查看小型转储文件内容¶

44.2.3. 通过GDB调试小型转储¶

44.2.4. 生成核心转储代替小型转储¶

44.3. CPU性能分析¶

44.3.1. 安装必要程序¶

44.3.1.1. perf Utility¶

44.3.1.2. FlameGraph¶

44.3.2. 提取性能分析结果¶

44.3.1.1. `perf` Utility¶

44.3.1.2. FlameGraph ¶