원래 pyKdump 관련해서는 작성한지 몇일 됐었는데, 귀찮아서 한번에 쓰는 중이다.
이 pyKdump 자체도 상당히 강력하고 도움이 되는 Extension 이지만,
실제 내가 자주 쓰는 명령은 dis 명령으로써, 리버스 분석 명령인데,
이를 더욱 더 쉽게 도와주는 pyKdump 의 Extension 이 있어 소개하고자 한다.
바로 레드햇에서 근무하시는 커널박사님이신 권성주님의 PyCrashext 이다.
https://github.com/sungju/pycrashext
설치는 간단하므로 역시 해당 위키페이지를 살펴보고 설치하면 된다.
KERNEL: vmlinux
DUMPFILE: 127.0.0.1-2017-12-17-18:34:31/vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Sun Dec 17 18:34:01 2017
UPTIME: 97 days, 02:15:47
LOAD AVERAGE: 1.39, 1.62, 1.34
TASKS: 1190
NODENAME: ******
RELEASE: 4.1.12-94.3.8.el6uek.x86_64
VERSION: #2 SMP Fri Jun 30 11:00:28 PDT 2017
MACHINE: x86_64 (3492 Mhz)
MEMORY: 71.7 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000068"
PID: 2064
COMMAND: "kworker/3:3"
TASK: ffff881191d4f000 [THREAD_INFO: ffff880b736c4000]
CPU: 3
STATE: TASK_RUNNING (PANIC)
crash64> extend /usr/local/lib64/mpykdump64.so
Setting scroll off while initializing PyKdump
/usr/local/lib64/mpykdump64.so: shared object loaded
crash64> epython /root/pycrashext/regext.py
** Execution took 0.01s (real) 0.01s (CPU)
crash64>
요로코롬 .bash_profile 과 .crashXXrc 에 epython 으로 regext.py 를 로드하면
크래쉬 툴에 명령어가 추가된다.
crash64> help
* edis ipcs ptob syscallinfo
alias emodinfo irq ptov task
ascii epstree kmem rd taskinfo
auditinfo epython list repeat timeinfo
autocheck eval lockup revs timer
bh exit log runq traceinfo
bpf extend mach schedinfo tree
bt files meminfo scsi tslog
btop foreach mod scsishow union
cgroupinfo fregs mount search vm
cmds_test fsinfo net seinfo vmwareinfo
cpuinfo fuser netinfo set vtop
crashinfo gdb nfsshow sig waitq
dev hangcheck p struct whatis
devinfo hanginfo ps swap wr
dis help psinfo sym xportshow
dmshow insights pte sys q
굵게 표시된 부분들이 성주님이 만들어 넣은 커맨드셋인데
상당한 양임을 알 수 있다.
일단 특이한점은 레드햇에서 근무하고 계시는 성주님의 특성으로 특정 rule 셋을 바탕으로
autocheck 와 insight 명령을 통해 알려진 버그를 찾아준다는 점이다.
crash64> autocheck
No issues detected
** Execution took 2.69s (real) 2.69s (CPU)
crash64> autocheck --help
Usage: autocheck.py [options]
Options:
-h, --help show this help message and exit
-l, --list Shows the currently available rules
** Execution took 0.00s (real) 0.00s (CPU)
crash64> autocheck -l
---------------------------------------------------------------------------
[rules.hung_task_check]: Checking hung tasks
---------------------------------------------------------------------------
There are 1 rules available for this vmcore
===========================================================================
** Execution took 0.02s (real) 0.02s (CPU)
안타깝게도, autocheck 와 insights 는 레드햇 버그질라에 접근이 가능하거나
insight 사이트에 접근이 가능해야 유용한 기능이다.
이걸 차치하더라도 다양한 기능을 제공한다.
cpu 상태에 대한 내용을 아래와 같이 쉽게 얻을 수 있고,
crash64> cpuinfo -h
Usage: cpuinfo.py [options]
Options:
-h, --help show this help message and exit
-f, --cpufreq CPU frequency details
-i, --cpuid Show CPU's physical and core ID
-t, --tlb Show CPU tlb state
** Execution took 0.00s (real) 0.00s (CPU)
crash64> cpuinfo -f
CPU 0 (0xffff881222542600) min = 1200000, max = 3700000, cur = 3492151
cpudata = 0xffff8812224bab00, current_pstate = 13, turbo_pstate = 37,
min_pstate = 12, max_pstate = 35, policy = CPUFREQ_POLICY_POWERSAVE
sample.freq = 1254941,
updated 56 ns (0 sec) ago
CPU 1 (0xffff881222542800) min = 1200000, max = 3700000, cur = 3492151
cpudata = 0xffff8812224bac00, current_pstate = 13, turbo_pstate = 37,
min_pstate = 12, max_pstate = 35, policy = CPUFREQ_POLICY_POWERSAVE
sample.freq = 1235390,
updated 56 ns (0 sec) ago
CPU 2 (0xffff881222542a00) min = 1200000, max = 3700000, cur = 3492151
cpudata = 0xffff8812224bad00, current_pstate = 13, turbo_pstate = 37,
min_pstate = 12, max_pstate = 35, policy = CPUFREQ_POLICY_POWERSAVE
sample.freq = 1255351,
updated 56 ns (0 sec) ago
CPU 3 (0xffff881222542c00) min = 1200000, max = 3700000, cur = 3492151
cpudata = 0xffff8812224bae00, current_pstate = 13, turbo_pstate = 37,
min_pstate = 12, max_pstate = 35, policy = CPUFREQ_POLICY_POWERSAVE
sample.freq = 1199980,
updated 56 ns (0 sec) ago
** Execution took 0.27s (real) 0.26s (CPU)
crash64> cpuinfo -i
<<< Physical CPU 0 >>>
CPU 0, core 0 : 0xffff88127fc0a040 Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
CPU 1, core 1 : 0xffff88127fc8a040 Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
<<< Physical CPU 1 >>>
CPU 2, core 0 : 0xffff88127fd0a040 Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
CPU 3, core 1 : 0xffff88127fd8a040 Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
For details, run 'cpuinfo_x86 <address>'
** Execution took 0.00s (real) 0.00s (CPU)
crash64> cpuinfo -t
CPU 0 : state = 2 [TLBSTATE_LAZY], active_mm = 0xffff8812070a8000 (oracle)
CPU 1 : state = 2 [TLBSTATE_LAZY], active_mm = 0xffff8812225fcb00 (ocssd.bin)
CPU 2 : state = 2 [TLBSTATE_LAZY], active_mm = 0xffff8811c3ef3c00 (crsd.bin)
CPU 3 : state = 2 [TLBSTATE_LAZY], active_mm = 0xffff8812070af800 (oracle)
** Execution took 0.03s (real) 0.03s (CPU)
CPU tlb status 보여주는것 보이는가!!
메모리 상태도 훨씬 더 잘 보여준다.
crash64> meminfo -h
Usage: meminfo.py [options]
Options:
-h, --help show this help message and exit
-u, --memusage Show memory usages by tasks
-n, --nogroup Show data in individual tasks
-a, --all Show all the output
-s, --slabtop Show slabtop-like output
-S SLABDETAIL, --slabdetail=SLABDETAIL
Show details of a slab
-i, --meminfo Show /proc/meminfo-like output
-p PERCPU, --percpu=PERCPU
Convert percpu address into virtual address
-t PERCPU_TYPE, --type=PERCPU_TYPE
Specify percpu type : u8, u16, u32, u64, s8, s16, s32, s64, int
-d, --details Show detailed output
-v, --vm Show 'vm' output with more details
-e ERROR_CODE, --error=ERROR_CODE
Interpret page_fault error code
** Execution took 0.00s (real) 0.00s (CPU)
crash64> meminfo -u
======================================================================
[ RSS usage ] [ Process name ]
======================================================================
65 GiB ( 68961924 KiB) oracle
545 MiB ( 558888 KiB) java
154 MiB ( 158336 KiB) ologgerd
152 MiB ( 155912 KiB) oraagent.bin
136 MiB ( 139460 KiB) crsd.bin
128 MiB ( 131452 KiB) ocssd.bin
104 MiB ( 106576 KiB) osysmond.bin
94 MiB ( 96352 KiB) orarootagent.bi
93 MiB ( 96208 KiB) cssdagent
93 MiB ( 95680 KiB) cssdmonitor
======================================================================
Total memory usage from user-space = 68.54 GiB
** Execution took 0.03s (real) 0.03s (CPU)
crash64> meminfo -s
====================================================================
kmem_cache NAME TOTAL OBJSIZE
====================================================================
0xffff88127e889c00 buffer_head 752812 K 104
0xffff88121e8c9200 ext4_inode_cache 237376 K 1112
0xffff88127e803f00 radix_tree_node 237312 K 576
0xffff88127e889e00 dentry 146756 K 192
0xffff88122a25ae00 ext4_extent_status 18920 K 40
0xffff88127e801b00 kmalloc-64 12036 K 64
0xffff88122aa7d400 proc_inode_cache 11312 K 600
0xffff88127e801600 kmalloc-512 10992 K 512
0xffff88127e889f00 inode_cache 9504 K 544
0xffff88127e889b00 vm_area_struct 9068 K 200
====================================================================
** Execution took 0.37s (real) 0.37s (CPU)
crash64> meminfo -v
PID: 2064 TASK: ffff881191d4f000 CPU: 3 COMMAND: "kworker/3:3"
MM PGD RSS TOTAL_VM
0 0 0k 0k
** Execution took 0.00s (real) 0.00s (CPU)
crash64> meminfo -i
MemTotal: 73645076 kB
MemFree: 3573856 kB
MemAvailable: 20948876 kB
Buffers: 1256148 kB
Cached: 61856232 kB
SwapCached: 875 kB
Active: 11171682 kB
Inactive: 5587827 kB
Active(anon): 4256688 kB
Inactive(anon): 331211 kB
Active(file): 6914994 kB
Inactive(file): 5256616 kB
Unevictable: 89004 kB
Mlocked: 89005 kB
SwapTotal: 16777212 kB
SwapFree: 16711872 kB
Dirty: 1466 kB
Writeback: 0 kB
AnonPages: 4282656 kB
Mapped: 14429048 kB
Shmem: 14350532 kB
Slab: 1516736 kB
SReclaimable: 1418400 kB
SUnreclaim: 98336 kB
KernelStack: 4756 kB
PageTables: 800684 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 13399938 kB
Committed_AS: 5643195 kB
VmallocTotal: 60471536787456 kB
VmallocUsed: 508719104 kB
VmallocChunk: 35183562584064 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap2M: 111648768 kB
DirectMap4M: 49242112 kB
** Execution took 3.48s (real) 3.48s (CPU)
User space 의 메모리사용량을 보여주는 부분에서 감탄하시는 분 있을거라 생각한다.
흔히들 많이 물어보는 내용이지만 메모리를 직접 검색해가야 하는 번거로움으로 인해
그냥 알수 없음으로 대답하곤 하는데 말이다... ( 고백한다 ㅋㅋㅋ 귀찮은 작업이라 그랬다 )
Asm 의 레지스터 및 인스트럭션에 대한 설명도 간략하게 제공한다.
crash64> revs
** function parameters for x86_64 **
%rdi - 1st argument (%rdi:64, %edi:32, %di:16, %dl:8)
%rsi - 2nd argument (%rsi:64, %esi:32, %si:16, %sl:8)
%rdx - 3rd argument (%rdx:64, %edx:32, %dx:16, %dl:8)
%rcx - 4th argument (%rcx:64, %ecx:32, %cx:16, %cl:8)
%r8 - 5th argument (%r8:64, %r8d:32, %r8w:16, %r8b:8)
%r9 - 6th argument (%r9:64, %r9d:32, %r9w:16, %r9b:8)
%rsp - Stack pointer
%rax - Return value
%rip - Instruction pointer
** Execution took 0.00s (real) 0.01s (CPU)
프로세스 트리 역시 향상된 트리형태로 색까지 입혀서 보여준다.
crash64> epstree -h
Usage: epstree.py [options]
Options:
-h, --help show this help message and exit
-p Print process ID
-g Print number of threads
-s Print task state
-t TASK_ID Print specific task and its children
** Execution took 0.00s (real) 0.00s (CPU)
crash64> epstree
swapper/0 -+- init -+- udevd -+- udevd
| | `- udevd
| |- init.tfa -+- sleep
| |- init.ohasd
| |- multipathd
| |- java
| |- auditd
| |- portreserve
| |- rsyslogd
| |- irqbalance
| |- rpcbind
| |- pbx_exchange
| |- dbus-daemon
| |- rpc.statd
| |- hald -+- hald-runner -+- hald-addon-acpi
| | `- hald-addon-inpu
| |- automount
| |- ntpd
| |- master -+- qmgr
시스템 콜 상태도 확인이 가능하다.
crash64> syscallinfo -c
2 ffffffffa04a95a0 (t) my_open [seos]
4 ffffffffa04a8260 (t) my_stat [seos]
5 ffffffffa04a8620 (t) my_fstat [seos]
6 ffffffffa04a8350 (t) my_lstat [seos]
42 ffffffffa04a07e0 (t) my_connect [seos]
43 ffffffffa04a0580 (t) my_accept [seos]
59 ffffffff816e51d0 (T) stub_execve arch/x86/kernel/entry_64.S: 509
callq 0xffffffffa04a88b0 <my_execve>
60 ffffffffa04a0410 (t) my_exit [seos]
62 ffffffffa04a3fe0 (t) my_kill [seos]
76 ffffffffa04aacf0 (t) my_truncate [seos]
80 ffffffffa04aafb0 (t) my_chdir [seos]
81 ffffffffa04ab0a0 (t) my_fchdir [seos]
82 ffffffffa04aa9e0 (t) my_rename [seos]
83 ffffffffa04aa820 (t) my_mkdir [seos]
84 ffffffffa04aa680 (t) my_rmdir [seos]
85 ffffffffa04a97a0 (t) my_creat [seos]
86 ffffffffa04a98f0 (t) my_link [seos]
87 ffffffffa04a9ca0 (t) my_unlink [seos]
88 ffffffffa04a9b20 (t) my_symlink [seos]
90 ffffffffa04a9fd0 (t) my_chmod [seos]
91 ffffffffa04aade0 (t) my_fchmod [seos]
92 ffffffffa04aa0c0 (t) my_chown [seos]
93 ffffffffa04aaec0 (t) my_fchown [seos]
94 ffffffffa04aa1c0 (t) my_lchown [seos]
101 ffffffffa04a0ed0 (t) my_ptrace [seos]
105 ffffffffa049e650 (t) my_setuid [seos]
106 ffffffffa049ee10 (t) my_setgid [seos]
113 ffffffffa049f990 (t) my_setreuid [seos]
114 ffffffffa049f390 (t) my_setregid [seos]
116 ffffffffa04a0230 (t) my_setgroups [seos]
117 ffffffffa049fdd0 (t) my_setresuid [seos]
119 ffffffffa049f500 (t) my_setresgid [seos]
122 ffffffffa049ea10 (t) my_setfsuid [seos]
123 ffffffffa049f0d0 (t) my_setfsgid [seos]
129 ffffffffa04a4190 (t) my_rt_sigqueueinfo [seos]
132 ffffffffa04aa2c0 (t) my_utime [seos]
133 ffffffffa04a9e00 (t) my_mknod [seos]
165 ffffffffa04a0f90 (t) my_mount [seos]
166 ffffffffa04a1170 (t) my_umount [seos]
169 ffffffffa049e520 (t) my_reboot [seos]
175 ffffffffa04a12c0 (t) my_init_module [seos]
176 ffffffffa04a1620 (t) my_delete_module [seos]
188 ffffffffa04accb0 (t) my_setxattr [seos]
189 ffffffffa04acdb0 (t) my_lsetxattr [seos]
190 ffffffffa04acbb0 (t) my_fsetxattr [seos]
197 ffffffffa04aceb0 (t) my_removexattr [seos]
198 ffffffffa04ad060 (t) my_lremovexattr [seos]
199 ffffffffa04acf90 (t) my_fremovexattr [seos]
231 ffffffffa04a04a0 (t) my_exit_group [seos]
235 ffffffffa04aa3b0 (t) my_utimes [seos]
257 ffffffffa04abcc0 (t) my_openat [seos]
258 ffffffffa04abed0 (t) my_mkdirat [seos]
259 ffffffffa04ac090 (t) my_mknodat [seos]
260 ffffffffa04ac260 (t) my_fchownat [seos]
261 ffffffffa04abbd0 (t) my_futimesat [seos]
263 ffffffffa04ac370 (t) my_unlinkat [seos]
264 ffffffffa04ac4f0 (t) my_renameat [seos]
265 ffffffffa04ac710 (t) my_linkat [seos]
266 ffffffffa04ac920 (t) my_symlinkat [seos]
268 ffffffffa04acaa0 (t) my_fchmodat [seos]
280 ffffffffa04ab9e0 (t) my_utimensat [seos]
288 ffffffffa04a06b0 (t) my_accept4 [seos]
322 ffffffff816e51f8 (T) stub_execveat arch/x86/kernel/entry_64.S: 532
callq 0xffffffffa04a8de0 <my_execveat>
===========================================================================
61 system calls were replaced
2 system calls were modified
** Execution took 6.46s (real) 6.44s (CPU)
CPU lock 정보도 확인할 수 있다.
crash64> lockup -h
Usage: lockup.py [options]
Options:
-h, --help show this help message and exit
-r, --reverse show longest holder at top
-t, --tasks show tasks in each runqueue
-s, --rt show RT statistics
-d, --details show task details
** Execution took 0.00s (real) 0.01s (CPU)
crash64> lockup -r
CPU 0: 0.00 sec behind by 0xffffffff81ab54e0, swapper/0 [N:120] (0 in queue)
CPU 3: 0.00 sec behind by 0xffff881191d4f000, kworker/3:3 [N:120] (1 in queue)
CPU 1: 0.00 sec behind by 0xffff88122a7aaa00, swapper/1 [N:120] (0 in queue)
CPU 2: 0.00 sec behind by 0xffff88122a7ab800, swapper/2 [N:120] (0 in queue)
무엇보다 사실 권성주샘의 익스텐션을 사용해야 하는 가장 큰 이유는 바로!
edis 이다. dis 명령의 향상판이라고 보면 되는데, 색깔도 이쁘게 넣어서
다소 어지러운 Deasm 코드를 보기 쉽게, 또 스택및 펑션에 대한 주소도
추가로 각주를 달아줄 뿐 아니라, Jmp 문등의 분기 발생 시,
색깔과 기호로 pair 를 표시해주어 손쉽게 따라갈 수 있도록 해준다!!
crash64> edis -h
Usage: edis.py [options]
Options:
-h, --help show this help message and exit
-r, --reverse displays all instructions from the start of the routine up to and including the designated address.
-l, --list Dummy argument to match with 'dis -l'
-S STACKADDR, --stack=STACKADDR
Set stack address for disasm operation
-g, --graph display jump graph on the left
-f, --full Dispaly full function code
-b, --symbol Translate symbols if possible
-j JUMP_OP_LIST, --jump=JUMP_OP_LIST
Shows graph for the specified jump operations only
-n, --noaction Only colorising the output and not connection to server
-s, --sourceonly Display source lines only, but not based on function
-c, --callgraph Shows call graph for a function
-m MAX_DEPTH, --max_depth=MAX_DEPTH
Maximum depth of graph for callgraph. default=2
** Execution took 0.00s (real) 0.01s (CPU)
crash64> edis -f ffffffff81288f49
CRASHEXT_SERVER environment variable not configured
/usr/src/debug/kernel-4.1.12/linux-4.1.12-94.3.8.el6uek/fs/kernfs/dir.c: 644
0xffffffff81288f30 <kernfs_find_ns>: push %rbp ; 0xffff880b736c7c08
0xffffffff81288f31 <kernfs_find_ns+1>: mov %rsp,%rbp
0xffffffff81288f34 <kernfs_find_ns+4>: sub $0x30,%rsp
0xffffffff81288f38 <kernfs_find_ns+8>: mov %rbx,-0x18(%rbp) ; 0x0000000000000000
0xffffffff81288f3c <kernfs_find_ns+12>: mov %r12,-0x10(%rbp) ; 0xffffffff81771ce0
0xffffffff81288f40 <kernfs_find_ns+16>: mov %r13,-0x8(%rbp) ; 0x0000000000000000
0xffffffff81288f44 <kernfs_find_ns+20>: nopl 0x0(%rax,%rax,1)
/usr/src/debug/kernel-4.1.12/linux-4.1.12-94.3.8.el6uek/include/linux/kernfs.h: 265
0xffffffff81288f49 <kernfs_find_ns+25>: movzwl 0x68(%rdi),%eax
** Execution took 0.08s (real) 0.02s (CPU), Child processes: 0.06s
보았는가? 인스트럭션에 색칠과 밑줄도 해주고, Null point 가 들어가는 부분까지
스택주소 및 16진 값으로 알려주는 부분!! 엄청 유용한 기능임을 알 수 있을것이다.
crash64> edis -c fc_starget_delete
{fc_starget_delete} -+- {fc_terminate_rport_io} -+- {scsi_is_host_device} ...
| |- {*%rax} ...
| `- {scsi_target_unblock} ...
`- {scsi_remove_target} -+- {scsi_is_host_device} ...
|- {_raw_spin_lock_irqsave} ...
|- {_raw_spin_unlock_irqrestore} ...
|- {_raw_spin_unlock_irqrestore} ...
|- {__scsi_remove_target} ...
|- {scsi_target_reap} ...
`- {warn_slowpath_null} ...
멋지지 않은가 ? (라는 말이 자꾸 나오지만 정말이다.) 이렇게 트리형태로도 보여주고,
아래 스샷처럼 화려하게 그렇지만 보기 쉽게 페어로 묶어준기도 한다.
클릭하면 더 자세히 볼 수 있다.
저 화려한 색으로 묶여있는 분기별 라인들이 보이는가?? 감동의 눈물이 흐른다..
소스코드까지 적절한 위치에 있을 경우, 이렇게 보여주기도 한다.
(현재 내 시스템에서는 이렇게 보여주지 않아서 손보고 있는 중이다.)
crash> edis -r ffffffff812461ec /usr/src/debug/kernel-3.10.0-693.11.6.el7/linux-3.10.0-693.11.6.el7.x86_64/fs/proc_namespace.c: 42 41 static int show_sb_opts(struct seq_file *m, struct super_block *sb) 42 { 0xffffffff81246190 <show_sb_opts>:show_sb_optsdata32 data32 data32 xchg %ax,%ax [FTRACE NOP] ... /usr/src/debug/kernel-3.10.0-693.11.6.el7/linux-3.10.0-693.11.6.el7.x86_64/fs/proc_namespace.c: 51 51 for (fs_infop = fs_info; fs_infop->flag; fs_infop++) { 0xffffffff812461da <show_sb_opts+0x4a>:0x4amovslq (%rbx),%rax 0xffffffff812461dd <show_sb_opts+0x4d>:0x4dtest %eax,%eax 0xffffffff812461df <show_sb_opts+0x4f>:0x4fjne 0xffffffff812461c3 <show_sb_opts+0x33> /usr/src/debug/kernel-3.10.0-693.11.6.el7/linux-3.10.0-693.11.6.el7.x86_64/fs/proc_namespace.c: 56 56 return security_sb_show_options(m, sb); 0xffffffff812461e1 <show_sb_opts+0x51>:0x51mov %r12,%rsi 0xffffffff812461e4 <show_sb_opts+0x54>:0x54mov %r13,%rdi 0xffffffff812461e7 <show_sb_opts+0x57>:0x57callq 0xffffffff812b3c70 <security_sb_show_options> /usr/src/debug/kernel-3.10.0-693.11.6.el7/linux-3.10.0-693.11.6.el7.x86_64/fs/proc_namespace.c: 57 57 } 0xffffffff812461ec <show_sb_opts+0x5c>:0x5cpop %rbx
정말 멋지지 않은가!
본문 글에 반영된 색들은 실제로 해당 기능을 사용시 보여주는 색을 그대로 표현한것이다.
사실 본문에 설명한역시 능력자님께서 손수 만들어 주셨다는 점이다.
다만 소스코드 보여주는 역할등은 일부 시스템(파이선 영향인듯) 에서는
완벽하게 돌아가지 않는것 같다.
아래와 같이 간단한 패치를 했다.( 성주샘에게 메일도 보냈으나 반영될지는 모르겠다.. 안되도 그냥 이렇게 수정해 쓰자 ㅋㅋ)
Report 는 반영되었고, 아래와 같이 변경되었다.
---
-----
일단 이것은 edis 뒤에 주소가 인자로 주어지지 않았을 경우 파이선 에러가 출력되는 것을
막고 주소를 넣어달라고 안내하는 것이며,
encode_url 이 Null 일 경우 noaction 옵션이 추가되고 해당 액션이 수행되어야 하는데,
제대로 체크가 되지 않아 보다 직관적으로 none 을 확인하도록 수정해 준 약소한 부분이다.
현재 코드트리가 제대로 출력되지 않는 문제는 아직 손보고 있다.
어찌됐든, 상당히 유용하고 도움이 엄청 되는 확장플러그인이므로,
가능하다면 앞으로 나도 pyKdump 와 pyCrashext 의 향상에 이바지를 하고 싶다.
이제 정말..... 난 먹고살길이 막막해 진것 같다.
오늘을 마지막으로 더이상의 커널 덤프 분석 공유 글은 중단하려고 한다.
그동안 나름대로 잘 읽어봐준 분들께 감사의 인사를 드리고,
다른 더 좋은 주제로 글을 쓸 수 있도록 노력하겠다.
'Skills > mY Technutz' 카테고리의 다른 글
| PyKdump extension - pycrashext (0) | 2019.12.14 |
|---|---|
| The effective crash-utility for vmcore analysis (PyKdump) (0) | 2019.12.13 |
| kernel Dump Analysis #18 (0) | 2019.05.11 |
| Kernel Dump Analysis #17 (0) | 2019.04.04 |
| libfc: Update rport reference counting bug - 1368175 (0) | 2018.03.29 |
| Kernel Dump Analysis #16 (0) | 2018.02.22 |
Trackback :
요즘 다시 혼자서 근무하다보니, 업무량이 훅 늘어나고
분석에 대해서는 반복적인 부분이 많이 발생하는데 그에 대한 시간은 여전히 동일하게 들어서
간편화 하는 방법이 없을까 해서 간단히 python 으로 crash utility 에 대한 extension 을 직접 만들고 있었는데,
하늘아래 새로운 것은 없듯이 역시나 이미 만들어져 있었다는 것을 발견하였다.
그것은 바로 mPyKdump 라는 crash extension 이다.
https://sourceforge.net/p/pykdump/wiki/Home/
일단 커널 코어덤프 분석을 위한 도구인 crash 툴은 C/Python 형태의 외부 스크립트를
내부에서 불러와 사용할 수 있다.
대부분 c 로 컴파일되어 모듈형태로 crash 툴이 실행된 후 로드하는 형태로 사용되는데
mPyKdump 는 파이선 기준으로 작성되어 내용을 수정하더라도 특별히 컴파일이 필요없이
바로바로 적용이 가능하다는 장점이 있었다.
해당 모듈을 이용하여 얼마나 간편하게 기존 삽질을 줄일 수 있는지 확인해보자.
(Host정보 삭제함)
설치는 매우 간단하며 해당 git 에 설명되어 있으므로 생략하겠다.
크게 crashinfo 와 xportshow 기능을 제공하는데, crashinfo 에 대해서 조금 살펴보겠다.
crash 를 수행하고 help 를 통해 추가된 명령어를 확인해 보자
crash64> extend /usr/local/lib64/mpykdump64.so
Setting scroll off while initializing PyKdump
/usr/local/lib64/mpykdump64.so: shared object loaded
crash64> help
* extend mach scsi tslog
alias files mod scsishow union
ascii foreach mount search vm
bpf fregs net set vtop
bt fuser nfsshow sig waitq
btop gdb p struct whatis
crashinfo hanginfo ps swap wr
dev help pte sym xportshow
dis ipcs ptob sys q
dmshow irq ptov task
epython kmem rd taskinfo
eval list repeat timer
exit log runq tree
crashinfo/dmshow/fregs/scsishow/tslog/nfsshow/hanginfo/taskinfo/dmshow/epython 등
몇가지 명령어들이 추가된 것을 확인 할 수 있다.
crashinfo 를 살펴보자
crash64> crashinfo -h
Usage: crashinfo [options]
Options:
-h, --help show this help message and exit
-v verbose output
-q quiet mode - print warnings only
--fast Fast mode - do not run potentially slow tests
--sysctl Print sysctl info.
--ext3 Print EXT3 info.
--blkreq Print Block I/O requests
--blkdevs Print Block Devices Info
--filelock Print filelock info.
--stacksummary Print stacks (bt) categorized summary.
--findstacks=FINDSTACKS
Print stacks (bt) containing functions that match the provided pattern
--checkstacks Check stacks of all threads for corruption
--decodesyscalls=DECODESYSCALLS
Decode Syscalls on the Stack
--keventd_wq Decode keventd_wq
--kblockd_wq Decode kblockd_workqueue
--lws Print Locks Waitqueues and Semaphores
--devmapper Print DeviceMapper Tables
--runq Print Runqueus
--semaphore=SEMA Print 'struct semaphore' info
--rwsemaphore=RWSEMA Print 'struct rw_semaphore' info
--mutex=MUTEX Print Mutex info
--umem Print User-space Memory Usage
--ls=LS Emulate 'ls'. You can specify either dentry address or full pathname
--workqueues Print Workqueues - just for some kernels
--radix_tree_element=root offset
Find and print a radix tree element
--pci Print PCI Info
--version Print program version and exit
** Execution took 1.97s (real) 1.89s (CPU)
요기 도움말에 나오는 옵션들만 봐도 눈이 확 커질것이다.
기존에는 직접 메모리주소를 찾아 찾아가며 확인해야 했던 것들인데
이걸 보면 한방에 지원해 준다는 것을 알 수 있기 때문이다.
일단 그냥 기본으로 수행해 보자.
crash64> crashinfo
************************ crashinfo *************************
/ovs1disk/nfs_local/vmcore/OS_var_crash_File/******/127.0.0.1-2017-12-17-18:34:31/vmcore (4.1.12-94.3.8.el6uek.x86_64)
+==========================+
| *** Crashinfo v1.3.4 *** |
+==========================+
+++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM)
KERNEL: vmlinux
DUMPFILE: 127.0.0.1-2017-12-17-18:34:31/vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Sun Dec 17 18:34:01 2017
UPTIME: 97 days, 02:15:47
LOAD AVERAGE: 1.39, 1.62, 1.34
TASKS: 1190
NODENAME: ******
RELEASE: 4.1.12-94.3.8.el6uek.x86_64
VERSION: #2 SMP Fri Jun 30 11:00:28 PDT 2017
MACHINE: x86_64 (3492 Mhz)
MEMORY: 71.7 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000068"
+--------------------------+
>------------------------| Per-cpu Stacks ('bt -a') |------------------------<
+--------------------------+
-- CPU#0 --
PID=0 CPU=0 CMD=swapper/0
#0 crash_nmi_callback+0x38
#1 nmi_handle+0x87
#2 default_do_nmi+0x5e
#3 do_nmi+0xf5
#4 end_repeat_nmi+0x1a
#-1 intel_idle+0xb4, 507 bytes of data
#5 intel_idle+0xb4
#6 cpuidle_enter_state+0x8d
#7 cpuidle_enter+0x17
#8 cpuidle_idle_call+0xe4
#9 cpu_idle_loop+0x1f5
#10 cpu_startup_entry+0x5f
#11 rest_init+0x7c
#12 start_kernel+0x427
#13 x86_64_start_reservations+0x2a
#14 x86_64_start_kernel+0x19c
-- CPU#1 --
PID=0 CPU=1 CMD=swapper/1
#0 crash_nmi_callback+0x38
#1 nmi_handle+0x87
#2 default_do_nmi+0x5e
#3 do_nmi+0xf5
#4 end_repeat_nmi+0x1a
#-1 intel_idle+0xb4, 507 bytes of data
#5 intel_idle+0xb4
#6 cpuidle_enter_state+0x8d
#7 cpuidle_enter+0x17
#8 cpuidle_idle_call+0xe4
#9 cpu_idle_loop+0x1f5
#10 cpu_startup_entry+0x5f
#11 start_secondary+0xbb
-- CPU#2 --
PID=0 CPU=2 CMD=swapper/2
#0 crash_nmi_callback+0x38
#1 nmi_handle+0x87
#2 default_do_nmi+0x5e
#3 do_nmi+0xf5
#4 end_repeat_nmi+0x1a
#-1 intel_idle+0xb4, 507 bytes of data
#5 intel_idle+0xb4
#6 cpuidle_enter_state+0x8d
#7 cpuidle_enter+0x17
#8 cpuidle_idle_call+0xe4
#9 cpu_idle_loop+0x1f5
#10 cpu_startup_entry+0x5f
#11 start_secondary+0xbb
-- CPU#3 --
PID=2064 CPU=3 CMD=kworker/3:3
#0 machine_kexec+0x1e0
#1 crash_kexec+0x68
#2 oops_end+0xe8
#3 no_context+0x161
#4 __bad_area_nosemaphore+0x12d
#5 bad_area_nosemaphore+0x13
#6 __do_page_fault+0x328
#7 do_page_fault+0x37
#8 page_fault+0x28
#-1 kernfs_find_ns+0x19, 477 bytes of data
#9 kernfs_find_and_get_ns+0x3c
#10 sysfs_unmerge_group+0x1d
#11 dpm_sysfs_remove+0x2c
#12 device_del+0x58
#13 attribute_container_class_device_del+0x1e
#14 transport_remove_classdev+0x59
#15 attribute_container_device_trigger+0xa0
#16 transport_remove_device+0x15
#17 scsi_target_reap_ref_release+0x32
#18 scsi_target_reap+0x2c
#19 scsi_remove_target+0xeb
#20 fc_starget_delete+0x26
#21 process_one_work+0x151
#22 worker_thread+0x120
#23 kthread+0xce
#24 ret_from_fork+0x42
+--------------------------------+
>---------------------| How This Dump Has Been Created |---------------------<
+--------------------------------+
+---------------+
>------------------------------| Tasks Summary |------------------------------<
+---------------+
Number of Threads That Ran Recently
-----------------------------------
last second 232
last 5s 357
last 60s 514
----- Total Numbers of Threads per State ------
TASK_INTERRUPTIBLE 1185
TASK_RUNNING 2
+++WARNING+++ There are 1 threads running in their own namespaces
Use 'taskinfo --ns' to get more details
+-----------------------+
>--------------------------| 5 Most Recent Threads |--------------------------<
+-----------------------+
PID CMD Age ARGS
----- -------------- ------ ----------------------------
7 rcu_sched 0 ms (no user stack)
10121 crsd.bin 0 ms (no user stack)
8736 ocssd.bin 0 ms (no user stack)
2064 kworker/3:3 0 ms (no user stack)
9313 ocssd.bin 0 ms (no user stack)
+------------------------+
>-------------------------| Memory Usage (kmem -i) |-------------------------<
+------------------------+
PAGES TOTAL PERCENTAGE
TOTAL MEM 18411269 70.2 GB ----
FREE 893464 3.4 GB 4% of TOTAL MEM
USED 17517805 66.8 GB 95% of TOTAL MEM
SHARED 10688869 40.8 GB 58% of TOTAL MEM
BUFFERS 314037 1.2 GB 1% of TOTAL MEM
CACHED 15464058 59 GB 83% of TOTAL MEM
SLAB 379184 1.4 GB 2% of TOTAL MEM
TOTAL HUGE 0 0 ----
HUGE FREE 0 0 0% of TOTAL HUGE
TOTAL SWAP 4194303 16 GB ----
SWAP USED 16335 63.8 MB 0% of TOTAL SWAP
SWAP FREE 4177968 15.9 GB 99% of TOTAL SWAP
COMMIT LIMIT 13399937 51.1 GB ----
COMMITTED 5643195 21.5 GB 42% of TOTAL LIMIT
+-------------------------------+
>----------------------| Scheduler Runqueues (per CPU) |----------------------<
+-------------------------------+
---+ CPU=0 <struct rq 0xffff88127fc17640> ----
| CURRENT TASK <struct task_struct 0xffffffff81ab54e0>, CMD=swapper/0
---+ CPU=1 <struct rq 0xffff88127fc97640> ----
| CURRENT TASK <struct task_struct 0xffff88122a7aaa00>, CMD=swapper/1
---+ CPU=2 <struct rq 0xffff88127fd17640> ----
| CURRENT TASK <struct task_struct 0xffff88122a7ab800>, CMD=swapper/2
---+ CPU=3 <struct rq 0xffff88127fd97640> ----
| CURRENT TASK <struct task_struct 0xffff881191d4f000>, CMD=kworker/3:3
+------------------------+
>-------------------------| Network Status Summary |-------------------------<
+------------------------+
TCP Connection Info
-------------------
ESTABLISHED 117
TIME_WAIT 92
LISTEN 40
NAGLE disabled (TCP_NODELAY): 90
UDP Connection Info
-------------------
550 UDP sockets, 0 in ESTABLISHED
+++WARNING+++ UDP buffer fill >=75% rcv=1 snd=0
Unix Connection Info
------------------------
ESTABLISHED 1626
CLOSE 32
LISTEN 89
Raw sockets info
--------------------
None
Interfaces Info
---------------
How long ago (in seconds) interfaces trasmitted/received?
Name RX TX
---- ---------- ---------
lo n/a n/a
eth8 n/a n/a
eth9 n/a n/a
eth10 n/a n/a
eth11 n/a n/a
eth0 n/a 1.2
eth1 n/a n/a
eth2 n/a n/a
eth3 n/a n/a
eth6 n/a 0.0
eth7 n/a n/a
eth4 n/a n/a
eth5 n/a n/a
bond0 n/a n/a
bond1 n/a n/a
RSS_TOTAL=71867908 pages, %mem= 93.4
+------------+
>-------------------------------| Mounted FS |-------------------------------<
+------------+
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff88122a69c000 ffff88127e810800 rootfs rootfs /
ffff88122a36cc00 ffff88127e815000 proc proc /proc
ffff88122a36cd80 ffff88122a327000 sysfs sysfs /sys
ffff88122a36cf00 ffff88122a320800 devtmpfs devtmpfs /dev
ffff88122a36d080 ffff88122a322800 devpts devpts /dev/pts
ffff88122a36d200 ffff88122a327800 tmpfs tmpfs /dev/shm
ffff88122a36d380 ffff88121f8cc000 ext4 /dev/mapper/vg00-lvroot /
ffff88122a69cc00 ffff881223114800 ext4 /dev/mapper/vg00-lvadmin /Admin
ffff881229af4780 ffff88121ffe6800 ext4 /dev/sda1 /boot
ffff881229af4480 ffff8812256b0000 ext4 /dev/mapper/vg00-lvhome /home
ffff881229af4900 ffff8812256b1800 ext4 /dev/mapper/vg00-lvtmp /tmp
ffff88122a69cf00 ffff881223117800 ext4 /dev/mapper/vg00-lvvar /var
ffff88122a69d080 ffff8812267e8800 ext4 /dev/mapper/vg00-lvcrash /var/crash
ffff881229af4a80 ffff8812256b3000 ext4 /dev/mapper/vg00-lvnetback /Netbackup
ffff881229af4c00 ffff8812256b4800 ext4 /dev/mapper/racoravg-lvracdb /IPMSRAC/oracle/DB
ffff88122a69d200 ffff8812267ea000 ext4 /dev/mapper/racoravg-lvracgrid /IPMSRAC/oracle/GRID
ffff88122a69d380 ffff8812267eb800 ext4 /dev/mapper/racarchvg-lvracarch /IPMSRAC/oracle/ARCH
ffff88122a69d500 ffff8812267ed000 ext4 /dev/mapper/racexpvg-lvracexp /IPMSRAC/oracle/EXP
ffff88122a36dc80 ffff88121fe66000 binfmt_misc none /proc/sys/fs/binfmt_misc
ffff881229af5800 ffff881224fcf000 autofs /etc/auto.misc /misc
ffff881229af5980 ffff881224fc9000 autofs -hosts /net
ffff88122254dc80 ffff881222ca8800 oracleasmfs oracleasmfs /dev/oracleasm
+-------------------------------+
>----------------------| Last 40 lines of dmesg buffer |----------------------<
+-------------------------------+
[8381884.056078] R10: 0000000000000003 R11: 0000000000000001 R12: ffffffff81771ce0
[8381884.056080] R13: 0000000000000000 R14: ffffffff814782e0 R15: ffff880c7c2b0000
[8381884.056083] FS: 0000000000000000(0000) GS:ffff88127fd80000(0000) knlGS:0000000000000000
[8381884.056086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[8381884.056088] CR2: 0000000000000068 CR3: 0000001226fe3000 CR4: 00000000003406e0
[8381884.056090] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[8381884.056092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[8381884.056093] Stack:
[8381884.056095] ffff880b736c7bb8 ffffffff816e0c0d ffff880b736c7bd8 0000000000000000
[8381884.056099] ffffffff81771ce0 0000000000000000 ffff880b736c7c08 ffffffff8128934c
[8381884.056103] ffff880b736c7c18 ffffffff81ba8700 ffff880c7c2b2c28 ffffffffa00d5020
[8381884.056107] Call Trace:
[8381884.056111] [<ffffffff816e0c0d>] ? _cond_resched+0x1d/0x30
[8381884.056116] [<ffffffff8128934c>] kernfs_find_and_get_ns+0x3c/0x70
[8381884.056120] [<ffffffff8128d2dd>] sysfs_unmerge_group+0x1d/0x60
[8381884.056124] [<ffffffff8147a28c>] dpm_sysfs_remove+0x2c/0x70
[8381884.056127] [<ffffffff8146f4e8>] device_del+0x58/0x230
[8381884.056131] [<ffffffff8146f329>] ? device_remove_file+0x19/0x20
[8381884.056135] [<ffffffff814782e0>] ? transport_add_device+0x20/0x20
[8381884.056138] [<ffffffff81477bbe>] attribute_container_class_device_del+0x1e/0x30
[8381884.056141] [<ffffffff81478339>] transport_remove_classdev+0x59/0x70
[8381884.056144] [<ffffffff81477de0>] attribute_container_device_trigger+0xa0/0xe0
[8381884.056148] [<ffffffff81478295>] transport_remove_device+0x15/0x20
[8381884.056151] [<ffffffff814c9ba2>] scsi_target_reap_ref_release+0x32/0x50
[8381884.056154] [<ffffffff814c9bec>] scsi_target_reap+0x2c/0x40
[8381884.056157] [<ffffffff814cd1ab>] scsi_remove_target+0xeb/0x120
[8381884.056164] [<ffffffffa00ce116>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[8381884.056168] [<ffffffff810a14e1>] process_one_work+0x151/0x4b0
[8381884.056173] [<ffffffff810a1960>] worker_thread+0x120/0x480
[8381884.056176] [<ffffffff816e05cb>] ? __schedule+0x30b/0x890
[8381884.056180] [<ffffffff810a1840>] ? process_one_work+0x4b0/0x4b0
[8381884.056184] [<ffffffff810a1840>] ? process_one_work+0x4b0/0x4b0
[8381884.056188] [<ffffffff810a6b3e>] kthread+0xce/0xf0
[8381884.056191] [<ffffffff810a6a70>] ? kthread_freezable_should_stop+0x70/0x70
[8381884.056196] [<ffffffff816e52a2>] ret_from_fork+0x42/0x70
[8381884.056199] [<ffffffff810a6a70>] ? kthread_freezable_should_stop+0x70/0x70
[8381884.056201] Code: 40 00 eb ef 89 c2 eb de 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 30 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 0f 1f 44 00 00 <0f> b7 47 68 48 8b 5f 48 49 89 f1 c1 e8 05 83 e0 01 48 85 d2 0f
[8381884.056244] RIP [<ffffffff81288f49>] kernfs_find_ns+0x19/0x110
[8381884.056248] RSP <ffff880b736c7ba8>
[8381884.056250] CR2: 0000000000000068
******************************************************************************
************************ A Summary Of Problems Found *************************
******************************************************************************
-------------------- A list of all +++WARNING+++ messages --------------------
PARTIAL DUMP with size(vmcore) < 25% size(RAM)
There are 1 threads running in their own namespaces
Use 'taskinfo --ns' to get more details
UDP buffer fill >=75% rcv=1 snd=0
------------------------------------------------------------------------------
** Execution took 6.40s (real) 4.21s (CPU), Child processes: 2.41s
이것만으로도 이미 crash dump 분석의 대부분이 해결되었다.
대다수의 상태가 이것으로 확인이 가능하기 때문이다.
마지막 요약분을 보면 덤프가 파셜리하다는 것과, 네임스페이스 확인이
필요하니 taskinfo --ns 를 사용해 보라는 것 그리고 UDP buffer 상태에 대해서
나름의 의견이 나왔다.
물론 마지막 요약분은 영 쓸모가 없는 내용이긴 한데, 차근차근 이 crashinfo 를 확인해보면
감탄밖에 안나올 것이다...
메모리 부분, 타스크 요약부분, 최근 5개의 스레드, 런큐에 있는 프로세스를 확인할 수 있다.
따로 아래와 같이 옵션을 줘 blk request 도 확인할 수 있고,
crash64> crashinfo --blkreq
-- Request Queues Analysis: Count=0, in_flight=0
-- Requests on blk_cpu_done: Count=0
-- Requests from SLAB Analysis: Count=0, STARTED=0 WRITE=0
** Execution took 0.40s (real) 0.38s (CPU), Child processes: 0.05s
스택서머리로 스택호출 횟수도 확인할 수 있다.
crash64> crashinfo --stacksummary
------- 391 stacks like that: ----------
#0 __schedule
#1 schedule
#2 futex_wait_queue_me
#3 futex_wait
#4 do_futex
#5 sys_futex
#6 system_call_fastpath
youngest=0s(pid=10121), oldest=8381851s(pid=7410)
........................
ReportAgent 2 times
auditd 1 times
automount 2 times
busagt 3 times
console-kit-dae 1 times
crsd.bin 29 times
cssdagent 13 times
cssdmonitor 13 times
... 생략
아래와 같이 locks waitqueue, semaphores 또한 검색에 시간이 매우 걸리는 편이라
하나하나 검색하기 부담스러운 부분이 있는데 한번에 볼 수 있다.
crash64> crashinfo --lws
-- rw_semaphores with count > 0 --
-- rw_semaphores with count <= 0 --
uts_sem 0
umhelper_sem 0
css_set_rwsem 0
all_cpu_access_lock 0
trace_event_sem 0
oom_sem 0
shrinker_rwsem 0
memcg_cache_ids_sem 0
namespace_sem 0
key_types_sem 0
keyring_serialise_link_sem 0
crypto_alg_sem 0
asymmetric_key_parsers_sem 0
pci_bus_sem 0
bus_type_sem 0
pcistub_sem 0
dmar_global_lock 0
pcmcia_socket_list_rwsem 0
ehci_cf_port_reset_rwsem 0
minor_rwsem 0
companions_rwsem 0
cpufreq_rwsem 0
leds_list_lock 0
triggers_list_lock 0
dquirks_rwsem 0
cb_lock 0
-- Non-empty wait_queue_head --
log_wait
PID: 8532 TASK: ffff88122378e200 CPU: 3 COMMAND: "rsyslogd"
kauditd_wait
PID: 7275 TASK: ffff881221ff7000 CPU: 0 COMMAND: "kauditd"
ksm_thread_wait
PID: 40 TASK: ffff88122a364600 CPU: 3 COMMAND: "ksmd"
destroy_waitq
PID: 52 TASK: ffff88122a354600 CPU: 3 COMMAND: "fsnotify_mark"
vt_event_waitqueue
PID: 13616 TASK: ffff8811efd87000 CPU: 1 COMMAND: "console-kit-dae"
PID: 13615 TASK: ffff8811efd86200 CPU: 0 COMMAND: "console-kit-dae"
PID: 13576 TASK: ffff8811efd15400 CPU: 1 COMMAND: "console-kit-dae"
PID: 13618 TASK: ffff881223238e00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13614 TASK: ffff8811efd85400 CPU: 1 COMMAND: "console-kit-dae"
PID: 13585 TASK: ffff8812066c7000 CPU: 0 COMMAND: "console-kit-dae"
PID: 13595 TASK: ffff8811efe3c600 CPU: 1 COMMAND: "console-kit-dae"
PID: 13579 TASK: ffff881206613800 CPU: 1 COMMAND: "console-kit-dae"
PID: 13577 TASK: ffff8811efd12a00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13591 TASK: ffff8811efe38e00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13596 TASK: ffff8811efe3d400 CPU: 1 COMMAND: "console-kit-dae"
PID: 13590 TASK: ffff8811efe38000 CPU: 3 COMMAND: "console-kit-dae"
PID: 13603 TASK: ffff8811efceb800 CPU: 2 COMMAND: "console-kit-dae"
PID: 13584 TASK: ffff8812066c5400 CPU: 1 COMMAND: "console-kit-dae"
PID: 13574 TASK: ffff8811efd16200 CPU: 3 COMMAND: "console-kit-dae"
PID: 13589 TASK: ffff881224395400 CPU: 2 COMMAND: "console-kit-dae"
PID: 13617 TASK: ffff881223238000 CPU: 1 COMMAND: "console-kit-dae"
PID: 13588 TASK: ffff88121fbe4600 CPU: 1 COMMAND: "console-kit-dae"
PID: 13578 TASK: ffff8811efd11c00 CPU: 2 COMMAND: "console-kit-dae"
PID: 13602 TASK: ffff8811efceaa00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13601 TASK: ffff8811efce9c00 CPU: 3 COMMAND: "console-kit-dae"
PID: 13628 TASK: ffff881223b99c00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13629 TASK: ffff881223b9aa00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13632 TASK: ffff881223b9d400 CPU: 1 COMMAND: "console-kit-dae"
PID: 13631 TASK: ffff881223b9c600 CPU: 1 COMMAND: "console-kit-dae"
PID: 13630 TASK: ffff881223b9b800 CPU: 1 COMMAND: "console-kit-dae"
PID: 13633 TASK: ffff881223b9e200 CPU: 0 COMMAND: "console-kit-dae"
PID: 13627 TASK: ffff881223b98e00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13597 TASK: ffff8811efe3e200 CPU: 0 COMMAND: "console-kit-dae"
PID: 13581 TASK: ffff8812066c4600 CPU: 1 COMMAND: "console-kit-dae"
PID: 13625 TASK: ffff88122323f000 CPU: 1 COMMAND: "console-kit-dae"
PID: 13626 TASK: ffff881223b98000 CPU: 1 COMMAND: "console-kit-dae"
PID: 13600 TASK: ffff8811efce8e00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13624 TASK: ffff88122323e200 CPU: 1 COMMAND: "console-kit-dae"
PID: 13623 TASK: ffff88122323d400 CPU: 1 COMMAND: "console-kit-dae"
PID: 13587 TASK: ffff88121fbe2a00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13583 TASK: ffff8812066c3800 CPU: 1 COMMAND: "console-kit-dae"
PID: 13592 TASK: ffff8811efe39c00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13608 TASK: ffff8811efd80000 CPU: 0 COMMAND: "console-kit-dae"
PID: 13613 TASK: ffff8811efd84600 CPU: 1 COMMAND: "console-kit-dae"
PID: 13612 TASK: ffff8811efd83800 CPU: 1 COMMAND: "console-kit-dae"
PID: 13594 TASK: ffff8811efe3b800 CPU: 0 COMMAND: "console-kit-dae"
PID: 13621 TASK: ffff88122323b800 CPU: 1 COMMAND: "console-kit-dae"
PID: 13609 TASK: ffff8811efd80e00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13619 TASK: ffff881223239c00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13599 TASK: ffff8811efce8000 CPU: 0 COMMAND: "console-kit-dae"
PID: 13620 TASK: ffff88122323aa00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13598 TASK: ffff8811efe3f000 CPU: 0 COMMAND: "console-kit-dae"
PID: 13622 TASK: ffff88122323c600 CPU: 1 COMMAND: "console-kit-dae"
PID: 13580 TASK: ffff8812066c0000 CPU: 0 COMMAND: "console-kit-dae"
PID: 13586 TASK: ffff88121fbe1c00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13607 TASK: ffff8811efcef000 CPU: 1 COMMAND: "console-kit-dae"
PID: 13610 TASK: ffff8811efd81c00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13611 TASK: ffff8811efd82a00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13593 TASK: ffff8811efe3aa00 CPU: 0 COMMAND: "console-kit-dae"
PID: 13606 TASK: ffff8811efcee200 CPU: 1 COMMAND: "console-kit-dae"
PID: 13605 TASK: ffff8811efced400 CPU: 3 COMMAND: "console-kit-dae"
PID: 13634 TASK: ffff881223b9f000 CPU: 1 COMMAND: "console-kit-dae"
PID: 13582 TASK: ffff8812066c1c00 CPU: 1 COMMAND: "console-kit-dae"
PID: 13604 TASK: ffff8811efcec600 CPU: 0 COMMAND: "console-kit-dae"
PID: 13575 TASK: ffff8811efd17000 CPU: 3 COMMAND: "console-kit-dae"
md_event_waiters
PID: 8715 TASK: ffff881222bd4600 CPU: 2 COMMAND: "hald"
-- Non-empty struct work_struct --
** Execution took 179.91s (real) 179.87s (CPU)
Semaphore 를 중요시 쓰는 RDBMS 관련 이슈에서 이용하기 좋을것 같다.
Workqueues 도 볼 수 있고,
crash64> crashinfo --workqueues
-----------------------WorkQueues - Active only-----------------------
--------fc_wq_13--------- <struct workqueue_struct 0xffff881220187c00>
<struct pool_workqueue 0xffff88127fda1300> active=1 delayed=0
<struct worker_pool 0xffff88127fd96e40> nr_workers=2 nr_idle=1
<struct worker 0xffff88040a93b6c0> kworker/3:3 fc_starget_delete
** Execution took 0.59s (real) 0.60s (CPU)
PCI 정보도 하드웨어 문제가 의심될 때 보려면 은근 귀찮은데 최대한 정리해준다.
crash64> crashinfo --pci
ff:08.0 0880: 8086:6f80 (rev 01)
ff:08.2 1101: 8086:6f32 (rev 01)
ff:08.3 0880: 8086:6f83 (rev 01)
83:00.1 0c04: 10df:f100 (rev 03)
..중략..
84:00.0 0c04: 10df:f100 (rev 03)
84:00.1 0c04: 10df:f100 (rev 03)
============================iomem_resource============================
00000000-00000fff : reserved
00001000-0009d3ff : System RAM
0009d400-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000c7fff : Video ROM
000c8000-000cfbff : Adapter ROM
000d0000-000d17ff : Adapter ROM
000d1800-000d2fff : Adapter ROM
000d3000-000d47ff : Adapter ROM
000d4800-000d5fff : Adapter ROM
000e0000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-6c0f3fff : System RAM
01000000-016ea9c4 : Kernel code
016ea9c5-01c23a7f : Kernel data
01e03000-020d1fff : Kernel bss
25000000-354fffff : Crash kernel
6c0f4000-6d679fff : reserved
6c55e018-6c55e018 : APEI ERST
6c55e01c-6c55e021 : APEI ERST
6c55e028-6c55e039 : APEI ERST
6c55e040-6c55e04c : APEI ERST
6c55e050-6c56004f : APEI ERST
6d67a000-6d6dbfff : ACPI Tables
6d6dc000-71810fff : ACPI Non-volatile Storage
71811000-8fffffff : reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
90000000-c7ffbfff : PCI Bus 0000:00
90000000-900fffff : PCI Bus 0000:02
90000000-9001ffff : 0000:02:00.0
90020000-9003ffff : 0000:02:00.0
90040000-9005ffff : 0000:02:00.1
90060000-9007ffff : 0000:02:00.1
90100000-904fffff : PCI Bus 0000:03
90100000-901fffff : 0000:03:00.0
90200000-902fffff : 0000:03:00.0
90300000-903fffff : 0000:03:00.1
90400000-904fffff : 0000:03:00.1
90500000-908fffff : PCI Bus 0000:04
90500000-905fffff : 0000:04:00.0
90600000-906fffff : 0000:04:00.0
90700000-907fffff : 0000:04:00.1
90800000-908fffff : 0000:04:00.1
c5000000-c5ffffff : PCI Bus 0000:07
c5000000-c5ffffff : 0000:07:00.0
c6000000-c61fffff : PCI Bus 0000:05
c6000000-c601ffff : 0000:05:00.3
c6000000-c601ffff : be2net
c6020000-c603ffff : 0000:05:00.3
c6020000-c603ffff : be2net
c6040000-c605ffff : 0000:05:00.2
c6040000-c605ffff : be2net
c6060000-c607ffff : 0000:05:00.2
c6060000-c607ffff : be2net
c6080000-c609ffff : 0000:05:00.1
c6080000-c609ffff : be2net
c60a0000-c60bffff : 0000:05:00.1
c60a0000-c60bffff : be2net
c60c0000-c60dffff : 0000:05:00.0
c60c0000-c60dffff : be2net
c60e0000-c60fffff : 0000:05:00.0
c60e0000-c60fffff : be2net
c6100000-c6103fff : 0000:05:00.3
c6100000-c6103fff : be2net
c6104000-c6107fff : 0000:05:00.2
c6104000-c6107fff : be2net
c6108000-c610bfff : 0000:05:00.1
c6108000-c610bfff : be2net
c610c000-c610ffff : 0000:05:00.0
c610c000-c610ffff : be2net
c6200000-c63fffff : PCI Bus 0000:04
c6200000-c627ffff : 0000:04:00.1
c6200000-c627ffff : ixgbe
c6280000-c62fffff : 0000:04:00.0
c6280000-c62fffff : ixgbe
c6300000-c6303fff : 0000:04:00.1
c6300000-c6303fff : ixgbe
c6304000-c6307fff : 0000:04:00.0
c6304000-c6307fff : ixgbe
c6400000-c65fffff : PCI Bus 0000:03
c6400000-c647ffff : 0000:03:00.1
c6400000-c647ffff : ixgbe
c6480000-c64fffff : 0000:03:00.0
c6480000-c64fffff : ixgbe
c6500000-c6503fff : 0000:03:00.1
c6500000-c6503fff : ixgbe
c6504000-c6507fff : 0000:03:00.0
c6504000-c6507fff : ixgbe
c6800000-c70fffff : PCI Bus 0000:07
c6800000-c6ffffff : 0000:07:00.0
c7000000-c701ffff : 0000:07:00.1
c7020000-c7023fff : 0000:07:00.0
c7024000-c70240ff : 0000:07:00.1
c7100000-c73fffff : PCI Bus 0000:02
c7100000-c71fffff : 0000:02:00.1
c7100000-c71fffff : igb
c7200000-c72fffff : 0000:02:00.0
c7200000-c72fffff : igb
c7300000-c737ffff : 0000:02:00.0
c7380000-c7383fff : 0000:02:00.1
c7380000-c7383fff : igb
c7384000-c7387fff : 0000:02:00.0
c7384000-c7387fff : igb
c7400000-c76fffff : PCI Bus 0000:01
c7400000-c74fffff : 0000:01:00.0
c7500000-c75fffff : 0000:01:00.0
c7600000-c760ffff : 0000:01:00.0
c7600000-c760ffff : megasas: LSI
c7700000-c78fffff : PCI Bus 0000:05
c7700000-c777ffff : 0000:05:00.3
c7780000-c77fffff : 0000:05:00.2
c7800000-c787ffff : 0000:05:00.1
c7880000-c78fffff : 0000:05:00.0
c7900000-c790ffff : 0000:00:14.0
c7900000-c790ffff : xhci-hcd
c7911000-c79110ff : 0000:00:1f.3
c7912000-c79127ff : 0000:00:1f.2
c7912000-c79127ff : ahci
c7913000-c79133ff : 0000:00:1d.0
c7913000-c79133ff : ehci_hcd
c7914000-c79143ff : 0000:00:1a.0
c7914000-c79143ff : ehci_hcd
c7916000-c791600f : 0000:00:16.1
c7917000-c791700f : 0000:00:16.0
c7918000-c79187ff : 0000:00:11.4
c7918000-c79187ff : ahci
c7919000-c7919fff : 0000:00:05.4
c7ffc000-c7ffcfff : dmar1
c8000000-fbffbfff : PCI Bus 0000:80
c8000000-c80fffff : PCI Bus 0000:81
c8000000-c801ffff : 0000:81:00.0
c8020000-c803ffff : 0000:81:00.0
c8040000-c805ffff : 0000:81:00.1
c8060000-c807ffff : 0000:81:00.1
fb900000-fbbfffff : PCI Bus 0000:81
fb900000-fb9fffff : 0000:81:00.1
fb900000-fb9fffff : igb
fba00000-fbafffff : 0000:81:00.0
fba00000-fbafffff : igb
fbb00000-fbb7ffff : 0000:81:00.0
fbb80000-fbb83fff : 0000:81:00.1
fbb80000-fbb83fff : igb
fbb84000-fbb87fff : 0000:81:00.0
fbb84000-fbb87fff : igb
fbc00000-fbcfffff : PCI Bus 0000:84
fbc00000-fbc3ffff : 0000:84:00.1
fbc40000-fbc7ffff : 0000:84:00.0
fbc80000-fbc83fff : 0000:84:00.1
fbc80000-fbc83fff : lpfc
fbc84000-fbc87fff : 0000:84:00.0
fbc84000-fbc87fff : lpfc
fbc88000-fbc88fff : 0000:84:00.1
fbc88000-fbc88fff : lpfc
fbc89000-fbc89fff : 0000:84:00.0
fbc89000-fbc89fff : lpfc
fbd00000-fbdfffff : PCI Bus 0000:83
fbd00000-fbd3ffff : 0000:83:00.1
fbd40000-fbd7ffff : 0000:83:00.0
fbd80000-fbd83fff : 0000:83:00.1
fbd80000-fbd83fff : lpfc
fbd84000-fbd87fff : 0000:83:00.0
fbd84000-fbd87fff : lpfc
fbd88000-fbd88fff : 0000:83:00.1
fbd88000-fbd88fff : lpfc
fbd89000-fbd89fff : 0000:83:00.0
fbd89000-fbd89fff : lpfc
fbe00000-fbefffff : PCI Bus 0000:82
fbe00000-fbe3ffff : 0000:82:00.1
fbe40000-fbe7ffff : 0000:82:00.0
fbe80000-fbe83fff : 0000:82:00.1
... 중략 ...
fbe80000-fbe83fff : lpfc
fbe84000-fbe87fff : 0000:82:00.0
fbe84000-fbe87fff : lpfc
fbe88000-fbe88fff : 0000:82:00.1
fbe88000-fbe88fff : lpfc
fbe89000-fbe89fff : 0000:82:00.0
fbe89000-fbe89fff : lpfc
fbf00000-fbf00fff : 0000:80:05.4
fbffc000-fbffcfff : dmar0
fec00000-fecfffff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec40000-fec403ff : IOAPIC 2
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed12000-fed1200f : pnp 00:01
fed12010-fed1201f : pnp 00:01
fed1b000-fed1bfff : pnp 00:01
fed1c000-fed44fff : reserved
fed1c000-fed3ffff : pnp 00:01
fed1f410-fed1f414 : iTCO_wdt
fed45000-fed8bfff : pnp 00:01
fee00000-feefffff : pnp 00:01
fee00000-fee00fff : Local APIC
... 중략 ...
ff000000-ff3fffff : reserved
ff500000-ffffffff : reserved
100000000-127fffffff : System RAM
===========================ioport_resource============================
0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0061-0061 : PNP0800:00
0064-0064 : keyboard
0070-0071 : rtc0
... 중략 ...
70e0-70e3 : ahci
70f0-70f7 : 0000:00:11.4
70f0-70f7 : ahci
7100-7103 : 0000:00:11.4
7100-7103 : ahci
7110-7117 : 0000:00:11.4
7110-7117 : ahci
8000-ffff : PCI Bus 0000:80
** Execution took 0.51s (real) 0.43s (CPU)
유저영역에서 사용된 메모리도 아래와 같이 간단히 계산해 준다.
crash64> crashinfo --umem
RSS_TOTAL=71867908 pages, %mem= 93.4
** Execution took 0.57s (real) 0.30s (CPU), Child processes: 0.30s
즐겨 쓰는 명령중 하나인 fregs 는 bt 찾은 프로세스 스택의 주요 Register를
스크롤 올려가며 찾아봐야 하는걸 일목요연하게 찾아주어
시간을 상당히 줄여준다.
crash64> fregs
PID: 2064 TASK: ffff881191d4f000 CPU: 3 COMMAND: kworker/3:3
#0 machine_kexec called from 0xffffffff81112c98 <crash_kexec+104>
+R12: 0xffff880b736c7898
+R13: 0xffff880b736c7af8
+R14: 0x9
+R15: 0xffff881191d4f000
+RBP: 0xffff880b736c7958
+RBX: 0xffff880b736c7af8
#1 crash_kexec called from 0xffffffff8101a7f8 <oops_end+232>
+R12: 0x96
+RBP: 0xffff880b736c7988
+RBX: 0x9
#2 oops_end called from 0xffffffff8106d931 <no_context+353>
+R12: 0x96
+R13: 0x68
+RBP: 0xffff880b736c79d8
+RBX: 0xffff880b736c7af8
1 RDI: 0x96
2 RSI: 0xffff880b736c7af8
#3 no_context called from 0xffffffff8106db2d <__bad_area_nosemaphore+301>
+R12: 0x68
+R13: 0xffff880b736c7af8
+R14: 0xffff881191d4f000
+R15: 0x30001
+RBP: 0xffff880b736c7a28
+RBX: 0x0
1 RCX: 0xb
#4 __bad_area_nosemaphore called from 0xffffffff8106dc43 <bad_area_nosemaphore+19>
+R12: 0x0
+R13: 0x0
+R14: 0xffffffff814782e0
+R15: 0xffff881191d4f000
+RBP: 0xffff880b736c7a38
+RBX: 0x68
1 RCX: 0x30001
#5 bad_area_nosemaphore called from 0xffffffff8106e1c8 <__do_page_fault+808>
+RBP: 0xffff880b736c7aa8
2 RDI: 0xffff880b736c7af8
3 RSI: 0x0
taskinfo 를 이용하면 ps 또한 tree 형태로 볼 수 있다!
crash64> taskinfo --pstree
init(1)-+-ReportAgent(3635)---4*[{ReportAgent}]
|-SVRemoteConnect(1095)
|-abrtd(13382)
|-atd(13409)
|-auditd(8495)---{auditd}
|-automount(8980)---4*[{automount}]
|-biosagt(25634)
|-bonobo-activati(13691)---{bonobo-activati}
|-bpcd(27885)
|-busagt(25574)---5*[{busagt}]
|-certmonger(13454)
|-console-kit-dae(13572)---63*[{console-kit-dae}]
|-crond(13394)
|-crsd.bin(9712)---46*[{crsd.bin}]
|-cssdagent(8648)---15*[{cssdagent}]
|-cssdmonitor(8625)---15*[{cssdmonitor}]
|-dbus-daemon(8641)
|-dbus-daemon(13643)
|-dbus-launch(13642)
|-devkit-power-da(13647)
|-eecd(24802)---40*[{eecd}]
|-etheragt(25619)
|-evmd.bin(9430)---16*[{evmd.bin}]
|-gconfd-2(13669)
|-gdm-binary(13513)-+-gdm-simple-slav(13542)-+-Xorg(13545)---4*[{Xorg}]
| | |-gdm-session-wor(13734)
| | |-gnome-session(13644)-+-at-spi-registry(13688)
| | | |-gdm-simple-gree(13714)
| | | |-gnome-power-man(13713)
| | | |-metacity(13711)---{metacity}
| | | |-plymouth-log-vi(13712)
메모리 상태역시 아래와 같이 쉽게 확인 할 수 있다.
crash64> taskinfo --memory
==== First 8 Tasks reverse-sorted by RSS+SHM ====
PID= 27510 CMD=oracle RSS=11.023 Gb shm=15.002 Gb
PID= 10887 CMD=oracle RSS=4.986 Gb shm=15.002 Gb
PID= 11163 CMD=oracle RSS=4.545 Gb shm=15.002 Gb
PID= 10873 CMD=oracle RSS=4.388 Gb shm=15.002 Gb
PID= 10859 CMD=oracle RSS=4.359 Gb shm=15.002 Gb
PID= 10863 CMD=oracle RSS=4.359 Gb shm=15.002 Gb
PID= 10879 CMD=oracle RSS=4.179 Gb shm=15.002 Gb
PID= 5269 CMD=oracle RSS=3.901 Gb shm=15.002 Gb
==== First 8 Tasks Reverse-sorted by RSS only ====
PID= 27510 CMD=oracle RSS=11.023 Gb shm=15.002 Gb
PID= 10887 CMD=oracle RSS=4.986 Gb shm=15.002 Gb
PID= 11163 CMD=oracle RSS=4.545 Gb shm=15.002 Gb
PID= 10873 CMD=oracle RSS=4.388 Gb shm=15.002 Gb
PID= 10859 CMD=oracle RSS=4.359 Gb shm=15.002 Gb
PID= 10863 CMD=oracle RSS=4.359 Gb shm=15.002 Gb
PID= 10879 CMD=oracle RSS=4.179 Gb shm=15.002 Gb
PID= 5269 CMD=oracle RSS=3.901 Gb shm=15.002 Gb
=== Total Memory in RSS 68.539 Gb
=== Total Memory in SHM 15.004 Gb
** Execution took 2.83s (real) 1.24s (CPU), Child processes: 0.69s
정말 멋지지 않은가?
이제 크래쉬 분석을 위한 기술을 연습하고 내용을 공유하기위한 나의 길은 끝난거 같다 ㅋㅋㅋ
hanginfo 에서 볼 수 있는 내용은 아래와 같다.
crash64> hanginfo -h
Usage: hanginfo [options]
Options:
-h, --help show this help message and exit
-v verbose output
--version Print program version and exit
--maxpids=MAXPIDS Maximum number of PIDs to print
--sortbypid Sort by pid (the default is by ran_ago)
--syslogger Print info about hangs on AF_UNIX sockets (such as used by syslogd
--tree Print tree of resources owners (experimental!)
--saphana Print recommendations for SAP HANA specific hangs
안타깝게 현재 사용하는 샘플에서 hanginfo 는 충분한 정보를 보여주지 않았다.
특이한점은 SAP Hana 를 위한 인포메이션도 따로 제공한다는 점이다. ㅋㅋㅋ
또한 tslog 를 통해 기존 dmesg 커널 로그에 타임스탬프를 추가하여
직관적으로 해당 로그가 발생 된 순간과 경과시간을 확인할 수 있다.
2017-12-17 18:27:22 [8381485.416618] st 13:0:8:0: reservation conflict
2017-12-17 18:28:05 [8381528.355935] rport-13:0-2: blocked FC remote port time out: removing target and saving binding
2017-12-17 18:28:05 [8381528.356115] lpfc 0000:83:00.0: 2:(0):0203 Devloss timeout on WWPN 50:05:07:60:44:41:c6:01 NPort x010e00 Data: x40000 x1 x0
2017-12-17 18:28:32 [8381555.333221] rport-13:0-3: blocked FC remote port time out: removing target and saving binding
2017-12-17 18:28:32 [8381555.333376] lpfc 0000:83:00.0: 2:(0):0203 Devloss timeout on WWPN 50:05:07:60:44:41:c6:05 NPort x011200 Data: x100 x5 x0
2017-12-17 18:28:35 [8381558.332433] scsi 13:0:0:0: Sequential-Access IBM ULT3580-TD5 F990 PQ: 0 ANSI: 6
2017-12-17 18:28:39 [8381562.334719] st 13:0:0:0: Attached scsi tape st0
2017-12-17 18:28:39 [8381562.334727] st 13:0:0:0: st0: try direct i/o: yes (alignment 4 B)
2017-12-17 18:28:39 [8381562.334933] st 13:0:0:0: Attached scsi generic sg49 type 1
2017-12-17 18:29:02 [8381585.306996] lpfc 0000:83:00.0: 2:(0):2756 LOGO failure DID:011200 Status:x3/x2
2017-12-17 18:29:19 [8381602.293186] st 13:0:10:0: [st10] Error e0008 (driver bt 0x0, host bt 0xe).
2017-12-17 18:29:25 [8381608.287600] rport-13:0-2: blocked FC remote port time out: removing target and saving binding
2017-12-17 18:29:25 [8381608.287693] st 13:0:0:0: rejecting I/O to offline device
2017-12-17 18:29:25 [8381608.287706] scsi 13:0:0:1: rejecting I/O to offline device
2017-12-17 18:29:25 [8381608.287711] scsi 13:0:0:1: killing request
2017-12-17 18:34:01 [8381884.050200] rport-13:0-12: blocked FC remote port time out: removing target and saving binding
2017-12-17 18:34:01 [8381884.050323] lpfc 0000:83:00.0: 2:(0):0203 Devloss timeout on WWPN 50:05:07:60:44:41:c6:0b NPort x010600 Data: x40000 x1 x0
2017-12-17 18:34:01 [8381884.055245] scsi 13:0:10:0: scsi scan: 70 byte inquiry failed. Consider BLIST_INQUIRY_36 for this device
2017-12-17 18:34:01 [8381884.055364] ------------[ cut here ]------------
2017-12-17 18:34:01 [8381884.055377] WARNING: CPU: 3 PID: 2064 at fs/kernfs/dir.c:1253 kernfs_remove_by_name_ns+0xac/0xc0()
2017-12-17 18:34:01 [8381884.055380] kernfs: can not remove 'node_name', no directory
2017-12-17 18:34:01 [8381884.055381] Modules linked in: bridge stp llc ipmi_devintf seos(POE) tcp_diag inet_diag oracleasm autofs4 cpufreq_powersave bonding ipv6 scsi_dh_alua dm_round_robin dm_multipath uinput iTCO_wdt iTCO_vendor_support pcspkr ch osst st sb_edac edac_core i2c_i801 lpc_ich mfd_core xhci_pci xhci_hcd ixgbe mdio igb dca i2c_algo_bit ptp pps_core sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler ext4 jbd2 mbcache2 sd_mod lpfc scsi_transport_fc ahci libahci be2net vxlan udp_tunnel ip6_udp_tunnel megaraid_sas mxm_wmi wmi dm_mirror dm_region_hash dm_log dm_mod
2017-12-17 18:34:01 [8381884.055448] CPU: 3 PID: 2064 Comm: kworker/3:3 Tainted: P OE 4.1.12-94.3.8.el6uek.x86_64 #2
2017-12-17 18:34:01 [8381884.055450] Hardware name: FUJITSU PRIMERGY RX2540 M2/D3289-B1, BIOS V5.0.0.11 R1.15.0 for D3289-B1x 02/24/2017
2017-12-17 18:34:01 [8381884.055462] Workqueue: fc_wq_13 fc_starget_delete [scsi_transport_fc]
2017-12-17 18:34:01 [8381884.055465] 0000000000000000 ffff880b736c7b78 ffffffff816e018f ffff880b736c7bc8
2017-12-17 18:34:01 [8381884.055470] 00000000000004e5 ffff880b736c7bb8 ffffffff810868c5 ffff880b736c7ba8
2017-12-17 18:34:01 [8381884.055474] 0000000000000000 ffffffffa00d22fd 0000000000000000 ffffffff814782e0
2017-12-17 18:34:01 [8381884.055478] Call Trace:
2017-12-17 18:34:01 [8381884.055488] [<ffffffff816e018f>] dump_stack+0x63/0x84
2017-12-17 18:34:01 [8381884.055497] [<ffffffff810868c5>] warn_slowpath_common+0x95/0xe0
2017-12-17 18:34:01 [8381884.055507] [<ffffffff814782e0>] ? transport_add_device+0x20/0x20
2017-12-17 18:34:01 [8381884.055512] [<ffffffff810869c6>] warn_slowpath_fmt+0x46/0x50
2017-12-17 18:34:01 [8381884.055519] [<ffffffff811ea590>] ? kfree+0x130/0x170
2017-12-17 18:34:01 [8381884.055525] [<ffffffff8128a33c>] kernfs_remove_by_name_ns+0xac/0xc0
2017-12-17 18:34:01 [8381884.055528] [<ffffffff8128c3c5>] sysfs_remove_file_ns+0x15/0x20
2017-12-17 18:34:01 [8381884.055533] [<ffffffff8146f329>] device_remove_file+0x19/0x20
2017-12-17 18:34:01 [8381884.055537] [<ffffffff81477b6b>] attribute_container_remove_attrs+0x5b/0x90
2017-12-17 18:34:01 [8381884.055542] [<ffffffff81477bb6>] attribute_container_class_device_del+0x16/0x30
2017-12-17 18:34:01 [8381884.055545] [<ffffffff81478339>] transport_remove_classdev+0x59/0x70
2017-12-17 18:34:01 [8381884.055549] [<ffffffff81477de0>] attribute_container_device_trigger+0xa0/0xe0
2017-12-17 18:34:01 [8381884.055553] [<ffffffff81478295>] transport_remove_device+0x15/0x20
2017-12-17 18:34:01 [8381884.055559] [<ffffffff814c9ba2>] scsi_target_reap_ref_release+0x32/0x50
2017-12-17 18:34:01 [8381884.055562] [<ffffffff814c9bec>] scsi_target_reap+0x2c/0x40
2017-12-17 18:34:01 [8381884.055567] [<ffffffff814cd1ab>] scsi_remove_target+0xeb/0x120
2017-12-17 18:34:01 [8381884.055587] [<ffffffffa00ce116>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
2017-12-17 18:34:01 [8381884.055596] [<ffffffff810a14e1>] process_one_work+0x151/0x4b0
2017-12-17 18:34:01 [8381884.055601] [<ffffffff810a1960>] worker_thread+0x120/0x480
2017-12-17 18:34:01 [8381884.055604] [<ffffffff816e05cb>] ? __schedule+0x30b/0x890
2017-12-17 18:34:01 [8381884.055608] [<ffffffff810a1840>] ? process_one_work+0x4b0/0x4b0
2017-12-17 18:34:01 [8381884.055613] [<ffffffff810a1840>] ? process_one_work+0x4b0/0x4b0
2017-12-17 18:34:01 [8381884.055617] [<ffffffff810a6b3e>] kthread+0xce/0xf0
2017-12-17 18:34:01 [8381884.055621] [<ffffffff810a6a70>] ? kthread_freezable_should_stop+0x70/0x70
2017-12-17 18:34:01 [8381884.055628] [<ffffffff816e52a2>] ret_from_fork+0x42/0x70
2017-12-17 18:34:01 [8381884.055632] [<ffffffff810a6a70>] ? kthread_freezable_should_stop+0x70/0x70
2017-12-17 18:34:01 [8381884.055639] ---[ end trace 614896ece49fce3d ]---
pyKdump 의 추가적인 장점은 epython 명령으로써,
내가 혹은 외부에서 가져온 extenstion python scripts 를 수행할 수 있다는 점이다.
crash64> epython -h
Usage:
epython [epythonoptions] [progname [--ehelp] [progoptions] [progargs]]
epythonoptions:
---------------
[-h|--help]
[-v|--version] - report versions
[-d|--debug n] - set debugging level
[-p|--path] - show Python version and syspath
[--ehelp] - show extra options, common for all programs
테스트 파이선을 작성해 로드해 보겠다.
# cat test.py
#!/usr/bin/env python
from optparse import OptionParser
use = "Usage: %prog [options] argument1 argument2"
parser = OptionParser(usage = use)
parser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False, help="Set mode to verbose.")
parser.add_option("-f", "--filename", dest="write", metavar="FILE", help="write output to FILE"),
encode_url = ""
if (encode_url == ""):
parser.add_option("-n", "--noaction",
action="store_true",
dest="noaction",
default=False,
help="Only colorising the output and not connection to server")
options, args = parser.parse_args()
if options.verbose:
print("Mode is set to verbose!")
if options.noaction:
print("Noaction Added!")
print("filename : ", options.write)
print("ARGs : ", args[0], args[1])
crash64> epython test.py -v -n -f myfile test1 test2
Mode is set to verbose!
Noaction Added!
filename : myfile
ARGs : test1 test2
** Execution took 0.00s (real) 0.00s (CPU)
crash64> epython test.py -v -f myfile test1 test2
Mode is set to verbose!
filename : myfile
ARGs : test1 test2
여기까지 PyKdump 모듈의 기능에 대해서 살펴보았다.
이제 커널정보들도 상당히 공개되고 알려져 먹고살기 참 힘든 세상이 되었다.
추가 팁 : 자동으로 mPyKdump 모듈을 로드시키고 싶을 경우,
.crashrc 파일을 만들어 아래와 같이 넣어주면 실행시 자동 로드한다.
.crashrc 는 크래쉬 툴 명령명과 동일하게 만들어 주면 된다.
(.crash64rc 와 같이.)
# cat ~/.crash64rc
extend /usr/local/lib64/mpykdump64.so
'Skills > mY Technutz' 카테고리의 다른 글
| PyKdump extension - pycrashext (0) | 2019.12.14 |
|---|---|
| The effective crash-utility for vmcore analysis (PyKdump) (0) | 2019.12.13 |
| kernel Dump Analysis #18 (0) | 2019.05.11 |
| Kernel Dump Analysis #17 (0) | 2019.04.04 |
| libfc: Update rport reference counting bug - 1368175 (0) | 2018.03.29 |
| Kernel Dump Analysis #16 (0) | 2018.02.22 |
Trackback :
댓글을 달아 주세요
벌써 18번째 인가... 사실 더 있지만 귀찮아서 안하다 보니.. ㅋㅋ
이번엔 GPU/DRM 관련 버그.. 상당히 따끈따근한 새 버그에 대해서 Digging 해 본다.
KERNEL: /share/linuxrpm/vmlinux_repo/64/4.14.35-1818.3.3.el7uek.x86_64/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 16
DATE: Tue Apr 16 15:42:21 2019
UPTIME: 6 days, 01:18:20
LOAD AVERAGE: 0.00, 0.00, 0.00
TASKS: 330
NODENAME: *********
RELEASE: 4.14.35-1818.3.3.el7uek.x86_64
VERSION: #2 SMP Mon Sep 24 14:45:01 PDT 2018
MACHINE: x86_64 (3600 Mhz)
MEMORY: 63.3 GB
PANIC: "general protection fault: 0000 [#1] SMP PTI"
PID: 14887
COMMAND: "plymouthd"
TASK: ffff994e9c1adc40 [THREAD_INFO: ffff994e9c1adc40]
CPU: 2
STATE: TASK_RUNNING (PANIC)
crash7latest> mod -s drm usr/lib/debug/lib/modules/4.14.35-1818.3.3.el7uek.x86_64/kernel/drivers/gpu/drm/drm.ko.debug
MODULE NAME SIZE OBJECT FILE
ffffffffc0306b00 drm 385024 usr/lib/debug/lib/modules/4.14.35-1818.3.3.el7uek.x86_64/kernel/drivers/gpu/drm/drm.ko.debug
crash7latest> mod -s drm_kms_helper usr/lib/debug/lib/modules/4.14.35-1818.3.3.el7uek.x86_64/kernel/drivers/gpu/drm/drm_kms_helper.ko.debug
MODULE NAME SIZE OBJECT FILE
ffffffffc0600500 drm_kms_helper 163840 usr/lib/debug/lib/modules/4.14.35-1818.3.3.el7uek.x86_64/kernel/drivers/gpu/drm/drm_kms_helper.ko.debug
모듈을 불러와준다.
crash7latest> bt -l
PID: 14887 TASK: ffff994e9c1adc40 CPU: 2 COMMAND: "plymouthd"
#0 [ffffa5a8096e7908] machine_kexec at ffffffffbd066191
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/kernel/machine_kexec_64.c: 342
#1 [ffffa5a8096e7968] __crash_kexec at ffffffffbd12f4c2
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/kernel/kexec_core.c: 947
#2 [ffffa5a8096e7a38] crash_kexec at ffffffffbd13061c
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/include/linux/compiler.h: 206
#3 [ffffa5a8096e7a58] oops_end at ffffffffbd03148a
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/kernel/dumpstack.c: 278
#4 [ffffa5a8096e7a80] die at ffffffffbd031b42
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/kernel/dumpstack.c: 357
#5 [ffffa5a8096e7ab0] do_general_protection at ffffffffbd02ec12
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/kernel/traps.c: 561
#6 [ffffa5a8096e7ae0] general_protection at ffffffffbda0388c
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/entry/entry_64.S: 1272
[exception RIP: drm_helper_probe_single_connector_modes+188]
RIP: ffffffffc05e80ac RSP: ffffa5a8096e7b98 RFLAGS: 00010203
RAX: 00ffff994e98cb08 RBX: ffff994aa7c62088 RCX: 000000000000001e
RDX: ffffa5a8096e7c08 RSI: ffffa5a8096e7c08 RDI: ffff994aa7c65240
RBP: ffffa5a8096e7c50 R8: ffff994e9b493fc0 R9: 0000000000000001
R10: 0000000000000001 R11: 00000000fffffff2 R12: ffff994aa7c65218
R13: ffff994aa7c62028 R14: ffffffffc03321a0 R15: ffff994aa7c62000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/drivers/gpu/drm/drm_probe_helper.c: 423
#7 [ffffa5a8096e7c58] drm_mode_getconnector at ffffffffc02db97e [drm]
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/drivers/gpu/drm/drm_connector.c: 1321
#8 [ffffa5a8096e7d10] drm_ioctl_kernel at ffffffffc02c4dec [drm]
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/drivers/gpu/drm/drm_ioctl.c: 741
#9 [ffffa5a8096e7d50] drm_ioctl at ffffffffc02c51e4 [drm]
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/include/linux/uaccess.h: 155
#10 [ffffa5a8096e7e60] do_vfs_ioctl at ffffffffbd29263a
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/fs/ioctl.c: 47
#11 [ffffa5a8096e7ee8] sys_ioctl at ffffffffbd292c09
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/fs/ioctl.c: 701
#12 [ffffa5a8096e7f28] do_syscall_64 at ffffffffbd003949
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/entry/common.c: 295
#13 [ffffa5a8096e7f50] entry_SYSCALL_64_after_hwframe at ffffffffbda00195
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/arch/x86/entry/entry_64.S: 247
RIP: 00007f3beba00997 RSP: 00007fffc950e968 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 000055beb6890720 RCX: 00007f3beba00997
RDX: 00007fffc950e9f0 RSI: 00000000c05064a7 RDI: 0000000000000009
RBP: 00007fffc950e9f0 R8: 0000000000000000 R9: 0000000000000004
R10: 0000000000000009 R11: 0000000000000246 R12: 00000000c05064a7
R13: 0000000000000009 R14: 00007fffc950e9f0 R15: 0000000000000009
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
crash7latest>
drm_probe_helper.c 의 423 라인의 drm_helper_probe_single_connector_modes 함수가 호출되면서
갑작스럽게 Kernel Protection 이 동작하였음을 확인 할 수 있다.
라인을 확인해 보면,
421: /* set all old modes to the stale state */
422: list_for_each_entry(mode, &connector->modes, head)
423: mode->status = MODE_STALE;
425: old_status = connector->status;
Disassemble 을 해보자.
crash7latest> dis -rl drm_helper_probe_single_connector_modes+188
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/drivers/gpu/drm/drm_probe_helper.c: 419
0xffffffffc05e8098 <drm_helper_probe_single_connector_modes+168>: test %eax,%eax
0xffffffffc05e809a <drm_helper_probe_single_connector_modes+170>: js 0xffffffffc05e8726 <drm_helper_probe_single_connector_modes+1846>
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/drivers/gpu/drm/drm_probe_helper.c: 422
0xffffffffc05e80a0 <drm_helper_probe_single_connector_modes+176>: mov 0x88(%r15),%rax
0xffffffffc05e80a7 <drm_helper_probe_single_connector_modes+183>: cmp %rax,%rbx
0xffffffffc05e80aa <drm_helper_probe_single_connector_modes+186>: je 0xffffffffc05e80bb <drm_helper_probe_single_connector_modes+203>
/usr/src/debug/kernel-4.14.35/linux-4.14.35-1818.3.3.el7uek/drivers/gpu/drm/drm_probe_helper.c: 423
0xffffffffc05e80ac <drm_helper_probe_single_connector_modes+188>: movl $0xfffffffd,0x50(%rax)
structure 를 확인해보자.
crash7latest> struct drm_device 0xffff994aa7c65240
struct drm_device {
legacy_dev_list = {
next = 0xffffa5a8096e7c08,
prev = 0xffffa5a8096e7c08
},
if_version = 0,
ref = {
refcount = {
refs = {
counter = 0
}
}
},
dev = 0x0,
driver = 0x0,
dev_private = 0xffff994aa7c65268,
control = 0xffff994aa7c65268,
primary = 0x6000000,
render = 0xffff994e9f1976a1,
registered = false,
master = 0x6000000,
[[[ 중략 ]]]
crash7latest> struct drm_device ffff994aa7c62000
struct drm_device {
legacy_dev_list = {
next = 0xffff994aa7c65000,
prev = 0xffff994e9bc55800
},
if_version = 0,
ref = {
refcount = {
refs = {
counter = 0
}
}
},
dev = 0xffff994aa7c652f8,
driver = 0xffff994aa7c652f8,
dev_private = 0xc0c0c0c00000001e,
control = 0xffff994aa7c62148,
primary = 0x5,
render = 0xffffffffc02d9da0 <drm_connector_free>,
registered = 192,
master = 0x0,
unplugged = {
counter = 0
},
anon_inode = 0xffff994aa7c62060,
unique = 0xffff994aa7c62060 "` \306\247J\231\377\377` \306\247J\231\377\377",
struct_mutex = {
owner = {
counter = 4294967296
},
wait_lock = {
{
rlock = {
raw_lock = {
val = {
counter = 1
}
}
이제 실제 라인에 있는 connector mode, status 의 내용들을 살펴볼 차례다.
crash7latest> struct drm_connector.modes 0xffff994aa7c65240
modes = {
next = 0x1,
prev = 0xffff994e9bc54d28
}
crash7latest> struct drm_display_mode.status ffff994aa7c62000
status = MODE_OK
crash7latest> rd 0xffff994aa7c65240
ffff994aa7c65240: ffffa5a8096e7c08 .|n.....
crash7latest> rd -a 0xffff994e9bf77c08 << Null
crash7latest> device 0xffff994e9bf77c08
struct device {
parent = 0xffff994aa7c65310,
p = 0xffff994aa7c65310,
kobj = {
name = 0xe0e0e0e00000001d <Address 0xe0e0e0e00000001d out of bounds>,
entry = {
next = 0x0,
prev = 0x0
},
parent = 0x0,
kset = 0xffff994e9b493fb8,
ktype = 0x1,
sd = 0x1,
kref = {
refcount = {
refs = {
counter = -1480171520
}
}
},
state_initialized = 0,
state_in_sysfs = 1,
state_add_uevent_sent = 0,
state_remove_uevent_sent = 1,
uevent_suppress = 0
},
init_name = 0x0,
자, 대략 설명하자면.. DRM connector 의 mode 는 MODE_OK 로 정상적인 상태였고,
DRM 을 위한 device 에 대한 내용을 확인 결과 Out of Bound 로 표현되는
잘못된 주소를 참조하고 있다는 것을 알 수 있었다.
따라서 커널은 시스템 보호를 위해 protection 을 수행하게 되었고, 그로 인해 패닉이 발생한 것이다.
사실은 MODE_STALE 로 바뀌어야 했던 상황이며,
이는 drm 모듈이 이미 해제하고 사용하지 않아야 할 메모리 번지를 접근하여 재사용하려 했기 때문에 발생한 문제이다.
커널 4.14.35-1844.4.4 버젼 에 픽스가 포함되어있으므로,
해당 버젼 이상으로 업데이트를 해야 해결할 수 있다.
분석한지 한시간도 안되서 알려진 버그를 찾았다는게 개인적으로 나름 웃긴 부분이며,
물론 실제로 고객이 분석을 요청했을때, 이렇게 상세하게 지원 해주지는 않는다.
업무 특성상 사실, 솔루션만 제공하면 되니까 그러는 것도 있지만,
돈내고 받는 서비스라는 생각에 이해할 수 있는 기반도 없이 스터디를 요구하는 경우가 많아서이기도 하고..
(이전에 사과먹다만 회사 내부 커널개발자라는 놈이
아주 미친/미친듯이 질문을 한적이 있는데 답답해 미치는줄 알았었...)
블로그에 상세내용을 올리는 것은 분석에 대한 기법을 공유하기 위해서다.
끝.
'Skills > mY Technutz' 카테고리의 다른 글
| PyKdump extension - pycrashext (0) | 2019.12.14 |
|---|---|
| The effective crash-utility for vmcore analysis (PyKdump) (0) | 2019.12.13 |
| kernel Dump Analysis #18 (0) | 2019.05.11 |
| Kernel Dump Analysis #17 (0) | 2019.04.04 |
| libfc: Update rport reference counting bug - 1368175 (0) | 2018.03.29 |
| Kernel Dump Analysis #16 (0) | 2018.02.22 |
Trackback :
댓글을 달아 주세요
재밌는 리턴코드가 발견되어서 간만에 좀 디깅을 해봤다.
KERNEL: /share/linuxrpm/vmlinux_repo/64/3.10.0-693.el7.x86_64/vmlinux
DUMPFILE: 127.0.0.1-2019-01-17-00_38_36.zip_extract/vmcore [PARTIAL DUMP]
CPUS: 40
DATE: Thu Jan 17 00:38:38 2019
UPTIME: 39 days, 18:26:52
LOAD AVERAGE: 0.19, 0.20, 0.24
TASKS: 1420
NODENAME: **********
RELEASE: 3.10.0-693.el7.x86_64
VERSION: #1 SMP Wed Aug 2 06:49:08 PDT 2017
MACHINE: x86_64 (2199 Mhz)
MEMORY: 63.9 GB
PANIC: "BUG: unable to handle kernel paging request at ffffffffc0500790"
PID: 72639
COMMAND: "kworker/15:0"
TASK: ffff880dc2744f10 [THREAD_INFO: ffff881052fb8000]
CPU: 15
STATE: TASK_RUNNING (PANIC)
crash64> log
[ 37.351715] ip6_tables: (C) 2000-2006 Netfilter Core Team
[236233.529248] perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
[346636.325549] perf: interrupt took too long (3127 > 3126), lowering kernel.perf_event_max_sample_rate to 63000
[602033.211929] perf: interrupt took too long (3916 > 3908), lowering kernel.perf_event_max_sample_rate to 51000
[2952215.839114] perf: interrupt took too long (4941 > 4895), lowering kernel.perf_event_max_sample_rate to 40000
[3436125.331154] BUG: unable to handle kernel paging request at ffffffffc0500790
[3436125.331196] IP: [<ffffffffc0500790>] 0xffffffffc050078f
[3436125.331224] PGD 19f5067 PUD 19f7067 PMD 8598de067 PTE 0
[3436125.331251] Oops: 0010 [#1] SMP
[3436125.331270] Modules linked in: ip6table_filter ip6_tables sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc iptable_filter bonding sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass ipmi_ssif crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt dcdbas iTCO_vendor_support mxm_wmi sg ipmi_si ipmi_devintf ipmi_msghandler pcspkr mei_me mei lpc_ich shpchp wmi acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ixgbe ahci crct10dif_pclmul crct10dif_common igb crc32c_intel libahci megaraid_sas libata mdio i2c_algo_bit ptp i2c_core pps_core dca dm_mirror dm_region_hash
[3436125.331620] dm_log dm_mod
[3436125.331629] CPU: 15 PID: 72639 Comm: kworker/15:0 Not tainted 3.10.0-693.el7.x86_64 #1
[3436125.331661] Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.5.5 08/16/2017
[3436125.331721] Workqueue: xfs-cil/dm-3 xlog_cil_push_work [xfs]
[3436125.331746] task: ffff880dc2744f10 ti: ffff881052fb8000 task.ti: ffff881052fb8000
[3436125.331776] RIP: 0010:[<ffffffffc0500790>] [<ffffffffc0500790>] 0xffffffffc050078f
[3436125.331810] RSP: 0018:ffff881052fbbca0 EFLAGS: 00010286
[3436125.331832] RAX: ffffc90007396450 RBX: ffff88100ce8dc08 RCX: 0000000000000000
[3436125.331860] RDX: 0000000000000480 RSI: ffff880c9fbd7980 RDI: ffffc900073968d0
[3436125.331887] RBP: ffff881052fbbd38 R08: 0000000000000000 R09: ffffc90007396450
[3436125.331915] R10: 0000000000000480 R11: 0000000000018d8c R12: 0000000000000480
[3436125.331943] R13: 0000000000000033 R14: 0000000000000002 R15: ffff880c9fbd7000
[3436125.331971] FS: 0000000000000000(0000) GS:ffff88105dfc0000(0000) knlGS:0000000000000000
[3436125.332002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3436125.332026] CR2: ffffffffc0500790 CR3: 000000067136f000 CR4: 00000000003407e0
[3436125.332053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3436125.332081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3436125.332113] Stack:
[3436125.332124] ffff880a3aa68010 0000000000000000 ffff88085522f400 000000000c041837
[3436125.332158] 0000000000018d8c 0000000000000480 ffff88085522f528 0000000000000000
[3436125.332192] 0000000000000480 ffff881000006244 ffff880c9fbd7038 0000625000000001
[3436125.332224] Call Trace:
[3436125.332254] [<ffffffffc04021d8>] xlog_cil_push+0x2a8/0x430 [xfs]
[3436125.332293] [<ffffffffc0402375>] xlog_cil_push_work+0x15/0x20 [xfs]
[3436125.332322] [<ffffffff810a881a>] process_one_work+0x17a/0x440
[3436125.332347] [<ffffffff810a9638>] worker_thread+0x278/0x3c0
[3436125.332371] [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[3436125.332398] [<ffffffff810b098f>] kthread+0xcf/0xe0
[3436125.332420] [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[3436125.332443] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[3436125.332470] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[3436125.332493] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[3436125.332518] Code: Bad RIP value.
[3436125.332538] RIP [<ffffffffc0500790>] 0xffffffffc050078f
[3436125.332564] RSP <ffff881052fbbca0>
[3436125.332580] CR2: ffffffffc0500790
일단 xfs 모듈이 올라가 있으므로, xfs 모듈을 불러와야한다.
Debuginfo 패키지에서 모듈을 추출하는 것은 이전 포스트를 참조하라.
crash64> mod -s xfs 3.10.0-693.el7.x86_64/kernel/fs/xfs/xfs.ko.debug
MODULE NAME SIZE OBJECT FILE
ffffffffc044e280 xfs 978100 3.10.0-693.el7.x86_64/kernel/fs/xfs/xfs.ko.debug
backtrace 를 해서 펑션들의 코드가 있는 소스파일이 무엇인지 확인해 보자.
crash64> bt -l
PID: 72639 TASK: ffff880dc2744f10 CPU: 15 COMMAND: "kworker/15:0"
#0 [ffff881052fbb948] machine_kexec at ffffffff8105c4cb
/linux-3.10.0-693.el7.x86_64/arch/x86/kernel/machine_kexec_64.c: 320
#1 [ffff881052fbb9a8] __crash_kexec at ffffffff81104a32
/linux-3.10.0-693.el7.x86_64/kernel/kexec_core.c: 875
#2 [ffff881052fbba78] crash_kexec at ffffffff81104b20
/linux-3.10.0-693.el7.x86_64/arch/x86/include/asm/atomic.h: 38
#3 [ffff881052fbba90] oops_end at ffffffff816ad278
/linux-3.10.0-693.el7.x86_64/arch/x86/kernel/dumpstack.c: 225
#4 [ffff881052fbbab8] no_context at ffffffff8169d29a
/linux-3.10.0-693.el7.x86_64/arch/x86/mm/fault.c: 703
#5 [ffff881052fbbb08] __bad_area_nosemaphore at ffffffff8169d330
/linux-3.10.0-693.el7.x86_64/arch/x86/mm/fault.c: 782
#6 [ffff881052fbbb50] bad_area_nosemaphore at ffffffff8169d49a
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/arch/x86/mm/fault.c: 790
#7 [ffff881052fbbb60] __do_page_fault at ffffffff816b013e
/linux-3.10.0-693.el7.x86_64/arch/x86/mm/fault.c: 1227
#8 [ffff881052fbbbc0] do_page_fault at ffffffff816b02e5
/linux-3.10.0-693.el7.x86_64/arch/x86/mm/fault.c: 1237
#9 [ffff881052fbbbf0] page_fault at ffffffff816ac508
/linux-3.10.0-693.el7.x86_64/arch/x86/kernel/entry_64.S: 1316
[exception RIP: unknown or invalid address]
RIP: ffffffffc0500790 RSP: ffff881052fbbca0 RFLAGS: 00010286
RAX: ffffc90007396450 RBX: ffff88100ce8dc08 RCX: 0000000000000000
RDX: 0000000000000480 RSI: ffff880c9fbd7980 RDI: ffffc900073968d0
RBP: ffff881052fbbd38 R8: 0000000000000000 R9: ffffc90007396450
R10: 0000000000000480 R11: 0000000000018d8c R12: 0000000000000480
R13: 0000000000000033 R14: 0000000000000002 R15: ffff880c9fbd7000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff881052fbbd40] xlog_cil_push at ffffffffc04021d8 [xfs]
/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 717
#11 [ffff881052fbbe10] xlog_cil_push_work at ffffffffc0402375 [xfs]
/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 799
#12 [ffff881052fbbe20] process_one_work at ffffffff810a881a
/linux-3.10.0-693.el7.x86_64/kernel/workqueue.c: 2252
#13 [ffff881052fbbe68] worker_thread at ffffffff810a9638
/linux-3.10.0-693.el7.x86_64/include/linux/list.h: 188
#14 [ffff881052fbbec8] kthread at ffffffff810b098f
/linux-3.10.0-693.el7.x86_64/kernel/kthread.c: 202
#15 [ffff881052fbbf50] ret_from_fork at ffffffff816b4f18
/linux-3.10.0-693.el7.x86_64/arch/x86/kernel/entry_64.S: 369
crash64>
물론 xfs 모듈을 로드하지 않으면 소스파일을 안알랴준다.
일단 해당 스택을 stack tracer 에 넣어보았다.
오라클에서는 stack checker 라고 해서 스택을 넣어주면 비슷하다고 생각되는 버그를 찾아준다.
나오는 몇가지 케이스를 살펴보았으나 일치하는것은 없는것같다.
좀 더 digging 해 보자... 코드를 좀 살펴보자.
레드햇에서는 매우 유용하게도 웹페이지로 된 Code browser 를 제공한다.
진짜진짜 좋고 유용하지만 난 레드햇 서브스크립션이 없기때문에
직접 ctag 등으로 구축한 코드를 직접 살펴보겠다.
linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 717
716 error = xlog_write(log, &lvhdr, tic, &ctx->start_lsn, NULL, 0);
717 if (error)
718 goto out_abort_free_ticket;
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 555
0xffffffffc0401f30 <xlog_cil_push>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffc0401f35 <xlog_cil_push+5>: push %rbp
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 566
0xffffffffc0401f36 <xlog_cil_push+6>: mov $0x7,%ecx
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 555
0xffffffffc0401f3b <xlog_cil_push+11>: mov %rsp,%rbp
0xffffffffc0401f3e <xlog_cil_push+14>: push %r15
0xffffffffc0401f40 <xlog_cil_push+16>: push %r14
0xffffffffc0401f42 <xlog_cil_push+18>: push %r13
0xffffffffc0401f44 <xlog_cil_push+20>: push %r12
0xffffffffc0401f46 <xlog_cil_push+22>: push %rbx
0xffffffffc0401f47 <xlog_cil_push+23>: mov %rdi,%rbx
0xffffffffc0401f4a <xlog_cil_push+26>: sub $0x98,%rsp
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 556
0xffffffffc0401f51 <xlog_cil_push+33>: mov 0x10(%rdi),%r14
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 555
0xffffffffc0401f55 <xlog_cil_push+37>: mov %rdi,-0xc0(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 566
0xffffffffc0401f5c <xlog_cil_push+44>: lea -0x68(%rbp),%rdi
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 555
0xffffffffc0401f60 <xlog_cil_push+48>: mov %gs:0x28,%rax
0xffffffffc0401f69 <xlog_cil_push+57>: mov %rax,-0x30(%rbp)
0xffffffffc0401f6d <xlog_cil_push+61>: xor %eax,%eax
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 566
0xffffffffc0401f6f <xlog_cil_push+63>: xor %eax,%eax
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 570
0xffffffffc0401f71 <xlog_cil_push+65>: test %r14,%r14
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 566
0xffffffffc0401f74 <xlog_cil_push+68>: rep stos %rax,%es:(%rdi)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 570
0xffffffffc0401f77 <xlog_cil_push+71>: je 0xffffffffc0402039 <xlog_cil_push+265>
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/kmem.h: 77
0xffffffffc0401f7d <xlog_cil_push+77>: mov $0x15,%esi
0xffffffffc0401f82 <xlog_cil_push+82>: mov $0x70,%edi
0xffffffffc0401f87 <xlog_cil_push+87>: callq 0xffffffffc03fd010 <kmem_alloc>
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 52
0xffffffffc0401f8c <xlog_cil_push+92>: xor %esi,%esi
0xffffffffc0401f8e <xlog_cil_push+94>: xor %r8d,%r8d
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 703
0xffffffffc040217d <xlog_cil_push+589>: movl $0x5452414e,-0x78(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 704
0xffffffffc0402184 <xlog_cil_push+596>: movl $0x28,-0x74(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 705
0xffffffffc040218b <xlog_cil_push+603>: mov 0x18(%r12),%eax
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 710
0xffffffffc0402190 <xlog_cil_push+608>: subl $0x1c,0x20(%r12)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 716
0xffffffffc0402196 <xlog_cil_push+614>: mov %r12,%rdx
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 706
0xffffffffc0402199 <xlog_cil_push+617>: mov %r13d,-0x6c(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 708
0xffffffffc040219d <xlog_cil_push+621>: movl $0x10,-0x80(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 709
0xffffffffc04021a4 <xlog_cil_push+628>: movl $0x13,-0x7c(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 712
0xffffffffc04021ab <xlog_cil_push+635>: movl $0x1,-0x60(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 705
0xffffffffc04021b2 <xlog_cil_push+642>: mov %eax,-0x70(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 707
0xffffffffc04021b5 <xlog_cil_push+645>: lea -0x78(%rbp),%rax
0xffffffffc04021b9 <xlog_cil_push+649>: mov %rax,-0x88(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 713
0xffffffffc04021c0 <xlog_cil_push+656>: lea -0x88(%rbp),%rax
0xffffffffc04021c7 <xlog_cil_push+663>: mov %rax,-0x58(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 714
0xffffffffc04021cb <xlog_cil_push+667>: mov 0x40(%rbx),%rax
0xffffffffc04021cf <xlog_cil_push+671>: mov %rax,-0x68(%rbp)
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 716
0xffffffffc04021d3 <xlog_cil_push+675>: callq 0xffffffffc0400320 <xlog_write>
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log_cil.c: 717
0xffffffffc04021d8 <xlog_cil_push+680>: test %eax,%eax
끄트머리를 살펴보자...
crash64> eval 0xffff881052fbbd38-0x68
hexadecimal: ffff881052fbbcd0
decimal: 18446612202425924816 (-131871283626800)
octal: 1777774201012276736320
binary: 1111111111111111100010000001000001010010111110111011110011010000
crash64>
코드 714 라인을 보면 rbx 에서 0x40 만큼 더한 주소를 rax 에 저장시키고,
그 rax 를 rbp 의 -0x68 만큼 이동한 주소에 저장시킨다.
코드는 rax 가 0xffff881052fbbcd0 이여야 한다고 하지만 실질적인 값을 보면 전혀 다른값으로 되어 있다.
RAX: ffffc90007396450 RBX: ffff88100ce8dc08 RCX: 0000000000000000
일단 최종적으로 호출된 xlog_writer 로 가보자.
crash64> dis -rl 0xffffffffc0400320
/usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/xfs/xfs_log.c: 2299
0xffffffffc0400320 <xlog_write>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
xfs_log.c line 2299 :
2292 int
xlog_write(
struct xlog *log,
struct xfs_log_vec *log_vector,
struct xlog_ticket *ticket,
xfs_lsn_t *start_lsn,
struct xlog_in_core **commit_iclog,
uint flags)
2299 {
여기엔 별 다른게 없다.
Assemble 내용을 보면 nopl 오퍼랜드가 수행되었다는것을 보았을 때 실질적으로 xlog_write 는 호출하였으나
아무것도 기록하지 않았음을 알 수 있다.
메모리 번지가 임의로 변경된 정황을 확인하였으므로 이젠 xfs 에 관련된 버그를 찾아봐야 한다.
다행히 커널 버젼 3.10.0-853 에서 아래와 같은 버그픽스들이 있었고 의심스러운 부분들이 보인다 :
[3.10.0-853]
- [sound] alsa: seq: Make ioctls race-free (CVE-2018-1000004) (Jaroslav Kysela) [1537203] {CVE-2018-1000004}
- [gpu] drm/i915/gvt: move write protect handler out of mmio emulation function (Paul Lai) [1525419]
- [gpu] drm/i915/gvt: Factor intel_vgpu_page_track (Paul Lai) [1525419]
- [fs] xfs: eliminate duplicate icreate tx reservation functions (Brian Foster) [1397653]
- [fs] xfs: refactor inode chunk alloc/free tx reservation (Brian Foster) [1397653]
- [fs] xfs: include an allocfree res for inobt modifications (Brian Foster) [1397653]
- [fs] xfs: truncate transaction does not modify the inobt (Brian Foster) [1397653]
- [fs] xfs: fix up agi unlinked list reservations (Brian Foster) [1397653]
- [fs] xfs: include inobt buffers in ifree tx log reservation (Brian Foster) [1397653]
- [fs] xfs: print transaction log reservation on overrun (Brian Foster) [1397653]
- [fs] xfs: dump transaction usage details on log reservation overrun (Brian Foster) [1397653]
- [fs] xfs: refactor xlog_cil_insert_items() to facilitate transaction dump (Brian Foster) [1397653]
- [fs] xfs: separate shutdown from ticket reservation print helper (Brian Foster) [1397653]
- [s390] gs: add compat regset for the guarded storage broadcast control block (Hendrik Brueckner) [1537067]
- [x86] intel_rdt/cqm: avoid negative static key counts (Joe Lawrence) [1524901]
- [x86] efi: Fix boot crash by always mapping boot service regions into new EFI page tables (Lenny Szubowicz) [1535243]
안타깝게 레드햇 가이가 아니라 버그 내용을 볼수 없었으나,
대략적으로 xfs 의 로그 기능 및 transaction 에 대한 메모리 처리과정에서
잘못된 메모리 참조 및 변환이 발생 할 수 있다는 내용으로 판단된다.
아쉽지만, 여기까지만 하고 다음에는 uek 와 관련된 내용으로 포스팅을 해야할것 같다.
내용을 볼수 없어서 비교가 불가하기때문이다 흑... 갑자기 용두사미가 된거같은데 뭐 내맴.
'Skills > mY Technutz' 카테고리의 다른 글
| The effective crash-utility for vmcore analysis (PyKdump) (0) | 2019.12.13 |
|---|---|
| kernel Dump Analysis #18 (0) | 2019.05.11 |
| Kernel Dump Analysis #17 (0) | 2019.04.04 |
| libfc: Update rport reference counting bug - 1368175 (0) | 2018.03.29 |
| Kernel Dump Analysis #16 (0) | 2018.02.22 |
| Kernel Dump Analysis #15 (0) | 2018.02.19 |
Trackback :
댓글을 달아 주세요
간단히 요약하자면,
- Global Network Security
망분리 및 Cloud network routing 뿐만 아니라 데이타 자체에 대한 보안을 지원하는것.
사실 별로 관심도 없다.
어차피 데이타센터 레벨 이야기나 마찬가지기 때문에....
역시 늘 똑같은 레퍼토리로 지속적인 LDC (Local DataCentre) 또는
HDC (Hub DataCentre) 구축에 박차를 가할거라고 '공약' 만 "공략" 한다.
- Machine Learning/AI protection
ML 을 통해 AI 를 이용하여 고객의 장애에 자동적/즉각적으로 대처하고
장애 요소를 사전 파악하여 장애가 발생하기 전에 사전 처리도
자동화하여 서비스 무중단을 지속적으로 제공하는것.
*** '암웨이' 라는 다국적 다단계 방문팡매를 전문으로 하던 상사(기업) 에 대한
한국식 발음이자 무시하는 의미의 단어이다.
*** 다단계의 경우 '입소문' 이라는 기반을 바탕으로 제품의 우수성을 홍보하던 방식을 많이 썼는데,
사실상 거짓과 사실이 혼재하였고 검증하기 어려웠기 때문에 무시당하는 결과에 적잖은 영향을 미쳤다.
'Skills > Cloud Computing' 카테고리의 다른 글
| 2018 Oracle Open World - Cloud Generation 2 (0) | 2018.11.01 |
|---|---|
| VXbench 라는 툴을 아십니까~? (3) | 2012.02.21 |
| Swiss army knife 에 비견되는 BtrFS - 2부 (0) | 2012.02.20 |
| Swiss army knife 에 비견되는 BtrFS - 1부 (0) | 2012.02.17 |
| 클라우드 시스템에서의 DBMS 방식 (2) | 2011.03.30 |
| 그대들은 유칼립투스를 아는가? (0) | 2009.12.23 |
댓글을 달아 주세요