Kernel Dump Analysis - #11

Kernel Crash 분석 업데이트 하나 더.

일이.... 있긴있는데, 분석하기 좀 집중안되는 놈이라 다른짓 하고 있음..

(우선순위가 이게 더 높기에 하고 있는거다. 절대 우선순위 무시하는법 없다... )

백업용도로 사용하는 몇개 서버군들중 두대의 서버가 비 동시적 시간에 Crash 되어

리붓되었다는 것이 배경...

1. 백업에 관련된 서버군들중 하나라고 한다. ( nodename 수정됨 )

  KERNEL: 2.6.32-300.39.5.el5uek/vmlinux
 DUMPFILE: 3-8598812281_vmcore
 CPUS: 16
 DATE: Thu Feb 20 09:12:31 2014
 UPTIME: 77 days, 10:27:16
 LOAD AVERAGE: 0.12, 0.11, 0.06
 TASKS: 656
 NODENAME: b***13
 RELEASE: 2.6.32-300.39.5.el5uek
 VERSION: #1 SMP Wed Mar 13 11:26:53 PDT 2013
 MACHINE: x86_64  (3299 Mhz)
 MEMORY: 20 GB
 PANIC: "kernel BUG at fs/buffer.c:3281!"
 PID: 8126
 COMMAND: "archiveInd**"
 TASK: ffff8804c673e680  [THREAD_INFO: ffff8803cb48e000]
 CPU: 0
 STATE: TASK_RUNNING (PANIC)

crash64> gdb set print symbol-filename
crash64> set edit emacs
edit: emacs
crash64> bt
PID: 8126   TASK: ffff8804c673e680  CPU: 0   COMMAND: "archiveInd**"
 #0 [ffff8803cb48f880] machine_kexec at ffffffff8102de3b
 #1 [ffff8803cb48f900] crash_kexec at ffffffff810982cb
 #2 [ffff8803cb48f9d0] oops_end at ffffffff81458919
 #3 [ffff8803cb48fa00] die at ffffffff8101565d
 #4 [ffff8803cb48fa30] do_trap at ffffffff8145826c
 #5 [ffff8803cb48fa80] do_invalid_op at ffffffff81013926
 #6 [ffff8803cb48fb20] invalid_op at ffffffff81012b7b
    [exception RIP: free_buffer_head+22]
    RIP: ffffffff8113f236  RSP: ffff8803cb48fbd8  RFLAGS: 00010217
    RAX: ffff8802508c9cd8  RBX: ffff8802508c9c90  RCX: 0000000000000034
    RDX: ffff8802508c9c90  RSI: 0000000000001000  RDI: ffff8802508c9c90
    RBP: ffff8803cb48fbd8   R8: 0000000000000000   R9: ffff880144cf9b00
    R10: ffff8802508c9cf8  R11: 000000000000000e  R12: ffff88024e744af0
    R13: 0000000000000001  R14: 0000000000000001  R15: ffffea000677aa38
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff8803cb48fbe0] try_to_free_buffers at ffffffff8113f719
 #8 [ffff8803cb48fc20] journal_invalidatepage at ffffffff811c22df
 #9 [ffff8803cb48fc80] ext3_invalidatepage at ffffffff811813e9
#10 [ffff8803cb48fcb0] do_invalidatepage at ffffffff810e0f6f
#11 [ffff8803cb48fcc0] truncate_inode_page at ffffffff810e14bc
#12 [ffff8803cb48fce0] truncate_inode_pages_range at ffffffff810e15ab
#13 [ffff8803cb48fdc0] truncate_inode_pages at ffffffff810e17a9
#14 [ffff8803cb48fdd0] ext3_delete_inode at ffffffff81181486
#15 [ffff8803cb48fdf0] generic_delete_inode at ffffffff8112fd50
#16 [ffff8803cb48fe20] generic_drop_inode at ffffffff8112fdc5
#17 [ffff8803cb48fe40] iput at ffffffff8112f4b8
#18 [ffff8803cb48fe60] do_unlinkat at ffffffff8112731b
#19 [ffff8803cb48ff30] sys_unlink at ffffffff811273a8
#20 [ffff8803cb48ff40] my_unlink at ffffffffa0546c76 [seos]
#21 [ffff8803cb48ff80] system_call_fastpath at ffffffff81011db2
    RIP: 0000003372cc82a7  RSP: 00007fff8af3a828  RFLAGS: 00000206
    RAX: 0000000000000057  RBX: ffffffff81011db2  RCX: 000000000000000f
    RDX: 0000000000000000  RSI: 00007fff8af39d70  RDI: 00007fff8af3a220
    RBP: 00007fff8af3a650   R8: 0000000000000007   R9: 2f3239372f322f78
    R10: 3030333832393331  R11: 0000000000000206  R12: 0000000000000000
    R13: 0000000000000000  R14: 00007fff8af49340  R15: 0000000000000000
    ORIG_RAX: 0000000000000057  CS: 0033  SS: 002b
crash64> x/s 0xffffffff8113f236
0xffffffff8113f236 <free_buffer_head+22 at fs/buffer.c:3281>:    "\017\v\353\376H\213=?\225\f\001H\211\326\350\231\336\374\377eH\213\024%\340", <incomplete sequence \343>
crash64> dis
Usage:
  dis [-rludx][-b [num]] [address | symbol | (expression)] [count]
Enter "help dis" for details.
crash64> dis ffffffff8113f236
0xffffffff8113f236 <free_buffer_head+22>:3281>: ud2   

Disassembly 를 해보자.

crash64> dis ffffffff8113f236 10

0xffffffff8113f236 <free_buffer_head+22>:3281>: ud2   
crash64> dis ffffffff8113f236 10
0xffffffff8113f236 <free_buffer_head+22>:3281>: ud2   
0xffffffff8113f238 <free_buffer_head+24>:3281>: jmp    0xffffffff8113f238 <free_buffer_head+24 at fs/buffer.c:3281>
0xffffffff8113f23a <free_buffer_head+26>:3282>: mov    0x10c953f(%rip),%rdi        # 0xffffffff82208780
0xffffffff8113f241 <free_buffer_head+33>:3282>: mov    %rdx,%rsi
0xffffffff8113f244 <free_buffer_head+36>:3282>: callq  0xffffffff8110d0e2 <kmem_cache_free at mm/slab.c:3731>
0xffffffff8113f249 <free_buffer_head+41>:3283>: mov    %gs:0xe3e0,%rdx
0xffffffff8113f252 <free_buffer_head+50>:3283>: mov    $0x12260,%rax
0xffffffff8113f259 <free_buffer_head+57>:3283>: decl   (%rdx,%rax,1)
0xffffffff8113f25c <free_buffer_head+60>:3284>: callq  0xffffffff8113f189 <recalc_bh_state at fs/buffer.c:3254>
0xffffffff8113f261 <free_buffer_head+65>:3286>: leaveq
crash64> dis 0xffffffff8113f243 20
0xffffffff8113f243 <free_buffer_head+35>:3282>: (bad) 
0xffffffff8113f244 <free_buffer_head+36>:3282>: callq  0xffffffff8110d0e2 <kmem_cache_free at mm/slab.c:3731>
0xffffffff8113f249 <free_buffer_head+41>:3283>: mov    %gs:0xe3e0,%rdx

(중략)

0xffffffff8113f273 <alloc_buffer_head+16>:3268>:        mov    0x10c9506(%rip),%rdi        # 0xffffffff82208780
0xffffffff8113f27a <alloc_buffer_head+23>:3268>:        mov    %eax,%esi
0xffffffff8113f27c <alloc_buffer_head+25>:3268>:        callq  0xffffffff8110ef0d <kmem_cache_alloc at mm/slab.c:3548>
0xffffffff8113f281 <alloc_buffer_head+30>:3269>:        test   %rax,%rax
0xffffffff8113f284 <alloc_buffer_head+33>:3268>:        mov    %rax,%rbx
0xffffffff8113f287 <alloc_buffer_head+36>:3269>:        je     0xffffffff8113f2ad <alloc_buffer_head+74 at fs/buffer.c:3276>

--- 사실 위에거는 내 것이 아니기에 굳이 디테일 하게 다룰 필요 없이,

그냥 alloc_buffer_head 에 대한 instruction 만 확인 해보았다...

(bad) 라고 떡 하니 보이길래 ㅋ

Linux Kernel Mailling List 에서 해당 코드에 대한 Memory allocation and free 관련해서

Linked list 관련 버그가 있는 것으로 나온다. 

Linux kernel 3.x 에서...

하지만, UEK R1 버젼에 속하고, EOL 이기 때문에, 패치 및 정식 개발팀

engagement 는 불가능 했다.. 따라서 UEK R2 (2.6.39-x) 커널로 업그레이드를 권장했다.

UEK R2 는 2.6.39 대역과 3.x 대역을 일부 혼합하여 포함하고 있다.

2. The other server ( same environment - same role )

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

KERNEL: 2.6.32-300.39.5.el5uek/vmlinux
DUMPFILE: 3-8657514041_vmcore
CPUS: 16
DATE: Sun Feb 23 13:09:59 2014
UPTIME: 122 days, 14:09:53
LOAD AVERAGE: 1.42, 1.03, 0.87
TASKS: 597
NODENAME: b***14
RELEASE: 2.6.32-300.39.5.el5uek
VERSION: #1 SMP Wed Mar 13 11:26:53 PDT 2013
MACHINE: x86_64  (3300 Mhz)
MEMORY: 20 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 0
COMMAND: "swapper"
TASK: ffff880523c7e1c0  (1 of 16)  [THREAD_INFO: ffff8802a46bc000]
CPU: 14
STATE: TASK_RUNNING (PANIC)

crash64> gdb set print symbol-filename
crash64> set edit emacs
edit: emacs

crash64> bt
PID: 0      TASK: ffff880523c7e1c0  CPU: 14  COMMAND: "swapper"
 #0 [ffff8800402e3990] machine_kexec at ffffffff8102de3b
 #1 [ffff8800402e3a10] crash_kexec at ffffffff810982cb
 #2 [ffff8800402e3ae0] oops_end at ffffffff81458919
 #3 [ffff8800402e3b10] no_context at ffffffff810377aa
 #4 [ffff8800402e3b60] __bad_area_nosemaphore at ffffffff8103793c
 #5 [ffff8800402e3bb0] bad_area_nosemaphore at ffffffff810379e9
 #6 [ffff8800402e3bc0] do_page_fault at ffffffff81459e23
 #7 [ffff8800402e3c20] page_fault at ffffffff81457ea5
    [exception RIP: __alloc_skb+239]
    RIP: ffffffff813b95a5  RSP: ffff8800402e3cd0  RFLAGS: 00010202
    RAX: 0000000000000000  RBX: ffff8801ceb65280  RCX: 000000000000000e
    RDX: 0000000036288840  RSI: 0000000000000000  RDI: 0000000036288840
    RBP: ffff8800402e3d10   R8: 0000000000000004   R9: ffff8801ceb65280
    R10: ffff8802744d7a80  R11: ffff88046dbbf878  R12: 0000000000000840
    R13: 0000000036288000  R14: 0000000000000840  R15: 0000000000000020
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffff8800402e3d18] ixgbe_alloc_rx_buffers at ffffffffa01b0d93 [ixgbe]
 #9 [ffff8800402e3d78] ixgbe_poll at ffffffffa01b3c08 [ixgbe]
#10 [ffff8800402e3e98] net_rx_action at ffffffff813c45e9
#11 [ffff8800402e3ee8] __do_softirq at ffffffff8105e8d1
#12 [ffff8800402e3f38] call_softirq at ffffffff81012eec
#13 [ffff8800402e3f50] do_softirq at ffffffff81014695
#14 [ffff8800402e3f70] irq_exit at ffffffff8105e756
#15 [ffff8800402e3f80] do_IRQ at ffffffff8145b8d1
--- <IRQ stack> ---
#16 [ffff8802a46bde48] ret_from_intr at ffffffff81012713
    [exception RIP: mwait_idle+116]
    RIP: ffffffff810199d6  RSP: ffff8802a46bdef8  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: ffff8802a46bdef8  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000003  RDI: ffff8802a46bc000
    RBP: ffffffff8101270e   R8: ffff8802a46bded8   R9: ffff8802a46bdda8
    R10: ffff8802e349cb40  R11: ffff88046dbbf878  R12: ffff8800402f0480
    R13: 0000000000000082  R14: 00000000964d965e  R15: 0000000e774d5de2
    ORIG_RAX: ffffffffffffff55  CS: 0010  SS: 0018
#17 [ffff8802a46bdef0] mwait_idle at ffffffff810199c9
#18 [ffff8802a46bdf00] cpu_idle at ffffffff81010d6f

그냥 딱 보면 Swapper 가 뜨는걸 보고, Memory 관련 이슈겠구나, 해야한다.

crash64> x/s 0xffffffff813b95a5
0xffffffff813b95a5 <__alloc_skb+239 at net/core/skbuff.c:215>:   "\363\253\307B8\001"

crash64> dis -rl __alloc_skb+239
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 174
0xffffffff813b94b6 <__alloc_skb>:174>:  push   %rbp
0xffffffff813b94b7 <__alloc_skb+1>:174>:        mov    %rsp,%rbp
0xffffffff813b94ba <__alloc_skb+4>:174>:        push   %r15
0xffffffff813b94bc <__alloc_skb+6>:174>:        push   %r14
0xffffffff813b94be <__alloc_skb+8>:174>:        push   %r13
0xffffffff813b94c0 <__alloc_skb+10>:174>:       push   %r12
0xffffffff813b94c2 <__alloc_skb+12>:174>:       push   %rbx
0xffffffff813b94c3 <__alloc_skb+13>:174>:       sub    $0x18,%rsp
0xffffffff813b94c7 <__alloc_skb+17>:174>:       nopl   0x0(%rax,%rax,1)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 180
0xffffffff813b94cc <__alloc_skb+22>:180>:       mov    0x7aaecd(%rip),%rax        # 0xffffffff81b643a0
0xffffffff813b94d3 <__alloc_skb+29>:180>:       test   %edx,%edx
0xffffffff813b94d5 <__alloc_skb+31>:180>:       cmove  0x7aaebb(%rip),%rax        # 0xffffffff81b64398
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 174
0xffffffff813b94dd <__alloc_skb+39>:174>:       mov    %edx,-0x34(%rbp)
0xffffffff813b94e0 <__alloc_skb+42>:174>:       mov    %edi,%r12d
0xffffffff813b94e3 <__alloc_skb+45>:174>:       mov    %esi,%r15d
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 183
0xffffffff813b94e6 <__alloc_skb+48>:183>:       mov    %ecx,%edx
0xffffffff813b94e8 <__alloc_skb+50>:183>:       and    $0xfffffffe,%esi
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 174
0xffffffff813b94eb <__alloc_skb+53>:174>:       mov    %ecx,%r13d
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 184
0xffffffff813b94ee <__alloc_skb+56>:184>:       xor    %ebx,%ebx
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 183
0xffffffff813b94f0 <__alloc_skb+58>:183>:       mov    %rax,%rdi
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 180
0xffffffff813b94f3 <__alloc_skb+61>:180>:       mov    %rax,-0x30(%rbp)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 183
0xffffffff813b94f7 <__alloc_skb+65>:183>:       callq  0xffffffff8110dc06 <kmem_cache_alloc_node at mm/slab.c:3609>
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 184
0xffffffff813b94fc <__alloc_skb+70>:184>:       test   %rax,%rax
0xffffffff813b94ff <__alloc_skb+73>:184>:       je     0xffffffff813b95e1 <__alloc_skb+299 at net/core/skbuff.c:235>
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 183
0xffffffff813b9505 <__alloc_skb+79>:183>:       mov    %rax,%rbx
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/arch/x86/include/asm/processor.h: 874
0xffffffff813b9508 <__alloc_skb+82>:874>:       prefetcht0 (%rax)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 188
0xffffffff813b950b <__alloc_skb+85>:188>:       add    $0x3f,%r12d
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 189
0xffffffff813b950f <__alloc_skb+89>:189>:       mov    %r13d,%edx
0xffffffff813b9512 <__alloc_skb+92>:189>:       mov    %r15d,%esi
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 188
0xffffffff813b9515 <__alloc_skb+95>:188>:       and    $0xffffffc0,%r12d
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 189
0xffffffff813b9519 <__alloc_skb+99>:189>:       mov    %r12d,%r14d
0xffffffff813b951c <__alloc_skb+102>:189>:      lea    0x1f8(%r14),%rdi
0xffffffff813b9523 <__alloc_skb+109>:189>:      callq  0xffffffff8110e248 <__kmalloc_node at mm/slab.c:3651>
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 191
0xffffffff813b9528 <__alloc_skb+114>:191>:      test   %rax,%rax
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 189
0xffffffff813b952b <__alloc_skb+117>:189>:      mov    %rax,%r13
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 191
0xffffffff813b952e <__alloc_skb+120>:191>:      je     0xffffffff813b95d3 <__alloc_skb+285 at net/core/skbuff.c:232>
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/arch/x86/include/asm/processor.h: 874
0xffffffff813b9534 <__alloc_skb+126>:874>:      lea    (%rax,%r14,1),%rax
0xffffffff813b9538 <__alloc_skb+130>:874>:      prefetcht0 (%rax)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 200
0xffffffff813b953b <__alloc_skb+133>:200>:      mov    $0xcc,%edx
0xffffffff813b9540 <__alloc_skb+138>:200>:      xor    %esi,%esi
0xffffffff813b9542 <__alloc_skb+140>:200>:      mov    %rbx,%rdi
0xffffffff813b9545 <__alloc_skb+143>:200>:      callq  0xffffffff8123b600
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 201
0xffffffff813b954a <__alloc_skb+148>:201>:      lea    0xf0(%r12),%eax
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 205
0xffffffff813b9552 <__alloc_skb+156>:205>:      mov    %rbx,%rdi
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/arch/x86/include/asm/atomic_64.h: 35
0xffffffff813b9555 <__alloc_skb+159>:35>:       movl   $0x1,0xec(%rbx)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 203
0xffffffff813b955f <__alloc_skb+169>:203>:      mov    %r13,0xd8(%rbx)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 204
0xffffffff813b9566 <__alloc_skb+176>:204>:      mov    %r13,0xe0(%rbx)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 201
0xffffffff813b956d <__alloc_skb+183>:201>:      mov    %eax,0xe8(%rbx)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 205
0xffffffff813b9573 <__alloc_skb+189>:205>:      callq  0xffffffff813b6eab <skb_reset_tail_pointer at include/linux/skbuff.h:1097>
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 206
0xffffffff813b9578 <__alloc_skb+194>:206>:      add    0xcc(%rbx),%r12d
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 215
0xffffffff813b957f <__alloc_skb+201>:215>:      xor    %eax,%eax
0xffffffff813b9581 <__alloc_skb+203>:215>:      mov    $0xe,%ecx
0xffffffff813b9586 <__alloc_skb+208>:215>:      cld    
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 210
0xffffffff813b9587 <__alloc_skb+209>:210>:      movl   $0xffffffff,0xc8(%rbx)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 214
0xffffffff813b9591 <__alloc_skb+219>:214>:      mov    %r12d,%edx
0xffffffff813b9594 <__alloc_skb+222>:214>:      add    0xd8(%rbx),%rdx
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 206
0xffffffff813b959b <__alloc_skb+229>:206>:      mov    %r12d,0xd0(%rbx)
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 215
0xffffffff813b95a2 <__alloc_skb+236>:215>:      mov    %rdx,%rdi
0xffffffff813b95a5 <__alloc_skb+239>:215>:      rep stos %eax,%es:(%rdi)

__alloc_skb 에 대한 Struct 를 찾아보도록 하자.

__ 가 붙은 녀석들은 대부분 symbol 또는 Kernel internal fuction 들이기 때문에,

다음과 같이 확인이 가능하다.

crash64> __alloc_skb
__alloc_skb = $8 =
 {struct sk_buff *(unsigned int, gfp_t, int, int)} 0xffffffff813b94b6 <__alloc_skb at net/core/skbuff.c:174>

crash64> sk_buff -ox
struct sk_buff {
   [0x0] struct sk_buff *next;
   [0x8] struct sk_buff *prev;
  [0x10] struct sock *sk;
  [0x18] ktime_t tstamp;
  [0x20] struct net_device *dev;
  [0x28] long unsigned int _skb_dst;
  [0x30] struct sec_path *sp;
  [0x38] char cb[48];
  [0x68] unsigned int len;
  [0x6c] unsigned int data_len;
  [0x70] __u16 mac_len;
  [0x72] __u16 hdr_len;
         union {
  [0x74]     __wsum csum;
             struct {
  [0x74]         __u16 csum_start;
  [0x76]         __u16 csum_offset;
             };
         };
  [0x78] __u32 priority;
  [0x7c] int flags1_begin[];
  [0x7c] __u8 local_df : 1;
  [0x7c] __u8 cloned : 1;
  [0x7c] __u8 ip_summed : 2;
  [0x7c] __u8 nohdr : 1;
  [0x7c] __u8 nfctinfo : 3;
  [0x7d] __u8 pkt_type : 3;
  [0x7d] __u8 fclone : 2;
  [0x7d] __u8 ipvs_property : 1;
  [0x7d] __u8 peeked : 1;
  [0x7d] __u8 nf_trace : 1;
  [0x7e] __be16 protocol : 16;
  [0x80] int flags1_end[];
  [0x80] void (*destructor)(struct sk_buff *);
  [0x88] struct nf_conntrack *nfct;
  [0x90] struct sk_buff *nfct_reasm;
  [0x98] struct nf_bridge_info *nf_bridge;
  [0xa0] int iif;
  [0xa4] __u16 tc_index;
  [0xa6] __u16 tc_verd;
  [0xa8] __u32 rxhash;
  [0xac] int flags2_begin[];
  [0xac] __u16 queue_mapping : 16;
  [0xae] __u8 ndisc_nodetype : 2;
  [0xb0] int flags2_end[];
  [0xb0] dma_cookie_t dma_cookie;
  [0xb4] __u32 secmark;
  [0xb8] __u32 mark;
  [0xbc] __u16 vlan_tci;
  [0xc0] sk_buff_data_t transport_header;
  [0xc4] sk_buff_data_t network_header;
  [0xc8] sk_buff_data_t mac_header;
  [0xcc] sk_buff_data_t tail;
  [0xd0] sk_buff_data_t end;
  [0xd8] unsigned char *head;
  [0xe0] unsigned char *data;
  [0xe8] unsigned int truesize;
  [0xec] atomic_t users;
}
SIZE: 0xf0

/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 206
0xffffffff813b9578 <__alloc_skb+194>:206>:      add    0xcc(%rbx),%r12d
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 215

조기 보이는 0xcc 는 skb_buff 구조체의 sk_buff_data_t tail 을 가르키고,

Register 12 번의 값이 저 구조체 변수에 대입되었음을 나타낸다.

/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/net/core/skbuff.c: 206
0xffffffff813b959b <__alloc_skb+229>:206>:      mov    %r12d,0xd0(%rbx)

이 OPCODE 는 R12 의 값을 다시 sk_buff_data_t end 에 넣는 것을 뜻하는 것으로,

해당 구조체에 관련하여 Linked List 가 구성되고 있음을 알 수 있다.

R12: 0000000000000840 의 주소는 ffff8800402f0480 라고 bt 결과에 나와있다.

Struct 상태를 추적해 보자.

crash64> sk_buff ffff8800402f0480
struct sk_buff {
  next = 0xffff8805208a3958,
  prev = 0x0,
  sk = 0x0,
  tstamp = {
    tv64 = 10591793682000000
  },
  dev = 0x25a12e23f32880,
  _skb_dst = 18446744071579385198,
  sp = 0xffff8800402f03c8,
  cb = "\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\240\242\a\201\377\377\377\377swapper\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000",
  len = 2,
  data_len = 0,
  mac_len = 3390,
  hdr_len = 8001,
  {
    csum = 2466094,
    {
      csum_start = 41262,
      csum_offset = 37
    }
  },
  priority = 1,
  flags1_begin = 0xffff8800402f04fc,
  local_df = 1 '\001',
  cloned = 0 '\000',
  ip_summed = 0 '\000',
  nohdr = 0 '\000',
  nfctinfo = 0 '\000',
  pkt_type = 0 '\000',
  fclone = 0 '\000',
  ipvs_property = 0 '\000',
  peeked = 0 '\000',
  nf_trace = 0 '\000',
  protocol = 0,
  flags1_end = 0xffff8800402f0500,
  destructor = 0x3774d5de2,
  nfct = 0x2dfb85dde,
  nfct_reasm = 0x1e2369cf2,
  nf_bridge = 0x0,
  iif = 564196685,
  tc_index = 41262,
  tc_verd = 37,
  rxhash = 591871167,
  flags2_begin = 0xffff8800402f052c,
  queue_mapping = 41262,
  ndisc_nodetype = 1 '\001',
  flags2_end = 0xffff8800402f0530,
  dma_cookie = 523962298,
  secmark = 2466094,
  mark = 4006113326,
  vlan_tci = 64536,
  transport_header = 591871167,
  network_header = 2466094,
  mac_header = 38941491,
  tail = 0,
  end = 2001559051,
  head = 0x3774d5e32 <Address 0x3774d5e32 out of bounds>,              <<<<<<
  data = 0x25a12e23f32880 <Address 0x25a12e23f32880 out of bounds>,    <<<<<
  truesize = 0,
  users = {
    counter = 0
  }
}

뭐 굳이 더 뭘 볼 필요도 없이, 애초에 구조체의 메모리 location 이 잘못된 번지에서

시작되었다는 것을 알 수 있다. ( out of bound )

아래는 해당 커널의 소스이다.

-- net/core/skbuff.c:215

172 struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
 173                             int fclone, int node)
 174 {             
 175         struct kmem_cache *cache;
 176         struct skb_shared_info *shinfo;
 177         struct sk_buff *skb;
 178         u8 *data;
 179
 180         cache = fclone ? skbuff_fclone_cache : skbuff_head_cache;
 181
 182         /* Get the HEAD */     
 183         skb = kmem_cache_alloc_node(cache, gfp_mask & ~__GFP_DMA, node);
 184         if (!skb)
 185                 goto out;
 186         prefetchw(skb);
 187 
 188         size = SKB_DATA_ALIGN(size);
 189         data = kmalloc_node_track_caller(size + sizeof(struct skb_shared_info),
 190                         gfp_mask, node);
 191         if (!data)
 192                 goto nodata;
 193         prefetchw(data + size);
 194               
 195         /*    
 196          * Only clear those fields we need to clear, not those that we will
 197          * actually initialise below. Hence, don't put any more fields after
 198          * the tail pointer in struct sk_buff!
 199          */
 200         memset(skb, 0, offsetof(struct sk_buff, tail));
 201         skb->truesize = size + sizeof(struct sk_buff);
 202         atomic_set(&skb->users, 1);
 203         skb->head = data;
 204         skb->data = data;
 205         skb_reset_tail_pointer(skb);
 206         skb->end = skb->tail + size;
 207         kmemcheck_annotate_bitfield(skb, flags1);
 208         kmemcheck_annotate_bitfield(skb, flags2);
 209 #ifdef NET_SKBUFF_DATA_USES_OFFSET
 210         skb->mac_header = ~0U;
 211 #endif 
 212 
 213         /* make sure we initialize shinfo sequentially */
 214         shinfo = skb_shinfo(skb);
 215         memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));     <<<<<<<<<<<<< 해당 라인이 memory allocation 라인임을 알 수 있다.
 216         atomic_set(&shinfo->dataref, 1);
 217 
 218         if (fclone) {
 219                 struct sk_buff *child = skb + 1;
 220                 atomic_t *fclone_ref = (atomic_t *) (child + 1);

 221
 222                 kmemcheck_annotate_bitfield(child, flags1);
 223                 kmemcheck_annotate_bitfield(child, flags2);
 224                 skb->fclone = SKB_FCLONE_ORIG;
 225                 atomic_set(fclone_ref, 1);
 226
 227                 child->fclone = SKB_FCLONE_UNAVAILABLE;
 228         }
 229 out:
 230         return skb;
 231 nodata:
 232         kmem_cache_free(cache, skb);
 233         skb = NULL;
 234         goto out;
 235 }
 236 EXPORT_SYMBOL(__alloc_skb);
 237

이렇게 분석은 마무리 하기로 한다... 혹시 작성중 헛갈리는 부분때문에,

앞뒤가 바뀐 부분이 있을 수 있으니 ( 난 해태눈깔이라 못찾겠음 ),

의아하면 제발 "쫌" 알려주시라고! -_-

사실, 이 과정만 보아서는 igxbe 드라이버가 Invalid data 를 넘겨주었기 때문인지,

애초에 Kernel 에서 skb allocation 에 논리적 이슈가 있었던 것인지 알 수 없으나,

어찌되었든, 둘다 커널에 포함되어 제공되는 Internal 부분이고,

Shipped driver 이며, allocation 이슈이기 때문에,

커널 Defect 로 판단되어 업그레이드나 픽스가 필요 한 상황...

(실제로 UEK2 커널의 sk_buff allocation 부분을 보면, memory allocation 체크하는 부분이

더 간결하게 정리되어 있다. )

게다가 역시나 이녀석도 UEK R1 인지라.... UEK Release 2 로 업그레이드를

하라는 것으로 가이드 할 수 밖에 없었다..........

이번엔 좀 더 Advanced 한 crash 사용법을 보여준것 같아서.....

별로 뿌듯하진 않은데, 어차피 알아듣는 사람은 안알랴줌에도 스스로  깨우칠 것이고,

깨우치려 들지 않는 자는 알랴줌에도 깨우치지 못할 것이기에....

내 "밥줄" 좀 품....막 confidential 위반이라고 머라카면,

( 아... UEK 소스라인이 문제되려나? 근데 별반 차이 없는데 Vanila 커널이랑 ㅋ )

"옴마~? 앙대욤"

2014/01/06 - [Skills/System] - Kernel dump analysis about the bug called as "devide by zero"

2012/01/01 - [Skills/mY Technutz] - Kernel Dump Analysis #10