Kernel dump analysis about the bug called as "devide by zero"

자 막간을 이용해 간만에 kernel core dump 분석 들어간다...

사실 요즘 대박대박 대박사건으로 여러 사이트에서 터지고 있는 문제라,

뭐 보안이 필요한 부분인것도 아니고, patch 가 어차피 나와있으므로
( kspice 를 이용한 hot fix 이긴 하지만..)

과감하게 공개해 본다....

      KERNEL: /home/bde/vmlinux_repo/64/2.6.32-400.21.1.el5uek/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 16
        DATE: Wed Jan  1 19:42:13 2014
      UPTIME: 2 days, 22:00:06
LOAD AVERAGE: 0.84, 0.86, 0.81
       TASKS: 1188
    NODENAME: *****db02
     RELEASE: 2.6.32-400.21.1.el5uek
     VERSION: #1 SMP Wed Feb 20 01:35:01 PST 2013
     MACHINE: x86_64  (2893 Mhz)
      MEMORY: 256 GB
       PANIC: ""
         PID: 52273
     COMMAND: "oracle"
        TASK: ffff8832ca18a100  [THREAD_INFO: ffff8832eada2000]
         CPU: 3
       STATE: TASK_INTERRUPTIBLE (PANIC)

위에서 릴리즈 버젼을 보면 2.6.32-400 이라는 Redhat 측에서 보면 황당한
릴리즈 버젼이 눈에 확 띈다... 그렇다, UEK kernel 이다..
즉, 넘버링은 2.6.32 대역으로 나와있지만, 실제로는 3.0 kernel 이 상당수
적용되어 있다는 얘기이다. 유념해 보아야 한다.

crash64> gdb set print symbol-filename
crash64> set edit emacs
edit: emacs
crash64> log

( ...Snip... )

------------[ cut here ]------------
------------[ cut here ]------------
WARNING: at kernel/sched.c:11263 sched_warn_zero_power+0x53/0x66()
Hardware name: SUN FIRE X4170 M3    
group ffff88002820fd40 cpus 0,8 cpu_power = 0
Modules linked in: mpt2sas(U) scsi_transport_sas(U) raid_class(U) nfsd(U) exportfs(U) i2c_dev(U) i2c_core(U) nfs(U) fscache(U) nfs_acl(U) auth_rpcgss(U) lockd(U) sunrpc(U) bonding(U) ipmi_poweroff(U) ipmi_watchdog(U) ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) rds_rdma(U) rds(U) ib_ipoib(U) rdma_ucm(U) rdma_cm(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) ib_cm(U) iw_cm(U) ib_addr(U) ipv6(U) ib_sa(U) video(U) output(U) sbs(U) sbshc(U) fuse(U) parport_pc(U) lp(U) parport(U) mlx4_ib(U) ib_mad(U) ib_core(U) joydev(U) snd_seq_dummy(U) snd_seq_oss(U) snd_seq_midi_event(U) snd_seq(U) snd_seq_device(U) snd_pcm_oss(U) snd_mixer_oss(U) ixgbe(U) snd_pcm(U) snd_timer(U) snd(U) soundcore(U) mlx4_core(U) dca(U) snd_page_alloc(U) ahci(U) pcspkr(U) dm_round_robin(U) dm_multipath(U) scsi_dh_rdac(U) scsi_dh_hp_sw(U) scsi_dh_emc(U) scsi_dh_alua(U) usb_storage(U) ata_piix(U) shpchp(U) megaraid_sas(U)
Pid: 52273, comm: oracle Not tainted 2.6.32-400.21.1.el5uek #1
Call Trace:
 [<ffffffff810481b4>] ? sched_warn_zero_power+0x53/0x66
 [<ffffffff8105795c>] warn_slowpath_common+0x7c/0x94
 [<ffffffff810579f3>] warn_slowpath_fmt+0x69/0x6b
bonding: bondeth0: link status definitely up for interface eth2.
ADDRCONF(NETDEV_UP): eth0: link is not ready
bonding: bondib0: link status definitely up for interface ib1.
ixgbe 0000:20:00.0: eth0: NIC Link is Up 1 Gbps, Flow Control: RX/TX
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
type=1305 audit(1388382339.757:4): audit_pid=5278 old=0 auid=4294967295 ses=4294967295 res=1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Slow work thread pool: Starting up
Slow work thread pool: Ready
FS-Cache: Loaded
FS-Cache: Netfs 'nfs' registered for caching
svc: failed to register lockdv1 RPC service (errno 97).
i2c /dev entries driver
megasas: fasync_helper was not called first
warning: `ntpdate' uses 32-bit capabilities (legacy support in use)
eth0: no IPv6 routers present
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
mxMANAGER[7939]: segfault at 0 ip 0000003e1e2783c0 sp 00007fffcbde6048 error 4 in libc-2.5.so[3e1e200000+14e000]
RDS/IB: connected to 192.168.10.52 version 3.1
RDS/IB: connected to 192.168.10.52 version 3.1
RDS/IB: connected to 192.168.10.53 version 3.1
RDS/IB: connected to 192.168.10.54 version 3.1
RDS/IB: connected to 192.168.10.55 version 3.1
mpt2sas version 12.105.11.00 loaded
RDS/IB: connected to 192.168.10.51 version 3.1
ib0: Unicast, no dst: type 0000, QPN 060000 0020:0800:1404:0001:8000:0048:fe80:0000
ib0: Unicast, no dst: type 0000, QPN 060000 0020:0800:1404:0001:8000:0048:fe80:0000
ib0: Unicast, no dst: type 0000, QPN 060000 0020:0800:1404:0001:8000:0048:fe80:0000
ib0: Unicast, no dst: type 0000, QPN 060000 0020:0800:1404:0001:8000:0048:fe80:0000
ib0: Unicast, no dst: type 0000, QPN 060000 0020:0800:1404:0001:8000:0048:fe80:0000
ixgbe 0000:20:00.1: eth1: failed to finish FDIR re-initialization, ignored adding FDIR ATR filters
------------[ cut here ]------------
------------[ cut here ]------------
WARNING: at kernel/sched.c:11263 sched_warn_zero_power+0x53/0x66()
Hardware name: SUN FIRE X4170 M3    
group ffff88002820fd40 cpus 0,8 cpu_power = 0
Modules linked in: mpt2sas(U) scsi_transport_sas(U) raid_class(U) nfsd(U) exportfs(U) i2c_dev(U) i2c_core(U) nfs(U) fscache(U) nfs_acl(U) aut
h_rpcgss(U) lockd(U) sunrpc(U) bonding(U) ipmi_poweroff(U) ipmi_watchdog(U) ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) rds_rdma(U) rds(U)
ib_ipoib(U) rdma_ucm(U) rdma_cm(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) ib_cm(U) iw_cm(U) ib_addr(U) ipv6(U) ib_sa(U) video(U) output(U) sbs(U)
sbshc(U) fuse(U) parport_pc(U) lp(U) parport(U) mlx4_ib(U) ib_mad(U) ib_core(U) joydev(U) snd_seq_dummy(U) snd_seq_oss(U) snd_seq_midi_event(
U) snd_seq(U) snd_seq_device(U) snd_pcm_oss(U) snd_mixer_oss(U) ixgbe(U) snd_pcm(U) snd_timer(U) snd(U) soundcore(U) mlx4_core(U) dca(U) snd_
page_alloc(U) ahci(U) pcspkr(U) dm_round_robin(U) dm_multipath(U) scsi_dh_rdac(U) scsi_dh_hp_sw(U) scsi_dh_emc(U) scsi_dh_alua(U) usb_storage
(U) ata_piix(U) shpchp(U) megaraid_sas(U)
Pid: 52273, comm: oracle Not tainted 2.6.32-400.21.1.el5uek #1
Call Trace:
 [<ffffffff810481b4>] ? sched_warn_zero_power+0x53/0x66
 [<ffffffff8105795c>] warn_slowpath_common+0x7c/0x94
 [<ffffffff810579f3>] warn_slowpath_fmt+0x69/0x6b
 [<ffffffff813c39c1>] ? rcu_read_unlock_bh+0xe/0x10
 [<ffffffff8123ee86>] ? bitmap_scnlistprintf+0xc0/0x10a
 [<ffffffff8104815f>] ? cpulist_scnprintf+0x13/0x15
 [<ffffffff810481b4>] sched_warn_zero_power+0x53/0x66
 [<ffffffff810481dc>] sched_group_power+0x15/0x17
 [<ffffffff81053864>] find_busiest_group+0x3e9/0x88e
 [<ffffffff81457d3a>] schedule+0x293/0x76b
 [<ffffffff813edabe>] ? ip_queue_xmit+0x301/0x371
 [<ffffffff814585c9>] schedule_timeout+0x36/0xe7
 [<ffffffff8105e035>] ? local_bh_enable_ip+0xe/0x10
 [<ffffffff81459aff>] ? _spin_unlock_bh+0x15/0x17
------------[ cut here ]------------
WARNING: at kernel/sched.c:11263 sched_warn_zero_power+0x53/0x66()
 [<ffffffff813b6557>] ? release_sock+0xb8/0xc1
Hardware name: SUN FIRE X4170 M3    
group ffff88002820fd40 cpus 0,8 cpu_power = 0 [<ffffffff813b88ee>] sk_wait_data+0x8e/0xd0
Modules linked in: [<ffffffff8107713c>] ? autoremove_wake_function+0x0/0x3d
 mpt2sas(U) [<ffffffff813f5c70>] tcp_recvmsg+0x3fb/0x99f
 scsi_transport_sas
------------[ cut here ]------------
(U)
WARNING: at kernel/sched.c:11263 sched_warn_zero_power+0x53/0x66()
 [<ffffffff813b6557>] ? release_sock+0xb8/0xc1
 raid_class
Hardware name: SUN FIRE X4170 M3    
(U)group ffff88002820fd40 cpus 0,8 cpu_power = 0 [<ffffffff8141052b>] inet_recvmsg+0x5f/0x7d
 nfsd
Modules linked in:(U) mpt2sas [<ffffffff813b27ae>] __sock_recvmsg+0x72/0x7f
 exportfs(U)(U) scsi_transport_sas [<ffffffff813b2883>] sock_aio_read+0xc8/0xdc
 i2c_dev(U)(U) raid_class [<ffffffff8111b9fa>] do_sync_read+0xe7/0x12b
 i2c_core(U)(U) nfsd [<ffffffff8103faf1>] ? scale_utime+0x1b/0x1d
 nfs(U)(U)
------------[ cut here ]------------
 exportfs [<ffffffff810551b6>] ? mmput+0x14/0xe2
 fscache
WARNING: at kernel/sched.c:11263 sched_warn_zero_power+0x53/0x66()
(U)(U)
Hardware name: SUN FIRE X4170 M3    
 i2c_dev [<ffffffff8107713c>] ? autoremove_wake_function+0x0/0x3d
 nfs_aclgroup ffff88002820fd40 cpus 0,8 cpu_power = 0(U)(U)
Modules linked in: i2c_core [<ffffffff811ec9d8>] ? security_file_permission+0x16/0x18
 auth_rpcgss mpt2sas(U)(U)(U) nfs [<ffffffff8111c1c6>] vfs_read+0xc0/0x107
 lockd scsi_transport_sas(U)(U)(U) fscache [<ffffffff8111cc67>] sys_read+0x4c/0x70
 sunrpc raid_class(U)(U)(U) nfs_acl [<ffffffff81011df2>] system_call_fastpath+0x16/0x1b
 bonding nfsd(U)
---[ end trace 24f0026fcef83411 ]---
(U)(U) auth_rpcgss ipmi_poweroff exportfs(U)(U)(U) lockd ipmi_watchdog
divide error: 0000 [#1]  i2c_dev(U)(U)SMP (U) sunrpc ipmi_devintf
 i2c_core(U)(U)
last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
(U) bonding ipmi_siCPU 3  nfs(U)(U)
(U) ipmi_poweroff ipmi_msghandler
Modules linked in: fscache(U)(U) mpt2sas(U) ipmi_watchdog rds_rdma(U) nfs_acl(U)(U) scsi_transport_sas(U) ipmi_devintf rds(U) auth_rpcgss(U)(U) raid_class(U) ipmi_si ib_ipoib(U) lockd(U)(U) nfsd(U) ipmi_msghandler rdma_ucm(U) sunrpc(U)(U) exportfs(U) rds_rdma rdma_cm(U) bonding(U)(U) i2c_dev(U) rds ib_ucm(U) ipmi_poweroff(U)(U) i2c_core(U) ib_ipoib ib_uverbs(U) ipmi_watchdog(U)(U) nfs(U) rdma_ucm ib_umad(U) ipmi_devintf(U)(U) fscache(U) rdma_cm ib_cm(U) ipmi_si(U)(U) nfs_acl(U) ib_ucm iw_cm(U) ipmi_msghandler(U)(U) auth_rpcgss(U) ib_uverbs ib_addr(U) rds_rdma(U)(U) lockd(U) ib_umad ipv6(U) rds(U)(U) sunrpc(U) ib_cm ib_sa(U) ib_ipoib(U)(U) bonding(U) iw_cm video(U) rdma_ucm(U)(U) ipmi_poweroff(U) ib_addr output(U) rdma_cm(U)(U) ipmi_watchdog(U) ipv6 sbs(U) ib_ucm(U)(U) ipmi_devintf(U) ib_sa sbshc(U) ib_uverbs(U)(U) ipmi_si(U) video fuse(U) ib_umad(U)(U) ipmi_msghandler(U) output parport_pc(U) ib_cm(U)(U) rds_rdma(U) sbs lp(U) iw_cm(U)(U) rds(U) sbshc parport(U) ib_addr(U)(U) ib_ipoib(U) fuse mlx4_ib(U) ipv6(U)(U) rdma_ucm(U) parport_pc ib_mad(U) ib_sa(U)(U) rdma_cm(U) lp ib_core(U) video(U)(U) ib_ucm(U) parport joydev(U) output(U)(U) ib_uverbs(U) mlx4_ib snd_seq_dummy(U) sbs(U)(U) ib_umad(U) ib_mad snd_seq_oss(U) sbshc(U)(U) ib_cm(U) ib_core snd_seq_midi_event(U) fuse(U)(U) iw_cm(U) joydev snd_seq(U) parport_pc(U)(U) ib_addr(U) snd_seq_dummy snd_seq_device(U) lp(U)(U) ipv6(U) snd_seq_oss snd_pcm_oss(U) parport(U)(U) ib_sa(U) snd_seq_midi_event snd_mixer_oss(U) mlx4_ib(U)(U) video(U) snd_seq ixgbe(U) ib_mad(U)(U) output(U) snd_seq_device snd_pcm(U) ib_core(U)(U) sbs(U) snd_pcm_oss snd_timer(U) joydev(U)(U) sbshc(U) snd_mixer_oss snd(U) snd_seq_dummy(U)(U) fuse(U) ixgbe soundcore(U) snd_seq_oss(U)(U) parport_pc(U) snd_pcm mlx4_core(U) snd_seq_midi_event(U)(U) lp(U) snd_timer dca(U) snd_seq(U)(U) parport(U) snd snd_page_alloc(U) snd_seq_device(U)(U) mlx4_ib(U) soundcore ahci(U) snd_pcm_oss(U)(U) ib_mad(U) mlx4_core pcspkr(U) snd_mixer_oss(U)(U) ib_core(U) dca dm_round_robin(U) ixgbe(U)(U) joydev(U) snd_page_alloc dm_multipath(U) snd_pcm(U)(U) snd_seq_dummy(U) ahci scsi_dh_rdac(U) snd_timer(U)(U) snd_seq_oss(U) pcspkr scsi_dh_hp_sw(U) snd(U)(U) snd_seq_midi_event(U) dm_round_robin scsi_dh_emc(U) soundcore(U)(U) snd_seq(U) dm_multipath scsi_dh_alua(U) mlx4_core(U)(U) snd_seq_device(U) scsi_dh_rdac usb_storage(U) dca(U)(U) snd_pcm_oss(U) scsi_dh_hp_sw ata_piix(U) snd_page_alloc(U)(U) snd_mixer_oss(U) scsi_dh_emc shpchp(U) ahci(U)(U) ixgbe(U) scsi_dh_alua megaraid_sas(U) pcspkr(U)(U) snd_pcm(U) usb_storage
(U) dm_round_robin(U)Pid: 0, comm: swapper Tainted: G        W  2.6.32-400.21.1.el5uek #1
 snd_timer(U) ata_piixCall Trace:
(U) dm_multipath(U) <IRQ>  snd(U) shpchp(U) scsi_dh_rdac(U) [<ffffffff810481b4>] ? sched_warn_zero_power+0x53/0x66
 soundcore(U) megaraid_sas(U) scsi_dh_hp_sw(U) [<ffffffff8105795c>] warn_slowpath_common+0x7c/0x94
 mlx4_core(U)
(U) scsi_dh_emc [<ffffffff810579f3>] warn_slowpath_fmt+0x69/0x6b
Pid: 22409, comm: oracle Tainted: G        W  2.6.32-400.21.1.el5uek #1
 dca(U)Call Trace:
(U) scsi_dh_alua [<ffffffff8123ee86>] ? bitmap_scnlistprintf+0xc0/0x10a
 snd_page_alloc(U) [<ffffffff810481b4>] ? sched_warn_zero_power+0x53/0x66
(U) usb_storage [<ffffffff8104815f>] ? cpulist_scnprintf+0x13/0x15
 ahci(U) [<ffffffff8105795c>] warn_slowpath_common+0x7c/0x94
(U) ata_piix [<ffffffff810481b4>] sched_warn_zero_power+0x53/0x66
 pcspkr(U) [<ffffffff810579f3>] warn_slowpath_fmt+0x69/0x6b
(U) shpchp [<ffffffff810481dc>] sched_group_power+0x15/0x17
 dm_round_robin(U) [<ffffffff8123ee86>] ? bitmap_scnlistprintf+0xc0/0x10a
(U) megaraid_sas [<ffffffff81053864>] find_busiest_group+0x3e9/0x88e
 dm_multipath(U) [<ffffffff8104815f>] ? cpulist_scnprintf+0x13/0x15
(U)
 [<ffffffff8105475e>] rebalance_domains+0x177/0x442
 scsi_dh_rdac [<ffffffff810481b4>] sched_warn_zero_power+0x53/0x66
Pid: 52385, comm: oracle Tainted: G        W  2.6.32-400.21.1.el5uek #1
(U)Call Trace:
 scsi_dh_hp_sw [<ffffffff810481dc>] sched_group_power+0x15/0x17
 [<ffffffff81054a9e>] run_rebalance_domains+0x75/0xc7
(U) [<ffffffff810481b4>] ? sched_warn_zero_power+0x53/0x66
 scsi_dh_emc [<ffffffff81053864>] find_busiest_group+0x3e9/0x88e
 [<ffffffff8105e9d9>] __do_softirq+0xd7/0x19e
(U) [<ffffffff8105795c>] warn_slowpath_common+0x7c/0x94
 scsi_dh_alua [<ffffffff81457d3a>] schedule+0x293/0x76b
 [<ffffffff8108357a>] ? tick_program_event+0x2a/0x2c
(U) [<ffffffff810579f3>] warn_slowpath_fmt+0x69/0x6b
 usb_storage [<ffffffff81458cf6>] schedule_hrtimeout_range+0xc7/0x103
 [<ffffffff81012f2c>] call_softirq+0x1c/0x30
(U) [<ffffffff8123ee86>] ? bitmap_scnlistprintf+0xc0/0x10a
 ata_piix [<ffffffff81079a70>] ? hrtimer_wakeup+0x0/0x26
(U) [<ffffffff810146d5>] do_softirq+0x46/0x89
 [<ffffffff8104815f>] ? cpulist_scnprintf+0x13/0x15
 shpchp [<ffffffff8107a33c>] ? hrtimer_start_range_ns+0x14/0x16
(U) [<ffffffff8105e85e>] irq_exit+0x3b/0x7a
 [<ffffffff810481b4>] sched_warn_zero_power+0x53/0x66
 megaraid_sas [<ffffffff8112c7ce>] poll_schedule_timeout+0x4d/0x7d
(U) [<ffffffff8145d877>] smp_apic_timer_interrupt+0x7f/0x8d
 [<ffffffff810481dc>] sched_group_power+0x15/0x17

 [<ffffffff8112cd6e>] do_sys_poll+0x2b4/0x3a8
Pid: 52273, comm: oracle Tainted: G        W  2.6.32-400.21.1.el5uek #1 SUN FIRE X4170 M3    
 [<ffffffff810128f3>] apic_timer_interrupt+0x13/0x20
 [<ffffffff81053864>] find_busiest_group+0x3e9/0x88e
RIP: 0010:[<ffffffff81053c35>]  [<ffffffff8112c898>] ? __pollwait+0x0/0xdc
 <EOI>  [<ffffffff81053c35>] find_busiest_group+0x7ba/0x88e
 [<ffffffff81457d3a>] schedule+0x293/0x76b
RSP: 0018:ffff8832eada3928  EFLAGS: 00010046
 [<ffffffff8112c974>] ? pollwake+0x0/0x60
 [<ffffffff81019a4a>] ? mwait_idle+0x74/0x7f
RAX: 0000000000200000 RBX: 0000000000000000 RCX: 0000000000000000
 [<ffffffff81458cf6>] schedule_hrtimeout_range+0xc7/0x103
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000400
 [<ffffffff8112c974>] ? pollwake+0x0/0x60
 [<ffffffff81019a3d>] ? mwait_idle+0x67/0x7f
RBP: ffff8832eada3a98 R08: 0000000000000000 R09: ffff88002824fd70
 [<ffffffff81079a70>] ? hrtimer_wakeup+0x0/0x26
R10: ffff8832eada3658 R11: 0000000000000001 R12: 0000000000000100
 [<ffffffff8112c974>] ? pollwake+0x0/0x60
 [<ffffffff81010d6f>] ? cpu_idle+0xa5/0xd4
R13: ffff88002826fd40 R14: ffff88002826fc00 R15: 0000000000000000
 [<ffffffff8107a33c>] ? hrtimer_start_range_ns+0x14/0x16
FS:  00007fe4d1a9d6e0(0000) GS:ffff880028260000(0000) knlGS:0000000000000000
 [<ffffffff811e0a46>] ? sys_semctl+0x7c6/0x910
 [<ffffffff81453143>] ? start_secondary+0x1fd/0x23c
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 [<ffffffff8112c7ce>] poll_schedule_timeout+0x4d/0x7d
---[ end trace 24f0026fcef83412 ]---
CR2: 000000073043c00e CR3: 0000003287f69000 CR4: 00000000000406a0
 [<ffffffff8107713c>] ? autoremove_wake_function+0x0/0x3d
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 [<ffffffff8112cd6e>] do_sys_poll+0x2b4/0x3a8
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 [<ffffffffa009a086>] ? dma_sync_single+0x35/0x37 [mlx4_core]
Process oracle (pid: 52273, threadinfo ffff8832eada2000, task ffff8832ca18a100)
 [<ffffffff8112c898>] ? __pollwait+0x0/0xdc
Stack:
 [<ffffffffa02d90a7>] ? rds_ib_get_mr+0x44c/0x4cc [rds_rdma]
 ffff88002826fa60 [<ffffffff8112c974>] ? pollwake+0x0/0x60
 ffff8832eada3aec [<ffffffff813b3a97>] ? move_addr_to_kernel+0x45/0x4e
 0000000280000000 [<ffffffff8112c974>] ? pollwake+0x0/0x60
 ffff8832eada3ae0 [<ffffffff813bd091>] ? verify_iovec+0x54/0x9f

<0> [<ffffffff8112c974>] ? pollwake+0x0/0x60
 0000000000000003 [<ffffffff813b31db>] ? sys_sendmsg+0x27e/0x2b4
 ffffffff00000000 [<ffffffff8107713c>] ? autoremove_wake_function+0x0/0x3d
 0000000000000000 [<ffffffff8105cc4e>] ? timespec_add_safe+0x3c/0x69
 ffff88002824fd50 [<ffffffff813b27ae>] ? __sock_recvmsg+0x72/0x7f

<0> [<ffffffff81018429>] ? read_tsc+0xd/0x25
 0000000000000000 [<ffffffff813b2883>] ? sock_aio_read+0xc8/0xdc
 0000000000000000 [<ffffffff8112c75e>] ? poll_select_set_timeout+0x61/0x84
 ffff88002820fd40 [<ffffffff813b3a97>] ? move_addr_to_kernel+0x45/0x4e
 ffff88002826fd40 [<ffffffff8112ceb9>] sys_poll+0x57/0xb8

 [<ffffffff813bd091>] ? verify_iovec+0x54/0x9f
Call Trace:
 [<ffffffff81011df2>] system_call_fastpath+0x16/0x1b
 [<ffffffff813b31db>] ? sys_sendmsg+0x27e/0x2b4
---[ end trace 24f0026fcef83413 ]---
 [<ffffffff81457d3a>] schedule+0x293/0x76b
 [<ffffffff81458370>] ? _cond_resched+0xe/0x22
 [<ffffffff813edabe>] ? ip_queue_xmit+0x301/0x371
 [<ffffffff810551b6>] ? mmput+0x14/0xe2
 [<ffffffff814585c9>] schedule_timeout+0x36/0xe7
 [<ffffffff8105cc4e>] ? timespec_add_safe+0x3c/0x69
 [<ffffffff8105e035>] ? local_bh_enable_ip+0xe/0x10
 [<ffffffff81018429>] ? read_tsc+0xd/0x25
 [<ffffffff81459aff>] ? _spin_unlock_bh+0x15/0x17
 [<ffffffff8112c75e>] ? poll_select_set_timeout+0x61/0x84
 [<ffffffff813b6557>] ? release_sock+0xb8/0xc1
 [<ffffffff8112ceb9>] sys_poll+0x57/0xb8
 [<ffffffff813b88ee>] sk_wait_data+0x8e/0xd0
 [<ffffffff81011df2>] system_call_fastpath+0x16/0x1b
 [<ffffffff8107713c>] ? autoremove_wake_function+0x0/0x3d
---[ end trace 24f0026fcef83414 ]---
 [<ffffffff813f5c70>] tcp_recvmsg+0x3fb/0x99f
 [<ffffffff813b6557>] ? release_sock+0xb8/0xc1
 [<ffffffff8141052b>] inet_recvmsg+0x5f/0x7d
 [<ffffffff813b27ae>] __sock_recvmsg+0x72/0x7f
 [<ffffffff813b2883>] sock_aio_read+0xc8/0xdc
 [<ffffffff8111b9fa>] do_sync_read+0xe7/0x12b
 [<ffffffff8103faf1>] ? scale_utime+0x1b/0x1d
 [<ffffffff810551b6>] ? mmput+0x14/0xe2
 [<ffffffff8107713c>] ? autoremove_wake_function+0x0/0x3d
 [<ffffffff811ec9d8>] ? security_file_permission+0x16/0x18
 [<ffffffff8111c1c6>] vfs_read+0xc0/0x107
 [<ffffffff8111cc67>] sys_read+0x4c/0x70
 [<ffffffff81011df2>] system_call_fastpath+0x16/0x1b
Code: ff 83 bd 5c ff ff ff 00 75 27 48 8b 85 e0 fe ff ff 48 8b 95 40 ff ff ff 48 2b 95 48 ff ff ff 8b 48 08 48 c1 e2 14 48 89 d0 31 d2 <48> f7 f1 48 89 c1 48 8b 95 e0 fe ff ff 48 89 f8 48 29 f0 48 39
RIP  [<ffffffff81053c35>] find_busiest_group+0x7ba/0x88e
 RSP <ffff8832eada3928>

crash64> bt

PID: 52273  TASK: ffff8832ca18a100  CPU: 3   COMMAND: "oracle"
 #0 [ffff8832eada35d0] machine_kexec at ffffffff8102deef
 #1 [ffff8832eada3650] crash_kexec at ffffffff8109852b
 #2 [ffff8832eada3720] oops_end at ffffffff8145a829
 #3 [ffff8832eada3750] die at ffffffff8101569d
 #4 [ffff8832eada3780] do_trap at ffffffff8145a17c
 #5 [ffff8832eada37d0] do_divide_error at ffffffff810136b4
 #6 [ffff8832eada3870] divide_error at ffffffff81012b5b
    [exception RIP: find_busiest_group+1978]
    RIP: ffffffff81053c35  RSP: ffff8832eada3928  RFLAGS: 00010046
    RAX: 0000000000200000  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000400
    RBP: ffff8832eada3a98   R8: 0000000000000000   R9: ffff88002824fd70
    R10: ffff8832eada3658  R11: 0000000000000001  R12: 0000000000000100
    R13: ffff88002826fd40  R14: ffff88002826fc00  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff8832eada3920] find_busiest_group at ffffffff81053864
 #8 [ffff8832eada3aa0] schedule at ffffffff81457d3a
 #9 [ffff8832eada3b20] schedule_timeout at ffffffff814585c9
#10 [ffff8832eada3ba0] sk_wait_data at ffffffff813b88ee
#11 [ffff8832eada3c00] tcp_recvmsg at ffffffff813f5c70
#12 [ffff8832eada3ca0] inet_recvmsg at ffffffff8141052b
#13 [ffff8832eada3cf0] __sock_recvmsg at ffffffff813b27ae
#14 [ffff8832eada3d30] sock_aio_read at ffffffff813b2883
#15 [ffff8832eada3de0] do_sync_read at ffffffff8111b9fa
#16 [ffff8832eada3f10] vfs_read at ffffffff8111c1c6
#17 [ffff8832eada3f40] sys_read at ffffffff8111cc67
#18 [ffff8832eada3f80] system_call_fastpath at ffffffff81011df2
    RIP: 0000003e1ea0d9b0  RSP: 00007fff930f03b0  RFLAGS: 00000257
    RAX: 0000000000000000  RBX: ffffffff81011df2  RCX: 0000000000000000
    RDX: 0000000000002010  RSI: 000000000dbbbe46  RDI: 0000000000000011
    RBP: 00007fff93103bc0   R8: 0000000000000008   R9: 0000000000000011
    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000002010
    R13: 000000000dbbbe46  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b
crash64>
crash64> dis ffffffff81053c35   <<
0xffffffff81053c35 <find_busiest_group+1978>:4201>:     div    %rcx

-- div 는 뒤의 레지스터로 divide 를 하라는 것인데, RCX 가 0000000000000000 이다...

역시 논리적으로 0으로 나누기를 시도하였으므로 패닉이 발생했을 것이므로,

Code 를 보아 논리적 버그가 있지 않은지 확인해 봐야 할 것이다.

이런 류의 divide 버그는 보통 divede by zero bug 라고 부른다.

논리적 헛점으로 인해 Register 및 변수에 생각지도 못한 값이 들어가게 되어,

연산 오류를 일으키는 것이라고 한다. 코드를 살펴보자.

  -> kernel/sched.c
  4165 static inline void calculate_imbalance(struct sd_lb_stats *sds, int
  this_cpu,
  4166                 unsigned long *imbalance)
  
  4186         if (!sds->group_imb) {
  4187                 /*
  4188                  * Don't want to pull so many tasks that a group would
  go idle.
  4189                  */
  4190                 load_above_capacity = (sds->busiest_nr_running -
  4191                                                
  sds->busiest_group_capacity);
  4192
  4193                 load_above_capacity *= (SCHED_LOAD_SCALE *
  SCHED_LOAD_SCALE);
  4194
  4195                 load_above_capacity /= sds->busiest->cpu_power;
  4196         }

원인은 저기 보이는 sd_lb_stats 구조체 멤버중 busiest 의 cpu_power  값이

0으로 나와버리는 문제인데, 상세로직은 아래 커널메일링 을 참조하라.
https://lkml.org/lkml/2011/7/20/217

개발팀 말로는, 쉽게 설명해서, cpu loadbalancing 을 위해 cpu_power 를 계산하는
경우의 수는 장비 및 상황마다 조금씩 case by case 가 필요 할 수도 있기 때문에,
해당 자료를 수집하여 그 case 를 추가하는 것들이 필요하다고 한다.

현재 Fixed kernel 버젼은 V2.6.32-400.34.1 이며, 아직 QA/Release ready 이다.

어렵다.. 아무튼, 해당 패치는 나와 있는 상태이지만, 릴리즈 된 각 커널마다,
또한 장비상황등등 마다 재발 가능성은 언제든지 있는 부분이기 때문에,
최신 official released kernel 로 업데이트 한후, ksplice 를 통한
추가적인 hot fix 를 적용해야 한다는 점!

ksplice 를 이용하면 커널에 대한 official release 가 나와있지 않은 상태에서,
리붓 없이 수정된 kernel space 로 교체, 사용이 가능하다.. 쯔앙!

끝. 짧고 간단하며, 일이 있는 상태에서도 이정도 글은 뭐.. 가능하다 ㅠㅠ
비슷한 이슈 보이는 Oracle Linux UEK 커널 사용자는 업데이트 꼭 하도록 하자.

2012/01/01 - [Skills/mY Technutz] - Kernel Dump Analysis #10