on call { digital seance

Communing with the machine spirits of a sick machine is a lot like peeling back the lid on a tupperware you know has been sitting in the back of the fridge for a long time. You never know what you are going to get and often it stinks. You establish the connection and start sifting the data you get things like this:

Mar 23 04:58:24 2007 ^Mpanic[cpu450]/thread=2a10096bd40: BAD TRAP: type=31 rp=2a10096ba00 addr=91 mmu_fsr=0 occurred in module "unix" due to a NULL pointer dereference
Mar 23 04:58:24 2007
Mar 23 04:58:24 2007 sched: trap type = 0x31
Mar 23 04:58:24 2007 addr=0x91
Mar 23 04:58:24 2007 pid=0, pc=0x10395a0, sp=0x2a10096b2a1, tstate=0x8800001606, context=0x0
Mar 23 04:58:24 2007 g1-g7: 1048f2c, 141e000, 147d738, 3001edf2000, fffffffffffffffd, 0, 2a10096bd40
Mar 23 04:58:24 2007
Mar 23 04:58:24 2007 000002a10096b720 unix:die+80 (31, 2a10096ba00, 91, 0, 0, 0)
Mar 23 04:58:24 2007 %l0-3: 0000000000000000 0000000001423f58 000002a10096ba00 000002a10096b8f0
Mar 23 04:58:24 2007 %l4-7: 0000000000000031 000003002379b100 000003000e4553b0 0000000001038c00
Mar 23 04:58:24 2007 000002a10096b800 unix:trap+8e4 (2a10096ba00, 0, 10000, 10200, 0, fffe)
Mar 23 04:58:24 2007 %l0-3: 0000000000000001 0000000000000000 0000000001489bd8 0000000000000031
Mar 23 04:58:24 2007 %l4-7: 0000000001489ac0 0000000000000001 0000000000000000 0000000000000000
Mar 23 04:58:24 2007 000002a10096b950 unix:ktl0+48 (91, 3001edfa000, 30025798048, 1489bd8, 16, 0)
Mar 23 04:58:24 2007 %l0-3: 0000000000000007 0000000000001400 0000008800001606 000000000102ec9c
Mar 23 04:58:24 2007 %l4-7: 0000000001489800 0000030051e9a316 0000000000000000 000002a10096ba00
Mar 23 04:58:24 2007 000002a10096baa0 unix:setbackdq+29c (0, 0, 1489bd8, 1489bd8, 0, 0)
Mar 23 04:58:24 2007 %l0-3: 0000000001049ed0 0000000000000001 000003001edfa000 ffffffffffffffff
Mar 23 04:58:24 2007 %l4-7: 000003001edfa090 0000000000000003 0000000000000091 00000000014ec800
Mar 23 04:58:24 2007
Mar 23 04:58:25 2007 syncing file systems...
Mar 23 04:58:25 2007 ^Mpanic[cpu450]/thread=2a10096bd40: BAD TRAP: type=34 rp=1487830 addr=79 mmu_fsr=0
Mar 23 04:58:25 2007 dumping to /dev/md/dsk/d20, offset 8390377472, content: kernel
Mar 23 04:58:28 2007
Mar 23 04:58:28 2007 ^Mpanic[cpu450]/thread=2a10096bd40: BAD TRAP: type=34 rp=14860f0 addr=79 mmu_fsr=0
Mar 23 04:58:28 2007 dump aborted: please record the above information!
Mar 23 04:58:30 2007 rebooting...
Mar 23 04:58:33 2007 ERROR: CPU RED_State Reset has occurred.
Mar 23 04:58:33 2007 OBP->SC mailbox is busy, can't send.
Mar 23 04:58:33 2007 Failed to send Set Reboot Info mailbox command
Mar 23 04:58:33 2007 Resetting...

You then need to start stepping through the cryptic messages and start piecing together seemingly different events like a digital Sherlock Holmes. Its one of the aspects of my job I love. The mystery. The hunt for the truth. The interaction with the vendor. The sexy underwear.