synapseninferno

Recently, I was at a customer site to upgrade a SunFire E2900 SPARC based server: installing two fiber channel HBAs, connecting two Oracle StorEdge ST2540 disk arrays, and install an up-to-date patch set.

After breaking up the system mirror, we started patching. During patch 28 of 188 patches (recommended patch cluster), the system caught a kernel panic, obviously caused by some UFS problems. First idea: boot into single user and see, what happens:

{0} ok boot -s
reseting...
[...]
Rebooting with command: boot 
TL = 1, TT = 10. ERROR: Illegal Instruction
TSTATE= 0x1400 [ccr = 0x0, asi = 0x0, pstate = 0x14, cwp = 0x0]
TPC= 0000000000006004
TNPC= fffffffffd159118
TICK= 8000004be5bbfe34, TICKCMP = 8000000000000000

debugger entered
{0} ok

*ouch* Well then... there is a mirrored system disk, so let's boot that one:

{0} ok reset-all
resetting...
[...]
{0} ok boot disk1 -s
WARNING: rmalloc_picky: Invalid numeric argument: size = 0x0
WARNING: rmalloc_picky: Invalid numeric argument: size = 0x0
WARNING: rmalloc_picky: Invalid numeric argument: size = 0x0
WARNING: rmalloc_picky: Invalid numeric argument: size = 0x0
WARNING: rmalloc_picky: Invalid numeric argument: size = 0x0
WARNING: rmalloc_picky: Invalid numeric argument: size = 0x0
Boot load failed
{0} ok

Looks, like somebody forgot to installboot(1M) a valid bootblock on the mirror disk. Bad luck. Next, we tried to boot from a Solaris 10 DVD:

{0} ok reset-all
resetting...
[...]
{0} ok boot cdrom -s
Boot device: /pci@18,700000/pci@4/ide@2/cdrom@0,0:b File and args: -s
SunOS Release 5.10 Version Generic 64-bit
Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
ide: no major number
Cannot load drivers for /ssm@0,0/pci@18,700000/pci@4/ide@2/cdrom@0,0:b
Can't load the root filesystem

{0} ok 

Well then... even more bad luck. We then tried several things to get the machine back online. Swapping disks, removing HBAs and such. Nothing, that helped. Around 9pm we decided to stop at this point and get some sleep.

Next day, we brought another – more current – Solaris 10 DVD with us. You never know... And we also fetched the install server notebook from our office. Booting the machine from net was easy. Then a fsck(1M) on the system disk. Oh well... the filesystem looked really damaged. Lots of duplicate inodes and lots of purged directory entries. Including pretty important system files. Another attempt to boot the repaired filesystem ended up in the same RED_state alert as above.

Well... second boot from net, fscking the boot mirror, writing a boot block on it and hey! The machine boots off the system mirror. Neat. Of course, fsck purged some important entries on this machine as well. We checked for missing file by crosschecking the files on disk against the package database with pkgchk(1M), and copied those files – as much as possible – from another Solaris 10 system with same patchlevel. Some device nodes were created by another reconfigure boot and finally we could start installing patches on the system. Which even fixed some more missing files.

A couple of hours later, the system was up and running, the new LUNs were configured in the storage and a zpool(1M) was created. Another happy customer. The broken system disk will be replaced by a spare disk in a couple of days. I won't trust the old disk anymore, although it might not be broken at all.

Lessons learned: UFS is a pretty robust filesystem and we still prefer Solaris Volume Manager on system disks. Makes some troubleshooting just much simpler.

root@etain> uname -a
SunOS etain 5.10 Generic_118822-02 sun4u sparc SUNW,Ultra-2

root@etain> cat /etc/release 
                         Solaris 10 3/05 s10_74L2a SPARC
           Copyright 2005 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                            Assembled 22 January 2005

root@etain> /usr/platform/sun4u/sbin/prtdiag | head -12
System Configuration:  Sun Microsystems  sun4u Sun Ultra 2 UPA/SBus (2 X UltraSPARC-II 296MHz)
System clock frequency: 99 MHz
Memory size: 1024 Megabytes

========================= CPUs =========================

                    Run   Ecache   CPU    CPU
Brd  CPU   Module   MHz     MB    Impl.   Mask
---  ---  -------  -----  ------  ------  ----
 0     0     0      296     2.0   US-II    1.1
 0     1     1      296     2.0   US-II    1.1

root@etain> uptime
 11:29pm  up  3:40,  3 users,  load average: 0.02, 0.02, 0.13

root@etain> df -lh
Filesystem             size   used  avail capacity  Mounted on
/dev/md/dsk/d0         7.9G   5.4G   2.4G    70%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   1.7G   920K   1.7G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
fd                       0K     0K     0K     0%    /dev/fd
/dev/md/dsk/d30        994M   148M   786M    16%    /var
swap                   1.7G     0K   1.7G     0%    /tmp
swap                   1.7G    40K   1.7G     1%    /var/run
/dev/md/dsk/d60        6.7G    28M   6.6G     1%    /export
/export/home/stief     6.7G    28M   6.6G     1%    /home/stief
/export/home/root      6.7G    28M   6.6G     1%    /home/root

Seit knapp vier Stunden läuft sie also, die Ultra 2 hier unter dem Schreibtisch. Soll noch dieses Jahr Web- und Mailserver von der uralten Debian-Box übernehmen. Letztere braucht dann nur noch routen, böse Buben und Mädchen draußen halten und ein bisschen Address-Translation machen. Ich hab' endlich ein paar Gigabytes Platz für Mails und muss nicht dauernd Mailboxen zippen. Und der MUA läuft sicher auch deutlich schneller auf der Maschine. Und ich hab' endlich ein Solaris 10 zum Spielen zu Hause.

etain ist übrigens eine keltische Mondgöttin.

Hurra! Es läuft! Solaris 10 auf einer Ultra2. Build 63, entspricht Solaris Express 8/04:

ok reset-all
Resetting ... 

screen not found.
Can't open input device.
Keyboard not present.  Using ttya for input and output.

Sun Ultra 2 UPA/SBus (2 X UltraSPARC-II 296MHz), No Keyboard
OpenBoot 3.25, 1024 MB memory installed, Serial #8990760.
Ethernet address 8:0:20:89:30:28, Host ID: 80893028.

Initializing Memory
{0} ok boot
Boot device: disk  File and args: 
-
SunOS Release 5.10 Version s10_63 64-bit
Copyright 1983-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
-
configuring IPv4 interfaces: hme0.
add net default: gateway 212.185.24.161
Hostname: lugh
The system is coming up.  Please wait.
starting rpc services: rpcbind done.
Setting netmask of hme0 to 255.255.255.224
Setting default IPv4 interface for multicast: add net 224.0/4: gateway lugh
syslog service starting.
volume management starting.
The system is ready.

lugh console login:

War ein bisschen Mühe: Die Installation an der U2 kam nicht über das Preinstall-Script am Ende des SysID-Dialogs drüber. Also Installation auf einer E450. Das klappte. Problem: Eine E450 ist PCI based, die U2 ist eine SBus-Maschine. Also sind die Device-Pfade gründlich anders. Deswegen Platte in die U2, von CD in Single User booten, Platte mounten und Devices neu schreiben:

# mount /dev/dsk/c0t0d0s0 /mnt
# devfsadm -v -r /mnt
# umount /mnt
# init 6

Cool, Maschine kommt hoch. Allerdings findet sie kein Netzwerk. Verdacht: /etc/path_to_inst passt nicht zur Hardware. Also neu bauen:

# cp /etc/path_to_inst /etc/path_to_inst.Sol10ORIG
# echo "#path_to_inst_bootstrap_1" > /etc/path_to_inst
# init 6

Und siehe da: Netzwerk geht. Irgendwas mault das System noch von wegen
Permission denied auf /dev/null. Tatsächlich:
Permissions auf dem Linkziel unterhalb /devices sind
falsch. Wie auch immer das passieren kann. Also nochmal reparierend
eingreifen:

# pkgchk -af
# init 6

Und schon läuft das:

lugh [/]# uname -a
SunOS lugh 5.10 s10_63 sun4u sparc SUNW,Ultra-2
lugh [/]# cat /etc/release 
                          Solaris 10 s10_63 SPARC
        Copyright 2004 Sun Microsystems, Inc.  All Rights Reserved.
                     Use is subject to license terms.
                          Assembled 14 July 2004

Jetzt geht's dran, Zones, ZFS und DTrace zu erkunden. I'll keep you informed.

Und ja, ich hätte mir einige init 6 sparen können. Ich bevorzuge aber die Step-by-Step Variante. Typischerweise weniger fehleranfällig.

lugh ist übrigens der keltische Sonnengott (oder auch der druidische Gott der Weisheit).