For being sometimes called at night for work, I know this could be of two case: A little problem that could be solved on the phone, or a big mess that needs to be cleaned all the night long.
This one was a big mess.
A big solaris machine who was hosting a buch of zones had lost the disk of one of them.. causing the zone to be down. Veritas showed the disk properly online with two enabled configuration zones
On that disk. Until this, everything fine.
The problem was that the diskgroup associated with the disk could not be auto-imported when Veritas was starting. It was yelling a “Cannot auto-import group: Disk group has no valid configuration copies”.
OK, after spending 3Hours pondering what could we do to make this disk being imported, I saw some kind of weird messages in the veritas daemon debug mode:
VxVM vxconfigd DEBUG V-5-1-556 Disk XXXX, copy 1: Block 1: Invalid magic number VxVM vxconfigd DEBUG V-5-1-5522 IMPORT(vs4c9-dg): Try a different config copy VxVM vxconfigd DEBUG V-5-1-5324 dasup_dg_unjoin: config_db_disable() VxVM vxconfigd DEBUG V-5-1-5278 config_db_disable: disable config copy #1 VxVM vxconfigd DEBUG V-5-1-681 IOCTL KLOG_FREELOG drid=1024.53 copy=0: failed: errno=22 (Invalid argument) VxVM vxconfigd DEBUG V-5-1-5602 log_disable_copy: kernel_delete_copy failed, errno=22
Owwwwkay.. This looks like an altered disk.. But how the hell this could have happened ? While looking for a replacement for this disk, to make the restore (after all, the last resort solution), I stumbled upon disksuite yelling the following thing:
d42 2 1 c0t1d0s2 \ 1 c0t2d0s2
Ouch, the disk from veritas was also c0t2d0…
So, someone has added a disk inside veritas and used it, although it was already used by disksuite… the result of this was pretty scary, as both veritas and disksuite were writting to the disk at the same times. More than that, disksuite was mirrored accross four disk, but veritas wasn’t.
Finally, We have unsetup the disk in veritas and synced the softpartition in disksuite.
So, this time, disksuite has won the game. (as we have lost less data from disksuite, veritas disk was unusable after all)
