[University of Arkansas][Computing Services]

Disaster Recovery Plan
COMP System Recovery
(DRPCX001)

Last update: Tuesday, 21-Mar-2000 10:31:22 CST

IMPORTANT NOTE!

PREFACE

Instructions in this document are intended to be of a general nature wherever possible. A moderate amount of expertise in UNIX systems administration, with particular experience in support of Solaris 2.x. If local expertise is unavailable, contract support will be available from Sun Microsystems.

I. HARDWARE PLATFORM

Sun Ultra-Enterprise 5000 or functional equivalent, configured to the following specifications:


      Sun Part#      Qty         Description

       E5000          1      Enterprise 5000 System Cabinet,
                             8-slot card cage, one SunCD4,
                             two power/cooling modules, Solaris server license,
                             no processors, no memory,
                             no CPU/Memory Boards, no I/O boards

      2600A           4      Enterprise CPU/Memory board

      2500A           8      167MHz UltraSPARC CPU module, 512K external cache

      7022A           4      256MB memory expansion kit

      2610A           2      SBus I/O board

      954A            2      Additional power/cooling module, 300W

      791A            2      37.8GB SPARCstorage Array Model 112 with 18
                             2.1GB 7200 RPM Fast/Wide SCSI-2 SC disks and
                             fiber channel cable

      6206A           1      14GB 8mm internal tape drive

      595A            2      Fiber channel optical module

      1063A           1      SBus single-ended fast/wide intelligent SCSI-2 adapter

      3800A           1      Power cord for Enterprise System (NEMA L6-30P plug)

      X1023A          1      Sun FDDI 4.0 single-attach SBus adapter

      X6071A          3      140-280 GB SPARCstorage DLT4700 auto-loader,
                             desktop enclosure with SCSI cable

      non-SUN         1      VT-100 Compatible ASCII terminal with null-modem cable
                             for use as system console.

 Note: Part numbers are taken from the July 23, 1996 version of
       "Sun Microsystems Computer Company U.S. End User Price List".

 Note: This configuration differs slightly from the current production
       configuration. The current production configuration has 50.4GB
       current production configuration has 50.4GB of disk in two SSA112
       devices, plus another 17.4GB arranged as 6-2.9GB disks in a
       Differential/Wide SCSI disk tray. IF I had to rebuild from
       from scratch, I'd recommend allocating all DASD in a pair of
       SSA devices.
II. VENDOR CONTACT INFO:
       1.  Sun Microsystems (Hardware, Operating System, and general support)
 
               Tim Simmons (Account Rep)
               Sun Microsystems Computer Corp.
               5865 Ridgeway Center Parkway
               Suite 300
               Memphis, TN  38120
               (901) 763-3964 (voice)
               (901) 680-9951 (fax)
               tim.simmons@east.sun.com
               http://www.sun.com

      2.  OpenVision (Net*Backup, Axxion-HSM, and Volume Manager)
               OpenVision
               7133 Koll Center Parkway
               Pleasanton, CA  94566
               (510) 426-6400 (voice)
               (510)  426-6486 (fax)
               1-800-223-OPEN (Customer Support)
               support@ov.com
               http://www.ov.com
 
     3.  SAS
 
               SAS Institute, Inc.
               SAS Campus Drive
               Cary, NC  27513
               (919) 677-8003 (Customer Service)
               (919)  677-8008 (Technical Support)
               http://www.sas.com
 
      4.  SPSS
 
               SPSS, Inc.
               444 North Michigan Avenue
               Chicago, IL  60611
               (312) 329-3410 (Support voice line)
               (312) 329-3668 (Support fax)

      5.   LISREL
 
               Scientific Software, Inc.
               1525 E. 53rd Street, Suite 906
               Chicago, IL  60615
               (312) 684-4920 (voice)
               (312) 684-4979 (fax)
 
      6.   BMDP
               BMDP Statistical Software, Inc.
               12121 Wilshire Boulevard, Suite 300
               Los Angles, CA  90025
               (310) 207-8800 (voice)
               (310) 207-8844 (fax)
               support@bmdp.bmdp.com
 
      7.   uniREXX and uniXEDIT (REXX and XEDIT ports for Solaris)
               The Workstation Group
               1900 North Roselle Road, Suite 408
               Schaumburg, IL  60195
               800-228-0255 (voice)
               847-781-8573 (fax)
               http://www.wrkgrp.com

III. INSTALLATION MEDIA AND MATERIALS CHECKLIST

1. Solaris 2.5.1 (or later) server media kit. This list assumes Solaris 2.5.1 .

2. Sun language products installation CD-ROM. Current version is "SunSoft WorkShop Volume 4, Number 2". Required products are:

3. Supporting code and drivers for the SPARCstorage array. Current version is "SPARCstorage Array Software and SPARCstorage Volume Manager 2.1.1" (The current version of this software should ship with new SPARCstorage Arrays)

4. Supporting code and drivers for the Sun FDDI SBus Adapter 4.0. The current version of this CD should ship with new FDDI adapter(s).

5. Current install media for OpenVision Net*Backup, HSM, and Volume Manager. Version/Release numbers for this product vary according to the supported system level. Currently, under Solaris 2.5.1, we are using version 2.1GA of Net*Backup and Volume Manger, and version 2.1 of HSM. New media is available in either 8mm tape or CD-ROM format.

6. Current install media for SAS. Obtain this from the vendor. At present, we are running SAS 6.11.

7. Current install media for SPSS. Obtain this from the vendor. At present, we are running SPSS 6.1 for SOLARIS. This product uses platform-specific licensing, and will require you provide the hardware host id for the new CPU.

8. Current install media for LISREL. Obtain this from the vendor. At present, we are running LISREL 8.12 and PRELIS 2.12 for Solaris 2.3+. This product uses platform-specific licensing, and will require you to provide the hardware host id for the new CPU.

9. Current install media for BMDP. Obtain this from the vendor. At present, we are running BMDP 7.1.

IV. POWER & NETWORK CONNECTIVITY

The Ultra-Enterprise 5000 will require one NEMA L6-30P power drop (240V, single-phase). Additional power connections for the system console and DLT-4700 tape autoloaders must be provided. Two 20-amp 120V circuits with eight outlets per circuit are in place in the ADSB machine room for this application. A minimum of four outlets across two 20-amp circuits should be provided.

The hardware list in section I assumes that the system will be attached to the campus network via FDDI. If this is not the case, then each 2610A SBus I/O board provides a single 10/100-BASE-T EtherNet attachment port. Obviously a network connection of some sort has to be available before the system can be brought on-line to the network…

V. FILE SYSTEM LAYOUT

Boot volume (install defaults to first drive of first SSA):

Partition

Slice

Size

Comments

/ 064MB
/usr 6450MB
/opt 5512MB
/export/maint 7394MB(Instead of /export/home...)
/var 3256MB(Temporary - moves to 3G SSA volume)
swap 1350MB

The initial boot volume configuration assumes that you're installing a complete bootable system on a single SSA disk. In the current configuration, the boot volume is not encapsulated for management by the SSA volume manager, although this can be changed if enough disk is available to mirror the boot volume during recovery. Refer to the volume manager documentation if you want to consider encapsulating the boot disk.

After installation of the operating system, the /var partition has to be enlarged in order to provide sufficient disk space for logging, auditing, and accounting data as well as to provide enough temporary space under /var/tmp for regular use. At present, the production /var partition is a 3GB striped (not RAID-5) volume split across eight (8) disk slices between both SSA's. The file system is split over both arrays due to the extremely high i/o rates encountered during peak use periods; it's used as a simple striped file system to avoid unnecessary RAID-5 overhead. To move the /var partition to this configuration, create a suitable striped volume through the volume manager interface (see the Volume Manager documentation). Then,

Create other SSA volumes as follows:

Filesystem

Format

Size

Comments

/opt1 striped 4152000K shown as 'compvar' in SSA layout
/var/mail RAID-5 5400M shown as 'd041mail' in SSA layout
/export/home RAID-5 2560032K shown as 'home' in SSA layout
/export/home1 RAID-5 2560032K shown as 'home1' in SSA layout
/export/home2 RAID-5 2560032K shown as 'home2' in SSA layout
/export/home3 RAID-5 2560032K shown as 'home3' in SSA layout
/export/home4 RAID-5 2560032K shown as 'home4' in SSA layout
/export/home5 RAID-5 2560032K shown as 'home5' in SSA layout
/export/home6 RAID-5 2970M was 2.9G disk on old configuration
/export/home7 RAID-5 2970M was 2.9G disk on old configuration
/export/home8 RAID-5 2970M was 2.9G disk on old configuration
/export/HSM striped 4554M shown as 'hsm' in SSA layout
/opt1/sastmp striped 1500M was slice of 2.9G disk on old config

An additional 1GB disk slice should be allocated on another SSA disk for use as swap space.

Run 'newfs' against each logical volume to lay down new file systems. Current file systems were created with

 comp# newfs -m 1 -c 32 /dev/vx/rdsk/volname

Additional steps are required to create the HSM file system. This procedure is documented in the Openv*HSM manuals appropriate to the version being installed on the new system. All volumes are initialized and mounted as 'ufs' file systems except /export/HSM, which requires further setup for use as an HSM file system before it can be restored.

Create a symbolic link that redirects /opt2 to /opt1. This is necessary in order to ensure that CRSP data remains available to programs with an old path name embedded in them.

Create symbolic links that redirect /usr/openv/netbackup and /usr/openv/volmgr to /opt1/openv/netbackup and /opt1/openv/volmgr. It's necessary to relocate these two products to the /opt1 file system due to space requirements.

VI. OFF-SITE BACKUP INFORMATION

Off-site backup tapes for comp.uark.edu are stored with off-site backups of the IBM mainframe systems, and consist of:

Current backup schedules produce a full system backup after 12:01a.m. on Mondays. The off-site media is produced through the OpenVision Net*Backup "duplicate media" procedure.

IMPORTANT NOTE!

BE SURE THAT THESE PIECES OF MEDIA HAVE THE WRITE PROTECT TAB SET TO THE READ-ONLY POSITION BEFORE USE.

VII. RECOVERY OUTLINE

  1. Obtain new equipment as described in Section I.

  2. Assess "physical plant" facilities at the cold site; verify that correct power and data connectivity is in place (One 30A 240V, at least four 20A/120V; ethernet, fast ethernet, or FDDI; air conditioning, rack or table space, cables, etc).

  3. Connect the new system, tape drives, storage arrays, and network.

  4. Make sure you know the IP address and domain name assigned to the new system before proceeding.

  5. Complete any vendor-prescribed pre-installation diagnostics.

  6. Install Solaris and SSA support code per instructions.

  7. Create SSA volumes and file systems. (Refer to Section IV).

  8. Create enlarged /var partition (Refer to Section IV for detailed instructions).

  9. Install OpenVision Net*Backup, Volume Manager, and HSM per instructions. The vendor should be able to provide additional consultation on this procedure if necessary.

  10. Install the Sun language compilers. Components of these are tracked through the software package management tools, and you can't maintain them later if you just restore them from backups. The four major packages listed in Section III, item 2 (C, C++, Pascal, and FORTRAN) plus auxiliary support are all installed under the '/opt' directory. You may also wish to reload relevant sections of the on-line 'Answer Book' documentation at this point.

  11. Following directions in the Net*Backup documentation, restore the NetBackup catalog. The disaster recovery procedure outlined in the OpenVision documentation provides a couple of alternatives depending upon the exact recovery scenario; use your best judgment on the recovery procedure that best fits the situation. Your ultimate goal here is to restore the OV volume manager and backup products, along with the catalog data that ultimately goes under the NetBackup installation. On the current system configuration, this is stored under /opt1/openv/netbackup/db (and symbolically linked from /usr/openv/netbackup).

  12. If not previously done, prepare /export/HSM as an HSM file system per instructions in the OpenVision documentation.

  13. Restore the remaining file systems via Net*Backup. Your restoration checklist should include:

    --- except /opt1/openv/* ---

  14. Examine the contents of /etc/rc3.d and /etc/rc2.d on the backup tapes. Startup scripts for some third-party license managers as well as system fine-tuning scripts may be loaded here. This can change from one release to the next of the system and of add-on products, so apply "best judgment" - or, if in doubt, refer back to the backups if add-on products fail because license management software is not running.

  15. Perform post-installation setup of desired add-on packages Source for many of these can be found under /export/maint, though you should check your favorite web search engine to see if later versions are available. Additional things to examine include:

  16. SECURE THE OLD BACKUP TAPES AND CATALOG DATA JUST IN CASE YOU HAVE TO START OVER WITH ANY OF THIS.

  17. As soon as possible, re-implement routine backups. It should be possible to proceed with normal operations at this point.

It's difficult to offer a precise estimate of time required for "cold recovery" of comp.uark.edu under these circumstances. Once new hardware is received, a minimum of three working days should be expected. The time required will vary according to the ability of the person performing system recovery and the speed of the tape subsystem used to perform restores.

APPENDIX

  1. Current SPARCstorage Array File System Layout

    Layout diagrams for existing SPARCstorage Array (SSA) configurations are provided as examples only. In a new system configuration, it's probably best to allow the volume management software to determine appropriate physical layout of volumes.



[Home Page] [Table of Contents] [Send Mail]
Copyright © 1997 University of Arkansas
All rights reserved