DRPWWW01: WWW site server

Disaster Recovery Plan
WWW site server
(DRPWWW01)

Last update: Tuesday, 21-Mar-2000 10:35:57 CST

System: cavern.uark.edu

Cavern is the primary WWW site/server for the U of A campus, primarily supporting the UARKinfo CWIS (Campus Wide Information System) effort. Over 400 information provider accounts reside on this machine. Also, cavern hosts the primary anonymous FTP server nic.uark.edu.

See DRPWWW50: WWW Server Relationships for information on how the various web servers interact.

Base: SparcServer 20 model 50
Memory: 160 Meg
Disk: 10 Gig
Tape: 4mm DAT

Power & Space requirements

5 Physical units requiring 5 110Vlt outlets.

Component parts Hardware.

(Initial Specs taken from Sales Quote: ME-JK-6717-B)

Part Description Original Cost
------------------------------------------------------------------------------------------------------

S20S-51-32-P69 SPARCserver 20 Model 51 with 60MHz $8,697 SuperSPARC Processor with SuperCache 32 Mbytes, 1.05 Internal Fast SCSI-2 disk 644-Mbyte Internal SunCD Plus X3540A Type 6 Country Kits for U.S. and Canada only UNIX X327A 17-inch Color Monitor, 4-Mbyte SX Frame $2,340 Buffer, Cable and Auxiliary Video Connection Card. X569A 4.2-Gbyte Multi-Disk Pack (2 x 2.1-Gbyte Fast $2,940 SCSI-2 Drives. X567A 2.1-Gbyte Fast SCSI-2 Desktop Disk Pack $1,470 X822A 5-Gbyte 4mm Tape Desktop Storage Pack $1,200 X164P 64-Mbyte Memory Expansion (64-Mbyte SIMM) * $2,820 SOLS-C Solaris 2.X Media Kit Only CD-ROM for all $60 Server SPARCsystems

* An additional 64-Mbute SIMM was installed in March 96 bringing the total expansion SIMMs to two 64-Mbyte SIMMs.

Total disk space is now 10 Gig. The original system disk, the internal 1 Gig unit was replaced with a 2 Gig unit and a addition internal 2 Gig unit has been installed.

Component parts needed for remote recovery.

Current configuration

Software
OS: Solaris 2.5.1 CD
All system functions can be restored from tape after the designated machine has been attached to the network and a suitable tape drive is available.

Hardware:
At least 8 gig of available storage is needed. The tarpit multimedia system is a possible choice for a temporary system. It would be necessary to acquire two Sun Multidisk pack devices to be attached to tarpit to hold the files needed to bring up WWW and related servers. Also a tape drive of the type currently in use will be needed in order to restore from tape. Cavern currently uses 4mm DAT but it is expected that it will be replaced with 8mm DAT. Backups are now performed to the 8mm tape unit attached t o the apsara host. Dump files are being written for cavern, apsara and alumni to one tape. At some point we will exceed the capacity and either move to a two tape dump or some other method for backup.

New Configuration

Over the summer the server and user community was split accross two machines. The cavern machine now serves the role of a host for information providers and HTML/CGI development. A new server machine (Ultra class) has been implemented which runs the production servers and to which the user community file systems are mounted.

This machine adds 4gig of space so a backup schedule for this machine has been integrated into our current backup schedule.

Regardless of the configuration (one or two machines) used to serve the data, it should be possible to reinstate these services on any capable machine so long as sufficient disk space is available. Performance might suffer if the machine is under powered but since it would only be a temporary situation it should suffice.

Offsite Backup Procedures

Current backup schedules include the cavern, apsara and alumni hosts.

One tape written Friday night.

Tapes to be include in pickup by operations personnel for delivery offsite.

Recovery from tape

Procedures for restoring file systems, directories and files are outlined in the document DRPWWW02: Tape Restore Process. This document covers the basic steps required to use the ufsrestore command to pull archived material from the tapes.

Recovery Steps for Establishing WWW Site

Assemble new hardware.
Recreate cavern filesystems structure when OS install is performed.
1. Install security software.
Restore filesystems from tape.
Restore users (passwd files)
Adjust/restore system configuration files :
1. /etc/inetd.conf
2. /etc/init.d various startup script. Reestablish links to rc directories. See appendix A. /etc etc.,etc.,etc
Get 64meg memory installed.
Request DNS aliases for cavern & www (apsara).

Restore cavern filesystems on a slice by slice basis to match the previous configuration.


Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0      48023   13488   29735    32%    /
/dev/dsk/c0t3d0s6     288855  204672   55303    79%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
/dev/dsk/c0t3d0s3     492351  171424  271697    39%    /var
/dev/dsk/c0t5d0s2    1952573 1066630  690693    61%    /export/home/services
/dev/md/dsk/d0       3894490 1684740 1820310    49%    /export/home
/dev/dsk/c0t3d0s4     480919  345665   87164    80%    /export/home1
/dev/dsk/c0t3d0s5     480919       9  432820     1%    /export/home2
/dev/dsk/c0t4d0s4     985446  460738  426168    52%    /opt
/dev/dsk/c0t4d0s5     966382  720518  149234    83%    /opt1
swap                  188392    5220  183172     3%    /tmp

The output above is from the command 'df -k' performed on the cavern system at the time of this writing. The numbers will vary; however, the layout will remain the same. That is, slice designations such as '/dev/dsk/c0t4d0s5' and it's relationship to the file system '/opt1' will not change. When restoring a file system to a new volume on a new machine for instance the slice designation for the '/export/home' file system might not be what is shown above. What is important is that the slice is allocated enough space to hold the old file system data, or even more.

NOTE: file system /dev/md/dsk/d0 is a concatenation of two physical disks
      created using the Solstice disksuite package.  Refer to the information 
      regarding the creation of such file systems in the installation manual
      provided for the Solstice DiskSuite software.

The file system can be created on the new slice having the same name as the one on the archive tape and the tape contents can be written to the new volume/slice. The system will take care of the variations in slice/ volume naming so long as the archived file system matches directly the file system created to hold it.

Generating a new system

To create a replacement system from scratch will involve the following steps each of which will be documented in detail later in this document.

1) Acquire replacement component parts (as detailed above)
2) Determine suitable site for assembling replacement system.
3) Put together the replacement system. Hardware.
4) Establish base operating system and network capability.
5) Restore file system layout to match old cavern system.
6) Restore directly from tape file systems from previous system.
7) Test and evaluate to see if all systems are in place.

I. Replacement Hardware.

Follow procedure outline elsewhere for submitting detailed replacement parts request. Perform all administrative tasks required to get replacements expedited from SMCC.

II. Establish New Site.

Consult Disaster Recovery Coordinator for site assignment. Submit required space and power requirements.

III. Reconstruct Hardware.

Upon receipt of hardware, unpack and assemble components at the new site. Retain all paperwork, shipping info, etc.

IV. Bring up base system.

Establish a base operating system environment to match prior system. If the new system ships with an OS newer than the previous system. Locate and install the system that had been on cavern previously (e.g. Solaris 2.4 vs. Solaris 2.5) unless architectural differences will not allow. See DRPCX050: Solaris Installation Procedure for more information.
Install patches (current reommended patches)
Install current recommended security software. As outlined in documentation created by the DUST team regarding the latest recommended security steps to establish for a secure Sun system.
Coordinate IP# and DNS entries with NWS to recreate the 'cavern' and 'www' hostnames.

V. Recreate filesystems.

Using the preferred restore method (UFSrestore) restore individual filesystems that are not related to system operation (e.g. /export/home) such as the user community. See DRPWWW02: Tape Restore Process.
Restore individual configuration files on a one by one basis. The password files /etc/passwd and /etc/shadow are examples of this sort of file. There will be numerous files of this sort outlined in another section of this document.

Software Inventory

System Software

Solaris 2.5.1 OS Operating System CAP MacIntosh compatability ADSM Incremental backup software Samba NETBEIU connectivety (for Win95 & NT) Security sendmail.8.7.2 Berkley sendmail wu-ftpd-2.4 Washington University FTP daemon tcp_wrappers connection security and log enhancements pidentd-2.5 RFC-931 Remote identity checking fingerd-1.3 Secure finger daemon rpcbind_1.1 Secure Remote Procedure Call

Development tools Sun compilers C, C++ Pascal PD compilers GNU C,C++ gcc-2.6.3 SAS Statistical software NetPBM Graphic conversion utilities Servers Netscape commerce server WWW Netscape communications server WWW wwwstat-1.01 WWW stats package gwstat WWW stats graphing WAIS freeWAIS-0.202 Full text indexing Isite Full text indexing CSO E-mail directory qi server ph client Informix database Viewpoint Pro ESQLC Informix Online Server Sapphire ISQLperl POP qpopper2.1.4 Qualcomm POP3 server TeX LaTeX User progs Pine-3.91 Pine email package
Lynx Textmode Web browser Notes relating to origins of certain software packages. - E-mail dir CSO vixen.cso.uiuc.edu - Full text indexing wais ftp.cnidr.org Isite - sendmail ftp.cs.berkeley.edu /scr/ucb/sendmail 8.7.1 - Security - TCP wrappers ftp.win.tue.nl - rpcbind - fingerdi (in.fingerd) fiwi.foobar.com /pub /etc/hosts.allow Secure services - wu-ftpd-2.4 uiarchive.cso.uiuc.edu - rcs.5.6.0.1 - qterm - pidentd-2.5 ftp.lysator.liu.es /pub/servers -md5 encryption

Appendix A.

1. The startup scripts.
The directory /etc/init.d contains numerous scripts that provide the ability to start and stop processes necessary for normal operation on the system. During the OS install most of this is created automatically and need not be adjusted. There are several scripts here though that will have to be restored manually to complement the packages that they represent. Which will be restored as part of the overall process of placing file systems back onto the new machine. Below is a listing of the /etc/init.d directory. The files labeled restore should be retrieved.

ANNOUNCE
MOUNTFSYS
PRESERVE
README
RMTMPFILES
acct
asppp
audit
autofs
autoinstall
buildmnttab
buttons_n_dials-setup
cap_stuff --> restore
cron
devlinks
drvconfig
gsconfig
gtconfig
httpd --> restore
httpd-admin.netsc
httpd.NCSA
httpd.netsc
https --> restore
https-admin --> restore
inetinit
inetsvc
informix --> restore
informix_tool --> restore
lp
mkdtab
networker
nfs.client
nfs.server
perf
rootusr
rpc
rtvc-config
sendmail
standardmounts
sxcmem
sysetup
sysid.net
sysid.sys
syslog
tmpfsfix
ufs_quota
upsd
utmpd
uucp
volmgt
xntpd