How to Collect Solaris OS coredump for analysis

There will be situation when you might need to collect OS kernel core dumps for Sun support analysis. The procedure is below.


1. login to the Service Processor (root or admin)

2 send a break signal to the console

ILOM syntax: -> set /HOST send_break_action=dumpcore

ALOM syntax: sc> break -D

NOTE: the above commands will send a break signal and create
a coredump. Make sure you have enough disk space available
in /var or whatever directory you have dumpadm configured to
capture a corefile

3. connect to the system console after executing above "break"
command, and if Solaris OE boots, check for corefile and upload
to Oracle for analysis by a Kernel TSE

ILOM syntax: -> start /SP/console

ALOM syntax: sc> console -f

How to clear and delete "waiting for retry" status 50 job?

Are you ever wondered by seeing jobs in Netbackup java console which has state "Waiting for Retry" and status 50? If you try to delete it... ooops its not deleting :) What we will do???

This situation will come if the client process aborted. The solution is to stop the netbackup master services. And run:

/usr/openv/netbackup/bin/bpjobd -r

Once you cleared all the jobids start the netbackup services. You are back to buisiness...

Replacing a Failed Tape drive in netbackup

If you replace a tape drive attached to solaris media server some time it wont recognise the new drive and will endup in misconfiguration in media device databsase. This procedure is helpfull if you replacing failed Tape drive which attached to Solaris media server.

1. Remove the faulty tape drive from library
2. Login to the media server as root and identify which controller the removed tape drive was attached using cfgadm, you can identify using the condition failing

# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 disk connected configured unknown
c0::dsk/c0t1d0 disk connected configured unknown
c0::sg/c0t0l0 unknown connected configured unknown
c0::sg/c0t1l0 unknown connected configured unknown
c1 scsi-bus connected configured unknown
c1::dsk/c1t0d0 CD-ROM connected configured unknown
c2 fc connected unconfigured unknown
c3 fc-private connected configured unknown
c3::50060e801029f140 disk connected configured unknown
c5 fc connected unconfigured unknown
c6 fc-fabric connected configured unknown
c6::500104f0008e3031 med-changer connected configured unknown
c6::500104f0008e3041 tape connected configured unknown
c7 fc-fabric connected configured unknown
c7::500104f0008e3044 tape connected configured unknown
c7::500104f0008e3047 tape connected configured unknown
c8 fc-fabric connected configured unknown
c8::500104f0008e304a tape connected configured unknown
c8::500104f0008e304d tape connected configured unknown
c9 fc-fabric connected configured unknown
c9::500104f0008e3050 tape connected configured unknown
c9::500104f0008e3053 unavailable connected configured failing
c10 fc-private connected unconfigured unknown
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
usb1/1.1 unknown empty unconfigured ok
usb1/1.2 unknown empty unconfigured ok
usb1/1.3 unknown empty unconfigured ok
usb1/1.4 unknown empty unconfigured ok
usb1/2 unknown empty unconfigured ok

3. Shake controller which has the failed tape drive
#/usr/sbin/cfgadm -c configure c9

4. Do a cleanup of the device files (This will leanup the device files of removed tape drive.
# devfsadm -C -c tape -v
devfsadm[9600]: verbose: removing file: /dev/rmt/4
devfsadm[9600]: verbose: removing file: /dev/rmt/4n
devfsadm[9600]: verbose: removing file: /dev/rmt/4b
devfsadm[9600]: verbose: removing file: /dev/rmt/4bn
devfsadm[9600]: verbose: removing file: /dev/rmt/4l
devfsadm[9600]: verbose: removing file: /dev/rmt/4m
devfsadm[9600]: verbose: removing file: /dev/rmt/4h
devfsadm[9600]: verbose: removing file: /dev/rmt/4c
devfsadm[9600]: verbose: removing file: /dev/rmt/4u
devfsadm[9600]: verbose: removing file: /dev/rmt/4ln
devfsadm[9600]: verbose: removing file: /dev/rmt/4mn
devfsadm[9600]: verbose: removing file: /dev/rmt/4hn
devfsadm[9600]: verbose: removing file: /dev/rmt/4cn
devfsadm[9600]: verbose: removing file: /dev/rmt/4un
devfsadm[9600]: verbose: removing file: /dev/rmt/4lb
devfsadm[9600]: verbose: removing file: /dev/rmt/4mb
devfsadm[9600]: verbose: removing file: /dev/rmt/4hb
devfsadm[9600]: verbose: removing file: /dev/rmt/4cb
devfsadm[9600]: verbose: removing file: /dev/rmt/4ub
devfsadm[9600]: verbose: removing file: /dev/rmt/4lbn
devfsadm[9600]: verbose: removing file: /dev/rmt/4mbn
devfsadm[9600]: verbose: removing file: /dev/rmt/4hbn
devfsadm[9600]: verbose: removing file: /dev/rmt/4cbn
devfsadm[9600]: verbose: removing file: /dev/rmt/4ubn
devfsadm[9600]: verbose: removing file: /dev/rmt/5
devfsadm[9600]: verbose: removing file: /dev/rmt/5n
devfsadm[9600]: verbose: removing file: /dev/rmt/5b
devfsadm[9600]: verbose: removing file: /dev/rmt/5bn
devfsadm[9600]: verbose: removing file: /dev/rmt/5l
devfsadm[9600]: verbose: removing file: /dev/rmt/5m
devfsadm[9600]: verbose: removing file: /dev/rmt/5h
devfsadm[9600]: verbose: removing file: /dev/rmt/5c
devfsadm[9600]: verbose: removing file: /dev/rmt/5u
devfsadm[9600]: verbose: removing file: /dev/rmt/5ln
devfsadm[9600]: verbose: removing file: /dev/rmt/5mn
devfsadm[9600]: verbose: removing file: /dev/rmt/5hn
devfsadm[9600]: verbose: removing file: /dev/rmt/5cn
devfsadm[9600]: verbose: removing file: /dev/rmt/5un
devfsadm[9600]: verbose: removing file: /dev/rmt/5lb
devfsadm[9600]: verbose: removing file: /dev/rmt/5mb
devfsadm[9600]: verbose: removing file: /dev/rmt/5hb
devfsadm[9600]: verbose: removing file: /dev/rmt/5cb
devfsadm[9600]: verbose: removing file: /dev/rmt/5ub
devfsadm[9600]: verbose: removing file: /dev/rmt/5lbn
devfsadm[9600]: verbose: removing file: /dev/rmt/5mbn
devfsadm[9600]: verbose: removing file: /dev/rmt/5hbn
devfsadm[9600]: verbose: removing file: /dev/rmt/5cbn
devfsadm[9600]: verbose: removing file: /dev/rmt/5ubn
devfsadm[9600]: verbose: removing file: /dev/rmt/7
devfsadm[9600]: verbose: removing file: /dev/rmt/7n
devfsadm[9600]: verbose: removing file: /dev/rmt/7b
devfsadm[9600]: verbose: removing file: /dev/rmt/7bn
devfsadm[9600]: verbose: removing file: /dev/rmt/7l
devfsadm[9600]: verbose: removing file: /dev/rmt/7m
devfsadm[9600]: verbose: removing file: /dev/rmt/7h
devfsadm[9600]: verbose: removing file: /dev/rmt/7c
devfsadm[9600]: verbose: removing file: /dev/rmt/7u
devfsadm[9600]: verbose: removing file: /dev/rmt/7ln
devfsadm[9600]: verbose: removing file: /dev/rmt/7mn
devfsadm[9600]: verbose: removing file: /dev/rmt/7hn
devfsadm[9600]: verbose: removing file: /dev/rmt/7cn
devfsadm[9600]: verbose: removing file: /dev/rmt/7un
devfsadm[9600]: verbose: removing file: /dev/rmt/7lb
devfsadm[9600]: verbose: removing file: /dev/rmt/7mb
devfsadm[9600]: verbose: removing file: /dev/rmt/7hb
devfsadm[9600]: verbose: removing file: /dev/rmt/7cb
devfsadm[9600]: verbose: removing file: /dev/rmt/7ub
devfsadm[9600]: verbose: removing file: /dev/rmt/7lbn
devfsadm[9600]: verbose: removing file: /dev/rmt/7mbn
devfsadm[9600]: verbose: removing file: /dev/rmt/7hbn
devfsadm[9600]: verbose: removing file: /dev/rmt/7cbn
devfsadm[9600]: verbose: removing file: /dev/rmt/7ubn

5. Fix the drive and shake the controller:

#/usr/sbin/cfgadm -c configure c9

6. Check device files created for new drives, if not run a devfsadm and check

#ls -lrt /dev/rmt


7. if its created UP the drive in console and verify the backup

Netbackup Tape Drive configuration

Tape Drive configuration procedure for Netbackup
Introduction:
Below procedure is for configuring new tape drives in Netbackup environment. Below steps needs to be carried out only afrer new Tape drivers are installedin Tape Library.All the below steps are only tested in Sun Solaris.

1. Login to the media server and stop the netbackup services.
$/usr/openv/netbackup/bin/goodies/netbackup stop
2. confirm OS found new Tape devices
$iostat -Engrep-i st
3. Confirm device files are created or not
$cd /dev/rmt
$ls -lrt *cbn
lrwxrwxrwx 1 root root 80 Apr 2 15:58 3cbn -> /../../devices/pci@7c0/pci@0/pci@9/SUNW,qlc@0,1/fp@0,0/st@w500104f0008e3050,0:cbnlrwxrwxrwx 1 root root 80 Apr 2 15:58 4cbn -> /../../devices/pci@7c0/pci@0/pci@9/SUNW,qlc@0,1/fp@0,0/st@w500104f0008e3053,0:cbn
IF device files are not found new tape drive, please restart the server using reboot -- -r
NetBackup Media Manager provides its own driver for communicating with SCSI-controlled robotic peripherals. This driver is called the SCSA (Generic SCSI passthrudriver), also referred to as the sg driver.If os found the drive and device files are created we can reconfigure sg driver for netbackup.

1. Make a copy of below files.
/usr/openv/volmgr/bin/driver/sg.conf
/usr/openv/volmgr/bin/driver/sg.links
2. Identfy the WWN number of new Tape drive
$/usr/sbin/cfgadm -algrep -i tape
or
$/usr/openv/volmgr/bin/sgscan
THis command will identify the tape drive and wwn number which are not configured in sg.conf
3. Edit below files add new WWN number identified from above step.
$vi /usr/openv/volmgr/bin/driver/sg.conf
name="sg" class="scsi" target=15 lun=0;name="sg" class="scsi" target=15 lun=1;name="sg" parent="fp" target=0 lun=0 fc-port-wwn="500104f0008e3031";name="sg" parent="fp" target=0 lun=1 fc-port-wwn="500104f0008e3031";name="sg" parent="fp" target=0 lun=0 fc-port-wwn="500104f0008e304a";name="sg" parent="fp" target=0 lun=1 fc-port-wwn="500104f0008e304a";name="sg" parent="fp" target=0 lun=0 fc-port-wwn="500104f0008e3047";name="sg" parent="fp" target=0 lun=1 fc-port-wwn="500104f0008e3047";.............$sudo vi /usr/openv/volmgr/bin/driver/sg.links................type=ddi_pseudo;name=sg;addr=e,1; sg/c\N0t14l1type=ddi_pseudo;name=sg;addr=f,0; sg/c\N0t15l0type=ddi_pseudo;name=sg;addr=f,1; sg/c\N0t15l1type=ddi_pseudo;name=sg;addr=w500104f0008e3031,0; sg/c\N0t\A1l0type=ddi_pseudo;name=sg;addr=w500104f0008e3031,1; sg/c\N0t\A1l1type=ddi_pseudo;name=sg;addr=w500104f0008e304a,0; sg/c\N0t\A1l0type=ddi_pseudo;name=sg;addr=w500104f0008e304a,1; sg/c\N0t\A1l1type=ddi_pseudo;name=sg;addr=w500104f0008e3047,0; sg/c\N0t\A1l0type=ddi_pseudo;name=sg;addr=w500104f0008e3047,1; sg/c\N0t\A1l1................
4. Make copy of file /kernel/drv/sg.conf
5. Now we need to install new sg driver configuration.
$/usr/bin/rm -f /kernel/drv/sg.conf
$/usr/openv/volmgr/bin/driver/sg.install
6. Start netbackup services in media server.
7. Loggin to Netbackup Admin console and run Configure storage Devices wizard. This will detect the new drives and configure to device database.

How to stop and start Netbackup services-Best way!

Note: Below steps only applicable to netbackup server hosted in unix platform.


Cancel all active jobs using java console.
Close all open Java console windows
Go to /usr/openv/netbackup/bin/goodies
Execute below command
# netbackup stop
Delete the worklist files and the *lock files from /usr/openv/netbackup/bin/bpsched.d. Do the following in that directory:
#rm /usr/openv/netbackup/bin/bpsched.d/worklist.[0-9]*
#rm /usr/openv/netbackup/bin/bpsched.d/*.lock
On Netbackup 6
rm /usr/openv/netbackup/bin/bpsched.d/pempersist
#netbackup start

Let me know if you have any doubts...

How to enable netbackup ports in iptables.

Step1.Edit /etc/sysconfig/iptables
$sudo vi /etc/sysconfig/iptables
Append below entries.
-A INPUT -p tcp -i --dport 13782 -j ACCEPT-A INPUT -p tcp -i --dport 13724 -j ACCEPT
step 2:Refresh iptable configuration
$sudo /sbin/iptables-restore < ./iptables
step3:Verify these ruleas are loaded
$ sudo /sbin/iptables --listChain INPUT (policy ACCEPT)target prot opt source destinationRH-Firewall-1-INPUT all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)target prot opt source destinationRH-Firewall-1-INPUT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)target prot opt source destination
Chain RH-Firewall-1-INPUT (2 references)target prot opt source destinationACCEPT all -- anywhere anywhereACCEPT icmp -- anywhere anywhere icmp anyACCEPT esp -- anywhere anywhereACCEPT ah -- anywhere anywhereACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdnsACCEPT udp -- anywhere anywhere udp dpt:ippACCEPT tcp -- anywhere anywhere tcp dpt:ippACCEPT all -- anywhere anywhere state RELATED,ESTABLISHEDACCEPT tcp -- anywhere anywhere state NEW tcp dpt:sshACCEPT tcp -- anywhere anywhere tcp dpt:bpcdACCEPT tcp -- anywhere anywhere tcp dpt:vnetdREJECT all -- anywhere anywhere reject-with icmp-host-prohibited