Add Disk Space to Pool or Filesystem in ZFS

Grow Disk Space to Pool or FileSystems in ZFS

or

ZFS – How to increase rpool or FileSystems in Solaris

1.# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 1.13T 308G 1.13T /opt/openv
2.# zpool status rpool
pool: rpool
state: ONLINE
scan: scrub repaired 0 in 6h13m with 0 errors on Tue Apr 5 16:34:18 2016
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
emcpower0a ONLINE 0 0 0
emcpower1a ONLINE 0 0 0
emcpower2a ONLINE 0 0 0
3.# zpool add rpool emcpower4a
cannot open ‘/dev/dsk/emcpower4a’: I/O error

4.# Error shows that we need to format the disk as in same way other disk has been done .

5.# zpool add rpool emcpower4a

bash-3.2# df -kh
Filesystem size used avail capacity Mounted on
rpool 1.4T 1.1T 308G 79% /opt/openv

bash-3.2# zpool set autoexpand=on rpool
bash-3.2# df -kh /opt/openv
Filesystem size used avail capacity Mounted on
rpool 2.0T 1.1T 308G 50% /opt/openv

Too many connections error in Unix/Solaris

Too many connections error  in solaris

or

Multiple users are not able to connect servers

bash-3.2$ ssh sftp.unixplanetary.net
Received disconnect from 156.33.200.35: 12: Too many connections.
bash-3.2$

First login through console and restart the ssh service it will resolve the issue  .

 

Disk I/O Performance issue in Solaris

  1. Run  Iostat -Cxn for disk io
  2. Check for Column %b and check if there is spike in the no count
  3. Check for asvc_t and  wsvc_t

Here is the share busy percentage.

OracleSrv:# iostat -Cxn
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
2.4 1.1 147.0 11.7 0.0 0.0 0.2 3.7 0 1 c0d0
2.7 1.2 104.2 13.8 0.0 0.0 0.2 3.5 0 1 c0d1
2.5 2.8 63.5 36.3 0.0 0.0 0.1 3.1 0 0 c0d2
2.6 17.6 171.5 264.6 0.0 0.0 0.0 2.3 0 2 c0d3
20.3 11.2 778.9 132.9 0.0 0.2 0.0 6.9 0 3 c0d4
20.2 11.2 778.4 132.3 0.0 0.2 0.0 6.9 0 3 c0d5
20.1 11.2 778.1 132.4 0.0 0.2 0.0 6.9 0 3 c0d6
0.0 0.0 0.0 0.0 0.0 0.0 446.2 449.6 0 0 c0d7
0.0 0.0 0.0 0.0 0.0 0.0 278.1 279.6 0 80 File09  —————> %b is high appx 80 so it has high i/o

OracleSrv:# iostat -Cxn 5 10 |grep -i FILE09
22.0 7.2 96.8 9.2 0.0 0.3 0.1 11.8 0 22 File09:/ifs/cifs/File09/rm/RF_prod
8.3 7.1 2.9 1.0 0.0 0.1 0.0 8.4 0 6 File09:/ifs/cifs/File09/rm/RF_prod
48.6 33.9 18.0 4.1 0.0 0.7 0.0 8.6 0 19 File09:/ifs/cifs/File09/rm/RF_prod
21.9 12.5 6.6 1.8 0.0 0.6 0.0 17.0 0 11 File09:/ifs/cifs/File09/rm/RF_prod
3.4 5.9 5.4 1.0 0.0 0.1 0.0 6.8 0 5 File09:/ifs/cifs/File09/rm/RF_prod
1.6 0.4 12.8 0.3 0.0 0.1 0.0 25.4 0 5 File09:/ifs/cifs/File09/rm/RF_prod
103.6 0.8 827.0 0.6 0.0 0.9 0.0 8.4 0 79 File09:/ifs/cifs/File09/rm/RF_prod
23.1 1.0 182.8 0.7 0.0 1.0 0.0 39.8 0 92 File09:/ifs/cifs/File09/rm/RF_prod
33.0 0.8 263.2 0.6 0.0 0.8 0.0 22.3 0 70 File09:/ifs/cifs/File09/rm/RF_prod
8.4 0.8 67.1 0.6 0.0 0.9 0.0 93.8 0 85 File09:/ifs/cifs/File09/rm/RF_prod
OracleSrv:#

 

 

 

Memory issue in Solaris

Memory issue in Solaris 

Currently server has total 327 Gb

sapdev:# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
Kernel 6332481 49472 15%
ZFS File Data 19994158 156204 48%
Anon 12921795 100951 31%
Exec and libs 578486 4519 1%
Page cache 1068169 8345 3%
Free (cachelist) 101826 795 0%
Free (freelist) 946125 7391 2%
Total 41943040 327680—————-> Total Memory
Thx !!

sapdev:# prtconf | grep Mem
Memory size: 327680 Megabytes

We see that ZFS is using more physical memory on this server. We have to cap this usage and it requires a reboot of the server.

Looks like this host was using UFS for data filesystems earlier and ZFS ARC limit was not there in place.

Since FS’s are migrated to ZFS , ARC cache limit is mandatory.
Limit the ZFS ARC cache to 4GB. Otherwise ZFS consumes the vast majority of memory
which makes looking at resource consumption difficult.

we need to do the setting in /etc/system file .

set zfs:zfs_arc_max=4294967296

sapdev#

It is capped till 4Gb as per standard.It will be applied when the server re-reads /etc/system during boot.

So we need a reboot.

ZFS arc is a ZFS file system factor, which is necessary for better performance , especially for DB hosted systems.
But it grows in memory with the ZFS file system usage.
As the sap server exclusively use zfs, the arc usage is very high, which will ultimately consume all the RSS and do a performance bottleneck.

4 GB arc is a suitable size if we consider performance and RSS availability.

sapdev:# sar -r 5 5

SunOS sapdev 5.10 Generic_150400-38 sun4u 12/22/2016

13:57:40 freemem freeswap
13:57:45 16326755 693435562
13:57:50 16326117 693424515
13:57:55 16325316 693411808
13:58:00 16324695 693404640
13:58:05 16325036 693416429

Average 16325584 693418591

Note : Above sizes are given in Block , hence please convert it in MB/GB.

sapdev:# echo ::memstat|mdb -k
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 2578688 20146 6%
ZFS File Data 467305 3650 1%
Anon 19701001 153914 47%
Exec and libs 574865 4491 1%
Page cache 2308486 18035 5%
Free (cachelist) 1095285 8556 3%
Free (freelist) 15217410 118886 36%
Total 41943040 327680
sapdev:#

Note : Above values are given in KB , Hence please convert in MB/GB.

Performance issue in Solaris

 System performance  of CPU /Memory utilization

Prstat -Z  –> will show system performance with respect to All zones available in systems.

Unixplanetary:# prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
9911 oraq3m 17G 17G sleep 60 0 0:06:15 0.3% oracle/1
15041 oraq3m 17G 9266M cpu37 60 0 10:46:47 0.2% oracle/49
10006 oraq3m 17G 17G run 60 0 0:10:45 0.2% oracle/1
2478 oraq3m 17G 17G sleep 60 0 0:05:31 0.2% oracle/1
15166 oraq3m 17G 9267M cpu91 60 0 9:59:27 0.1% oracle/65
15000 oraq3m 17G 9265M sleep 59 0 10:55:00 0.1% oracle/45
15131 oraq3m 17G 9265M sleep 60 0 10:46:24 0.1% oracle/48
10353 oraq3m 17G 17G cpu138 0 0 0:00:13 0.1% oracle/1
15086 oraq3m 17G 9265M sleep 59 0 10:38:46 0.1% oracle/41
13779 oraq3m 17G 17G sleep 60 0 0:00:04 0.1% oracle/1
15151 oraq3m 17G 9265M sleep 59 0 11:32:46 0.0% oracle/46
8 root 0K 0K sleep 60 – 368:19:57 0.0% vmtasks/256
18279 root 10M 9656K cpu220 59 0 0:00:05 0.0% prstat/1
87 root 349M 312M sleep 0 0 383:37:06 0.0% vxconfigd/3
4610 oracle 2264K 1992K cpu119 0 2 0:00:03 0.0% cleanup_archive/1
1 root 3136K 2040K sleep 54 0 59:06:17 0.0% init/1
26586 oraq3m 17G 17G sleep 59 0 0:03:51 0.0% oracle/1
15024 oraq3m 17G 9265M sleep 59 0 12:42:20 0.0% oracle/45
11262 oracle 539M 474M sleep 59 0 1:33:59 0.0% java/261
4479 root 11M 8088K sleep 50 0 13:36:29 0.0% vxesd/7
8860 oracle 17G 17G sleep 59 0 0:00:03 0.0% oracle/1
24001 oraqj6 8121M 8097M sleep 59 0 0:52:45 0.0% oracle/1
24360 oraqo4 9469M 5200M sleep 49 0 7:56:58 0.0% oracle/1
20682 oraqu3 4126M 3974M sleep 59 0 6:33:23 0.0% oracle/1
18811 oraqc3 44G 44G sleep 49 0 7:57:38 0.0% oracle/1
20596 oraqu3 4125M 3973M sleep 59 0 7:52:54 0.0% oracle/1
29787 oraqe3 46G 46G sleep 59 0 36:12:27 0.0% oracle/1
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 2343 233G 227G 44% 2166:30:0 1.7% global
7 52 622M 595M 0.1% 14:26:48 0.0% Oracle_zone1  ———>First Zone
8 46 379M 326M 0.1% 13:09:34 0.0% Oracle_zone2
9 50 1440M 774M 0.1% 17:05:31 0.0% Oracle_zone3
1 45 529M 466M 0.1% 14:14:00 0.0%   Oracle_zone4
2 46 535M 462M 0.1% 15:15:03 0.0%   Oracle_zone5
4 45 584M 543M 0.1% 17:11:44 0.0%    Oracle_zone6
10 46 1397M 678M 0.1% 16:45:16 0.0%Oracle_zone7————->Seventh Zone
Total: 2824 processes, 42486 lwps, load averages: 11.87, 10.83, 9.61

 

Next Command is vmstat .

oraprd04:# vmstat 5 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr lf s1 s2 s3 in sy cs us sy id
0 1 0 382096040 199690752 4202 10036 183 1863 1861 0 0 1 11 1 10 25701 79019 35360 1 1 97
0 0 0 380268888 246577984 4045 5892 0 16 16 0 0 0 0 0 1 91782 47845 250393 1 3 96
1 0 0 380216760 246524600 3787 5607 6 55 52 0 0 0 1 0 2 116685 49020 289345 1 2 96
0 0 0 380116984 246528680 4223 5626 0 14 14 0 0 0 23 0 17 129216 58660 316467 1 3 96
1 0 0 380434072 246739096 5372 6174 0 24 24 0 0 0 5 0 3 135430 56415 333904 1 3 95

How to Analyse Vmstat .

r – it indicate  run que length in the cpu  or how many process are  in que and waiting  for the cpu  time.

Run Que indicate that Total no of active process in the current Que for CPU .

b- its mean blocked process and they are actually started  running but somehow they have blocked because of some other resource purpose  they are waiting for i/o process or  memory operation

w-they are waiting for cpu admission

 

oraprd04:# echo ::memstat |mdb -k
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 3558911 27803 5%
ZFS File Data 406120 3172 1%
Anon 30220448 236097 45%
Exec and libs 464719 3630 1%
Page cache 1546887 12085 2%
Free (cachelist) 3038332 23736 5%
Free (freelist) 27807911 217249 41%
Total 67043328 523776
oraprd04:#

 

Solaris 10 Live Patching Steps

Today I m explaining the procedure for Solaris 10 OS patching when the root File System is UFS .The process is  90% matching with zfs File System .Here we are creating alternative boot environment with sol10-20161209 and current boot environbment is sol10-20160412.Before starting the Patching the take system necessary backup.

 

Solaris 10 Patching Steps for UFS File Systems

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
sol10-20160412 yes yes yes no -
# metastat -p
d52 -m d53 1
d53 1 1 /dev/dsk/emcpower5c
d50 -m d501 1
d501 1 1 /dev/dsk/emcpower3c
d55 -m d57 1
d57 1 1 /dev/dsk/emcpower4c
d3 -m d13 d23 1
d13 1 1 c0t0d0s3
d23 1 1 c0t1d0s3
d1 -m d11 d21 1
d11 1 1 c1t0d0s0
d21 1 1 c1t1d0s0
d0 -m d10 d20 1
d10 1 1 c0t0d0s0
d20 1 1 c0t1d0s0

# df -kh
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d3 48G 22G 26G 46% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 621G 1.7M 621G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
fd 0K 0K 0K 0% /dev/fd
swap 8.0G 240K 8.0G 1% /tmp
swap 621G 144K 621G 1% /var/run
/dev/md/dsk/d55 313G 239G 68G 78% /SAP_DATA
/dev/md/dsk/d50 156G 129G 26G 84% /ZONES

# lucreate -c sol10-20160412 -n sol10-`date +%Y%m%d` -m /:/dev/md/dsk/d3:ufs -m /ZONES:/dev/md/dsk/d50:ufs -m /SAP_DATA:/dev/md/dsk/d55:ufs

Updating boot environment description database on all BEs.
ERROR: /dev/md/dsk/d3 is already mounted
ERROR: /dev/md/dsk/d3 is already in use by another boot environment sol10-20160412:/:ufs

luconfig: ERROR: /dev/md/dsk/d3 cannot be used for file system / in boot environment sol10-20161209.
ERROR: Configuration of boot environment failed.

Question : Why its failed ?
Answer : Because its created on mounted metadevices ( mounted metadevice is d3), Hence its get failed .
# lucreate -c sol10-20160412 -n sol10-`date +%Y%m%d` -m /:/dev/md/dsk/d0:ufs -m /ZONES:/dev/md/dsk/d52:ufs
sol10-20161209> successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
————————– ——– —— ——— —— ———-
sol10-20160412 yes yes yes no –
sol10-20161209 yes no no yes –

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
————————– ——– —— ——— —— ———-
sol10-main yes yes yes no –
sol10-alt yes no no yes –

cd /var/tmp/10_Recommended
# ls
10_Recommended.README Copyright installcluster patch_order patchset.conf
10_Recommended.html LEGAL_LICENSE.TXT installpatchset patches

# ./installpatchset --apply-prereq --s10patchsetSetup ...........................................

Recommended OS Patchset Solaris 10 SPARC (2016.07.14)Application of patches started : 2016.12.09 08:53:45
Application of patches finished : 2016.12.09 08:54:47
Following patches were applied :
119254-93

Following patches were skipped :
Patches already applied
120900-04 119317-01 138215-02 146054-07 125555-16
121133-02 121296-01 148336-02 142251-02

Installation of prerequisite patches complete.

Install log files written :
/var/sadm/install_data/s10s_rec_patchset_short_2016.12.09_08.53.45.log
/var/sadm/install_data/s10s_rec_patchset_verbose_2016.12.09_08.53.45.log

# cd /var/tmp/10_Recommended
# cd 10_Recommended
# ./installpatchset --s10patchset -B sol10-20161209
Setup .............................................
Recommended OS Patchset Solaris 10 SPARC (2016.07.14)

Application of patches started : 2016.12.09 11:14:26

Applying 120900-04 ( 1 of 406) … skipped
Applying 121133-02 ( 2 of 406) … skipped
Applying 119254-93 ( 3 of 406) … success
Applying 152264-01 (405 of 406) … success
Applying 152363-01 (406 of 406) … success

Application of patches finished : 2016.12.09 11:58:45
Following patches were applied :
121136-02 138195-04 145080-15 150616-01 152254-01
121211-02 138822-12 145200-13 150618-02

Installation of patch set to alternate boot environment complete.

Please remember to activate boot environment sol10-20161209 with luactivate(1M)
before rebooting.

Install log files written :
/.alt.sol10-20161209/var/sadm/install_data/s10s_rec_patchset_short_2016.12.09_11.14.26.log
/.alt.sol10-20161209/var/sadm/install_data/s10s_rec_patchset_verbose_2016.12.09_11.14.26.log

init 6