Creating 2 node GlusterFS for OpenStack Nova/KVM

Add repo and install on both nodes


# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo


# yum -y install glusterfs glusterfs-fuse glusterfs-server




From first node
Start the service and enable it
 # systemctl start glusterd.service
 # systemctl enable glusterd.service
 # gluster peer probe Node2


From Second node


Start the service and enable it
 # systemctl start glusterd.service
 # systemctl enable glusterd.service
 # gluster peer probe Node1




Create Bricks on both nodes


# gluster volume create glustervmstore replica 2 Node1:/vmstore  Node2:/vmstore
# gluster volume start glustervmstore




Since Nova instance will be created under /var/lib/nova/instances, execute the following commands on both nodes


# mount -t glusterfs Node1:/glustervmstore /var/lib/nova/instances


(Replace Node2 with Node1 when executing on second node)


Test it


# cd /var/lib/nova/instances
touch a file and this file will be replicated on the second node too.


Now create a instance in openstack and you will see image on both nodes (If you are getting error, check permission for /var/lib/nova/instance directory)

Zpool root mirror

If you created a mirrored ZFS root pool during an initial installation, you can boot from an alternate disk automatically. Identify an alternate disk in your mirrored ZFS root pool. For example, disk c1t1d0s0 in the output below.

# zpool status
  pool: rpool
 state: ONLINE
 scrub: none requested
config:
        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c1t0d0s0  ONLINE       0     0     0
            c1t1d0s0  ONLINE       0     0     0

ok boot /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/disk@1

If you used the zpool attach command to create a mirrored ZFS root pool after the installation, then you must run the installboot or installgrub command on the alternate disks before they are bootable. For example:

sparc# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t1d0s0

x86# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t1d0s

Add Memory to Red Hat VM under VMWare

Dynamically Add Memory to Red Hat VM under VMWare ESX 

1) Enable hot add memory for the VM (can only be done while VM is powered down).  This is under  Edit Settings -> Options -> Enable CPU/Memory Hotplug -> Enable Memory Hotplug

2) Add the desired memory in VMWare under Edit Settings on the shell VM

3) Run this script on the RHEL 5.x guest

#!/bin/bash

if [ "$UID" -ne "0" ]
   then
    echo -e "You must be root to run this script.\nYou can 'sudo' to get root access"
    exit 1
fi


for MEMORY in $(ls /sys/devices/system/memory/ | grep memory)
do
   SPARSEMEM_DIR="/sys/devices/system/memory/${MEMORY}"
   echo "Found sparsemem: \"${SPARSEMEM_DIR}\" ..."
   SPARSEMEM_STATE_FILE="${SPARSEMEM_DIR}/state"
   STATE=$(cat "${SPARSEMEM_STATE_FILE}" | grep -i online)
   if [ "${STATE}" == "online" ]; then
       echo -e "\t${MEMORY} already online"
   else
       echo -e "\t${MEMORY} is new memory, onlining memory ..."
       echo online > "${SPARSEMEM_STATE_FILE}"
   fi
done


Double check by typing free -m 

Checking password expiration on Solaris using Script

# Copy /etc/shadown to temp directory (I am copying to /home/ramkumar and delete   #after) 

#!/usr/bin/bash

if [ $# -eq 0 ]

   then

    echo "You need to pass user as argument"

    exit 1

fi

COUNT=`grep $1 /home/ramkumar/shadow|wc -l`

if [ $COUNT != 1 ]

  then

  echo "User is not valid"

  exit 1;

fi

CURRENT_CH=`grep $1 /home/ramkumar/shadow | cut -d: -f3`

HOST=`uname -n`

LAST_CHA=`/bin/perl -e 'print int(time/(60*60*24))'`

# Compute the age of the user's password

AGE=`echo $LAST_CHA - $CURRENT_CH | /usr/bin/bc`

echo "last Changed" $AGE " days ago"

# Password expiration

EXPIRE=`echo 90 - $AGE | /usr/bin/bc`

if [ $EXPIRE -lt 0 ]

  then

   EXP=`echo $EXPIRE|sed 's/-//g'`

    echo "$1 password on $HOST expired  $EXP days ago"

  else

    echo "$1 password on $HOST expires in $EXPIRE days"

fi

ASM on Solaris and ASMLib on Linux

• ASM Setup on Solaris (Make sure multipathing is enabled)

(Verify all partition is s6)

Format < c*t*d0>

Format > volname

Volname> “ORC ASM”

Label it > yes

Format > quit

# chown oragrid:asmadmin /dev/rdsk/c*t*d*s6

# chmod 660 /dev/rdsk/c*t*d*s6

 ASM on Linux using using ASMLib

As per  Metalink Document 1089399.1 ( Oracle is not supporting ASMLib  on RHEL 6) 

For Red Hat Enterprise Linux 6 (beginning with 6.4) the kernel driver package 'kmod-oracleasm' is available directly from Red Hat, and can be installed from the "RHEL Server Supplementary (v. 6 64-bit x86_64)" channel on Red Hat Network (RHN).  Updates and support for this module will be provided by Red Hat.  Please check with your Red Hat advisor for questions concerning use of the kernel driver package.  The 'oracleasmlib' and 'oracleasm-support' packages are maintained by Oracle; they are required in order to use kmod-oracleasm.  These are available for download from the Oracle Technology Network.

# yum install kmod-oracleasm

• oracleasm-support-2.1.8-1.el6.x86_64.rpm 
• oracleasmlib-2.0.4-1.el6.x86_64.rpm 

       #  yum install oracleasmlib-2.0.4-1.el6.x86_64.rpm

      #  yum install oracleasm-support-2.1.8-1.el6.x86_64.rpm

After installing 

# /usr/sbin/oracleasm configure –i

Default user to own the driver interface []: oragrid 

Default group to own the driver interface []: asmadmin 

Start Oracle ASM library driver on boot (y/n) [ n ]: y

 Fix permissions of Oracle ASM disks on boot (y/n) [ y ]: y

Set the ORACLEASM_SCANORDER parameter in /etc/sysconfig/oracleasm

NOTE: When setting the ORACELASM_SCANORDER to a value, specify a common string associated with your device mapper pseudo device name. For example, if all the device mapper device had a Deploying Oracle RAC 11g R2 Database on RHEL 6.x / Oracle Linux 6.x

19

prefix string of the word "asm", (/dev/mapper/asm-ocr1,/dev/mapper/asm-ocr2), populate the ORACLEASM_SCANORDER parameter as:ORACLEASM_SCANORDER="dm". This would ensure that oracleasm will scan these disks first.

Set the ORACLEASM_SCANEXCLUDE parameter in /etc/sysconfig/oracleasm to exclude non-multipath devices.

For example: ORACLEASM_SCANEXCLUDE=

NOTE: If we wanted to ensure to exclude our single path disks within /dev/ such as sda and sdb, our ORACLEASM_SCANEXCLUDE string would look like: ORACLEASM_SCANEXCLUDE="sda sdb"

# /usr/sbin/oracleasm createdisk  DISKNAME  /dev/mapper/diskpartition

NOTE: The fields DISKNAME and /dev/mapper/diskpartition should be substituted with the appropriate names for your environment respectively.

NOTE: It is highly recommended to have all of your Oracle related disks to be within Oracle ASM. This includes your OCR disks, voting disks, database disks, and flashback recovery disks.

# /usr/sbin/oracleasm listdisks 

(T0 verify disks)

On the other node:

# /usr/sbin/oracleasm scandisks.

Link Aggregation on Solaris 10

      Link Aggregation on Solaris 10

What is Link Aggregation?

An aggregation is similar to an IPMP link created for failover, but has other advantages:-

• An aggregation is treated as one device for configuration and management, and uses only one IP address.
• In-bound as well as outbound load spreading is provided, so extra bandwidth is available.
• Automatic failover is provided, to the remaining working links in the aggregation.
• Policies can be applied to the traffic flows.
• Parallel aggregations between servers can be created, without going through a switch. 

How to Set up Link Aggregation on Solaris 10?

è Configuration in the switch

First of all we need to make sure link all cables are connected to switch  ( same vlan)

1. Connect to switch console

# telnet XX.XX.XX.XX

2. Enter to privileged mode and open configuration terminal

BR-telnet@FastIron SuperX Router>en

Password:

BR-telnet@FastIron SuperX Router#config terminal

3. For each port of the bond enable link-aggregate

BR-telnet@FastIron SuperX Router(config)#interface ethernet 2/20

BR-telnet@FastIron SuperX Router(config-if-e1000-2/20)#link-aggregate active

BR-telnet@FastIron SuperX Router(config-if-e1000-2/20)#exit

BR-telnet@FastIron SuperX Router(config)#interface ethernet 2/21

BR-telnet@FastIron SuperX Router(config-if-e1000-2/21)#link-aggregate active

BR-telnet@FastIron SuperX Router(config-if-e1000-2/21)#exit

4. Quit configuration terminal and privileged mode, check for those ports in the link- aggregated ports table and exit

BR-telnet@FastIron SuperX Router(config)#exit

BR-telnet@FastIron SuperX Router(config)#write memory

BR-telnet@FastIron SuperX Router#show link-aggregate

System ID: 0012.f2b4.ae00

Long timeout: 120, default: 120

Short timeout: 3, default: 3

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]

------------------------------------------------------------------------------------------

2/20 1 1 483 Yes L Agg Syn Col Dis No No Ope

2/21 1 1 483 Yes L Agg Syn Col Dis No No Ope

------------------------------------------------------------------------------------------

BR-telnet@FastIron SuperX Router#exit

BR-telnet@FastIron SuperX Router>exit

è Configuration on the Server

Login as root and execute to verify what are the interfaces are up and available

# dladm show-dev ( Following command will tell you that nxge0 to nxg3 are up and available for link aggregate)

e1000g0         link: up        speed: 1000  Mbps       duplex: full

e1000g1         link: up        speed: 1000  Mbps       duplex: full

e1000g2         link: unknown   speed: 0     Mbps       duplex: half

e1000g3         link: unknown   speed: 0     Mbps       duplex: half

nxge0           link: up        speed: 1000  Mbps       duplex: full

nxge1           link: up        speed: 1000  Mbps       duplex: full

nxge2           link: up        speed: 1000  Mbps       duplex: full

nxge3           link: up        speed: 1000  Mbps       duplex: full

Another command to verify is dladm show-link

 

Create Link aggregation now:

# dladm  create -aggr  -d  nxge0  -d  nxge1 –d nxge2 –d nxge3 1

Create aggr1 interface and plumb it

# ifconfig  aggr1   plumb 10.25.156.39 –broadcast 10.25.156.63 –netmask 255.255.255.192 up

Check the interface

# ifconfig –a

aggr1: flags=1000842 mtu 1500 index 2

        inet 10.25.156.39 netmask ffffffe0 broadcast 10.25.156.63

        ether 0:21:28:30:50:c6

Activate it

# dladm   modify-aggr  -l   active  -t  short  1

You can check policy and other information by using

# dladm   show-aggr

key: 1 (0x0001) policy: L2      address: 0:21:28:30:50:c6 (auto)

           device       address                 speed           duplex  link    state

           nxge0        0:21:28:30:50:c6          1000  Mbps    full    up      attached

           nxge1        0:21:28:30:50:c7          1000  Mbps    full    up      attached

           nxge2        0:21:28:30:50:c8          1000  Mbps    full    up      attached

           nxge3        0:21:28:30:50:c9          1000  Mbps    full    up      attached

Reboot the server

You can Check packets transferred and validation

#dladm   show-aggr -s

key:1           ipackets   rbytes       opackets   obytes       %ipkts  %opkts

        Total   18817722   2325029336   7546157    1056461980

        nxge0   4696030    580745736    1886522    264113080      25.0    25.0

        nxge1   4696574    580788680    1886522    264113080      25.0    25.0

        nxge2   4738703    583317764    1886570    264119800      25.2    25.0

        nxge3   4686415    580177156    1886543    264116020      24.9    25.0

Other important commands:

# dladm show-linkprop

steps to enable web browser M5000/M3000

Steps to enable web browser


M5000 (Enable https for XSCF)


XSCF> sethttps -c genserverkey

Server key already exists. Do you still wish to update? [y
n] :y

Enter passphrase:


XSCF> sethttps -c selfsign US Georgia Atlanta GROUP myemal@unixlead.com
CA key and CA cert already exist. Do you still wish to update? [y
n] :y


XSCF> sethttps -c enable

Continue? [y
n] :y

Please reset the XSCF by rebootxscf to apply the https settings.

XSCF> rebootxscf

The XSCF will be reset. Continue? [y
n] :y


After reboot, we can access XSCF using web brower.

Steps to upgrade from Solaris 11 to Solaris 11.1 (update 1)

1) Make sure we have publisher setup already if not we have to use pkg command to set publisher (Using SSL certs if we have valid support)




# pkg publisher

PUBLISHER TYPE STATUS P LOCATION

solaris origin online F http://pkg.oracle.com/solaris/release/



2) Update the system with latest patches/release



# pkg update



3) Reboot the system to boot from alternate boot enviromnent



# reboot



3) Update the package itself



# pkg update pkg:/package/pkg



4) Update to Solaris 11.1 with accept option



#pkg update --accept



A clone of solaris-1 exists and has been updated and activated.

On the next boot the Boot Environment solaris-2 will be

mounted on '/'. Reboot when ready to switch to this updated BE.





---------------------------------------------------------------------------

NOTE: Please review release notes posted at:



https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=1372094.1



5) Reboot the system



# reboot



6) Check bootenvironment



# beadm list





BE Active Mountpoint Space Policy Created

-- ------ ---------- ----- ------ -------

solaris - - 14.56M static 2011-12-15 00:32

solaris-1 N / 20.56M static 2012-12-03 11:11

solaris-2 R - 14.22G static 2012-12-03 11:47

solaris-backup-1 - - 145.0K static 2011-12-15 00:50







7) Check /etc/release



Oracle Solaris 11.1 X86

Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.

Assembled 19 September 2012



8) Check pkg list entire



# pkg list entire

NAME (PUBLISHER) VERSION IFO

entire 0.5.11-0.175.1.0.0.24.2 i--

Increasing filesystem space on AIX

Increasing filesystem space on AIX

To check all filesystems:

# df -g

# lslv test (Name of the logical volume)
This will give volume group name

# lsvg rootvg (Volume group name)

Verify free PP and if we have enough space than

# chfs -a size=+1G /filesystem

It will add 1 GB to the filesystem

# df -g (Verify filesystem)

Make sendmail on Solaris listen to more than localhost

Check the current configuration:

# svcprop sendmail grep -i local_only
  config/local_only boolean true
# svccfg -v -s sendmail setprop config/local_only=false
# svcadm refresh sendmail # svcadm restart sendmail

Excluding directories and files in tar command

Create a file with all the dictories and files  that you want to exclude in the tar

# less exclude
 test1

 hello

Execute following command

# tar -cvfX test.tar ./exclude *
a hello excluded

a test/ 0K
a test1 excluded
a test2/ 0K
a test3/ 0K


Extract with tar as usual

# tar -xvf test.tar

 

Setting UP private interface between two zones:

        Setting UP private interface between two zones:

1)      Connect two physical server using cross over cable

2)      ifconfig nxge3 inet 192.168.0.10 netmask 255.255.255.0 broadcast 192.168.0.255

3)      ifconfig nxge3 up

4)      Make changes to the zone so that it will persistent

bash-3.00# zonecfg -z zone

zonecfg:testzone> add net

zonecfg:testzone:net> set address=192.168.0.15

zonecfg:testzone:net> set physical=nxge3

zonecfg:testzone:net> end

zonecfg:testzone> verify

zonecfg:testzone> commit

zonecfg:testzone> exit

5)      Create interface and attach to the zone from the global

# ifconfig nxge3 addif 192.168.0.15 netmask 255.255.255.0 broadcast 192.168.0.255 zone testzone up

Created new logical interface nxge3:1

Oracle settings in /etc/project

/etc/project

 

Command to check value for project.max-shm-memory

prctl -n project.max-shm-memory -i project user.oracle

Command to increase memory to 96 GB memory for running Process

prctl -n project.max-shm-memory -r -v 96GB -i project user.oracle

Command to Make it permanent by updating /etc/project file

projmod -sK "project.max-shm-memory=(privileged,103079215104,deny)" user.oracle

Command to check all parameters for current user

prctl $$

Command to check all parameters used for Oracle user

prctl -i project user.oracle

Adding Static IP on Solaris 11

Adding Static IP

root@solaris11:~# ipadm create-ip net2
root@solaris11:~# ipadm create-addr -T static -a 192.168.0.235/22 net2/ipv4
root@solaris11:~# route -p add default 192.168.0.254
add net default: gateway 192.168.0.254
add persistent net default: gateway 192.168.0.254

updating /etc/resolv.conf on Solaris 11

### updating /etc/resolv.conf on Solaris 11 ################
root@solaris11:~# svccfg
svc:> select dns/client
svc:/network/dns/client> setprop config/search = astring: ("search.com" "searchme.com" "searchyou.com")
svc:/network/dns/client> setprop config/nameserver = net_address: (192.10.10.10 192.10.10.11 192.10.10.12)
svc:/network/dns/client> select dns/client:default
svc:/network/dns/client:default> refresh
svc:/network/dns/client:default> validate
svc:/network/dns/client:default> select name-service/switch
svc:/system/name-service/switch> setprop config/host = astring: "files dns"
svc:/system/name-service/switch> select system/name-service/switch:default
svc:/system/name-service/switch:default> refresh
svc:/system/name-service/switch:default> validate
svc:/system/name-service/switch:default>
svc:/system/name-service/switch:default>
svc:/system/name-service/switch:default>
svc:/system/name-service/switch:default>
svc:/system/name-service/switch:default> exit
root@solaris11:~# svcadm enable dns/client

Stopping and Starting Ops Center 12c

Stopping Ops Center 12c

# /opt/SUNWxvmoc/bin/proxyadm stop -wv
# /opt/SUNWxvmoc/bin/satadm stop -wv

Starting Ops Center 12c

# /opt/SUNWxvmoc/bin/satadm start -wv
# /opt/SUNWxvmoc/bin/proxyadm start -wv

Installing IDR on solaris 11

Installing IDR on solaris 11 is different from Solaris 10 (patches). They called it idr (interim diagnostic relief) indentifier on Solaris 11

Following example, I am installing idr111 on Solaris 11 server

# pkgrepo create /var/tmp/idrrepo


# cd /var/tmp

# pkgrecv -s ./idr111p5p -d /var/tmp/idrrepo '*'

# pkg set-publisher -g file:///var/tmp/idrrepo solaris

# pkg publisher ( To verify that publisher is available)

solaris (non-sticky) origin online file:///var/tmp/idrrepo/


Install the idr using following command:

# pkg install idr111


Verify the package is installed by


# pkg verify -v idr111

PACKAGE STATUS

pkg://solaris/idr111 OK

Error while luactivate

/usr/sbin/luactivate: /etc/lu/DelayUpdate/: cannot create

Workaround:
execute folloing:

# export BOOT_MENU_FILE="menu.lst"

Debugging issues on Solaris using truss

In original ksh terminal, find out the PID of the ksh shell:


# ps

On 2nd terminal:

# truss -faeldo locale.truss -vall -r0 -w 1,2 -p (PID of ksh shell)

In original ksh terminal:

Execute the command which is having issues

On 2nd terminal:


Tail the locale.truss or read the contents for more details

Modules of Cloud

There are basically three service modules in the cloud computing:

1) SaaS -> Software as a Service, Providers will provide the client with a complete software application environment, including management and the user interface. Clients usually have access to the environment through a thin user interface such as browser, which enables data entry and manages user interactions. Provider must manage everything else, from the infrastructure to the architecture.

2) PaaS -> Platform as a Service, provides clients with the complete development environment incuding virtual machines . operating systems, applications, services. Clients can deploy their own applications programmed using PaaS-compatible languages and tools. With this model, the service provider is responsible for managing cloud infrastructure and the operating systems, and client develop, install and manage the applications they want to deploy.

3) IaaS-> Infrastructure as a Service provides hardware assets to clients, who can then provision themm. Example of these hardware assets are virtual computing systems, storage resources.Clients are responsible for deploying and managing all the other aspects of the infrastructure including Opeations sytstem, application and user interactions with the systems.