Migrating to PowerHA 7.1.3 from out of support PowerHA-HACMP [2015] by realbjornroden

BjĂśrn RodĂŠn (roden@ae.ibm.com) works for IBM System Lab Services and member of IBM WW Executive Advisory Practice, and as SME also part of IBM WW PowerCare Teams for Availability, Performance, and Security. Bjorn holds MSc, BSc and DiplSSc in Informatics and BCSc and DiplCSc in Computer Science, is a IBM Redbooks Platinum Author, IBM Certified Specialist etc, and has worked in different roles with architecting, designing, planning, leading, implementing, programming, and assessing high availability, resilient, secure, and high performance systems and solutions since 1990.

Migrating to PowerHA 7.1.3 from out of support PowerHA/HACMP

Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0

Session Objectives Björn Rodén @ IBM Technical University in Cannes, October 2015

• This session focus on PowerHA SystemMirror Standard Edition migration considerations, when migrating from HACMP/PowerHA 6.1 to 7.1.3, including current requirements and limitations. – Cluster design and configuration considerations, single points of failure. – We will also discuss the supported upgrade and migration methods, some Do and Don't, preparation and verification, and the clmigcheck utility.

objective Thanks to: Kunal L, Rakesh S, Ravi S, Rajeev N, Steve D, Paul M, Bernd, B, Gary C, Dino Q, Sean B, Michael H, et al

You will learn about designing PowerHA 7.1.3 and migrating PowerHA 6.1 to PowerHA 7.1.3 (Standard Editions)

Business challenges & needs Björn Rodén @ IBM Technical University in Cannes, October 2015

• Information management for business processes needs to… – Ensure appropriate level of service – Manage risks (mitigate, ignore, transfer) – Reduce cost (CAPEX/OPEX)

93% 40% of companies that suffer a massive data loss will never reopen 1

of companies that lost their data center for 10 days or more due to a disaster filed for bankruptcy within one year of the disaster2

Reference: (1) “Disaster Recovery Plans and Systems Are Essential”, Gartner Group, 2001 Reference: (2) US National Archives and Records Administration

What protection is the solution expected to provide? Björn Rodén @ IBM Technical University in Cannes, October 2015

Global Distance Recovery Compliance

Data Loss or Corruption

Metro Distance Recovery High Availability

Single System Failure Human error Software error Component failures Single system failures

Local Disaster Human error Electric grid failure HAVC or power failures Burst water pipe Building fire Architectural failures Gas explosion Terrorist attack

Regional Disaster Electric grid failure Floods Hurricanes Earthquakes Tornados Tsunamis Warfighting

Business Continuity in IT perspective BjĂśrn RodĂŠn @ IBM Technical University in Cannes, October 2015

Business Continuity

Ability to adapt and respond to risks as well as opportunities in order to maintain continuous business operations

High Availability

The attribute of a system to provide service during defined periods, at acceptable or agreed upon levels and masks unplanned outages

Disaster Recovery

Capability to recover a data center at a different site if the primary site becomes inoperable

Continuous Operations

The attribute of a system to continuously operate and mask planned outages

ÂŠ Copyright IBM Corporation 2015

IT Availability Life cycle Björn Rodén @ IBM Technical University in Cannes, October 2015

DESIGN > BUILD > OPERATE > REPLACE

maintenance and change management, skill building, migration and decommissioning …

A lot to analyze, plan, do and check…

architecture, solution design, deployment, governance, system

Björn Rodén @ IBM Technical University in Cannes, October 2015

PowerHA SystemMirror

PowerHA SystemMirror Edition basics Björn Rodén @ IBM Technical University in Cannes, October 2015

• PowerHA SystemMirror for AIX Standard Edition – Automated restart of failed application – same node or peer cluster node

– Monitors, detects and reacts to events – Multiple channels for heartbeat between the systems – IP Network – SAN – Central Repository

– Direct access to SAN shared storage, with LVM mirroring – IP syncronization to remote SAN storage on other cluster node – Smart Assists, IBM supported application integration – – – –

HA agent Support – Discover, Configure, and Manage Resource Group Management – Advanced Relationships Support for Custom Resource Management Out of the box support for – DB2, WebSphere, Oracle, SAP, TSM, LDAP, IBM HTTP, etc

• PowerHA SystemMirror for AIX Enterprise Edition – Cluster management for the Enterprise (Disaster Tolerance) – – – – –

Multi-site cluster management Automated or manual confirmation of swap-over Third site tie-breaker support Separate storage synchronization Metro Mirror, Global Mirror, GLVM, HyperSwap with DS8800 (<100KM)

PowerHA SystemMirror support Björn Rodén @ IBM Technical University in Cannes, October 2015

• PowerHA 6.1 End of Support (EOS): 30-Apr-2015 (moved from originally 30-Sep-2014) – End of Support (EOS) is the last date on which IBM will deliver standard support services for a given version/release of a product. – PowerHA 6.1 is currently in extended support mode, where Service Packs are available only to the customers who have purchased the extended support agreement. • From 05/01/15 until 04/30/18 • http://www-03.ibm.com/services/supline/products/ExtendedSupport/pSeries.pdf

– Check IBM Software support lifecycles on this website: •

http://www-01.ibm.com/software/support/aix/lifecycle/index.html

R=Rolling Upgrade S=Snapshot Upgrade O=Offline Upgrade or Uninstall-Install-Reconfigure

PowerHA life cycle (sampled 12SEP15) Björn Rodén @ IBM Technical University in Cannes, October 2015

7.2

http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/Other+software/PowerHAClusterManager&release=7.1.3&platform=AIX

Some key changes in PowerHA 7.1 vs. 6.1 Björn Rodén @ IBM Technical University in Cannes, October 2015

• Architectural changes from PowerHA 6.1 (CAA/RSCT, Heatbeating, RG) – PowerHA 7.1 is built on Cluster Aware AIX (CAA) functionality which provide fundamental clustering capabilities in the base operating system. PowerHA 6.1.0 use Reliable Scalable Clustering Technology (RSCT) for clustering framework. • PowerHA 7.1.3 require AIX 6.1 TL9 SP1 or AIX 7.1 TL3 SP1 – http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101347 • Cluster Aware AIX (CAA) manage the heartbeats in 7.1, not RSCT as in 6.1 – CAA use a Repository Disk to store configuration information persistent and must be shared by all cluster nodes. • Event management with AHAFS in 7.1 – Event management is handled by using AIX pseudo file-system architecture Autonomic Health Advisor File System (AHAFS), not cluster manager and RSCT • IP multicast or unicast TCP with gossip protocol in 7.1, not unicast UDP IP as in 6.1 – With 7.1.3 unicast (TCP) is the default option in addition to multicast • Non-IP networks – diskhb, mndhb, rs232 etc removed from 7.1 • No IPAT via Replacement (HW Address Takeover / HWAT) in 7.1 • Some restrictions on changing hostname in 7.1 – Communication Path to a node can be set from 7.1.2 (IP address mapping to hostname) – Eased further in 7.1.3 (capability to dynamically modify the host name of a clustered node) • Smart Assist technology improved and extended for 7.1.3

Basic implementation flow PowerHA 7.1 Björn Rodén @ IBM Technical University in Cannes, October 2015

1. Plan for network, storage, and application – Review requirements •

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_required_aix.htm

– Eliminate single points of failure(s) – POWER8 RAS whitepaper: •

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=SA&subtype=WH&htmlfid=POW03133USEN

2. Define, prepare and configure the infrastructure – Application planning, and start and stop scripts – Networks (IP interfaces, /etc/hosts, non-IP devices) – Storage (adapters, LVM volume group, filesystem)

3. Install the PowerHA filesets 4. Configure the PowerHA environment: – Topology: • • •

Cluster, node names, PowerHA IP networks, Repository Disk and SFWcomm Multicast or unicast network for heartbeat Cluster Aware AIX (CAA) cluster

– Resources, resource groups, attributes: • • •

Resources: Application server, service label, volume group Resource group: Identify name, nodes, policies Add attributes: Application server, service label, VG, filesystem.

5. Synchronize, save configuration (snapshot) 6. Start/stop cluster services 7. Verify, test configuration

Building a dual node PowerHA cluster Björn Rodén @ IBM Technical University in Cannes, October 2015

1. Baseline each cluster node (software levels & configuration files) 2. Check all disk devices has reservation_policy set to no_reserve (NPIV on LPAR, VSCSI on VIOS): – lsdev -Cc disk -Fname|xargs -I lsattr -Pl {} -a reservation_policy // check last configured/loaded – lsdev -Cc disk -Fname|xargs -I devrsrv -c query -l {} // check current locking – lsdev -Cc disk -Fname|xargs -I chpv -Pl {} -a reservation_policy=no_reserve // change for next boot/load

3. Correlate disks and paths between cluster nodes using PVID/UUID: – lspv -u AND lsmpio (or other vendor equivalent command)

4. 5. 6. 7. 8. 9. 10.

Create a cluster (clmgr add cluster) Add service IP (clmgr add service_ip) Define application controller (clmgr add application_controller) Create resource group (clmgr add rg) Verify and synchronize cluster (clmgr sync cluster) Start cluster (clmgr start cluster) Validate cluster functionality / test plan

clmgr command: http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.3/com.ibm.powerha.admngd/clmgr_cmd.htm # # # #

clmgr clmgr clmgr clmgr

# # # #

clmgr clmgr clmgr clmgr

add add add add

cluster CL1 repository=hdisk99,hdisk98 nodes=CL1N1,CL1N2 service_ip CL1VIP network=net_ether_01 application_controller AC1 startscript="/ha/start.sh" stopscript="/ha/stop.sh" rg RG1 nodes=CL1N1,CL1N2 startup=ohn fallback=nfb service_label=CL1VIP \ volume_group=cl1vg1 application=AC1 sync cluster start cluster query cluster add snapshot CL1$(date +"%Y%m%d") © Copyright IBM Corporation 2015

PowerHA 7.1 with dual node single/dual site Björn Rodén @ IBM Technical University in Cannes, October 2015

Baseline – Ordinary run-of-the-mill dual node cluster – Using Mirror Pools for LVM mirroring – Single Virtual Ethernet adapter per node backed by the same VIOS SEA LAGG PowerHA cluster

HA1 LPAR

HA2 LPAR

– Set "Communication Path to Node" to the cluster nodes hostname network interface (using IP-address and symbolic hostname from /etc/hosts) – netmon.cf configured for ping outside the box from partition (cluster file) – /usr/es/sbin/cluster/netmon.cf – rhosts configured cluster nodes (cluster file) – /etc/cluster/rhosts – netsvc.conf configured with DNS (system file) – /etc/netsvc.conf

– Single or dual SAN Fabric – If dual sites, within a few km distance for minimal latency and throughput degradation

LVM Mirror

– Single LAN with ISL – If dual sites, use VLAN spanning Single or Dual Enterprise Storage

If the cluster node (partition) have multiple Virtual Ethernet adapters, set the "Communication Path to Node" to the IP address and Virtual Ethernet network interface device which maps to the hostname. © Copyright IBM Corporation 2015

PowerHA 7.1 with dual node single/dual site Björn Rodén @ IBM Technical University in Cannes, October 2015

Multicast between nodes •

Multicast is optional from 7.1.3 •

•

Default with 7.1.3 is TCP unicast

If desired, verify multicast is working between nodes before creating the 7.1 cluster –

PowerHA cluster

HA1 LPAR

HA2 LPAR

•

Multicast IP can be set manually, or CAA will assign one based on the nodes lower 24-bit IP address after upper 8-bit multicast of 228, such as: 192.1.2.3 => 228.1.2.3

Check assigned multicast IP: lscluster -i | grep -i multi

•

LVM Mirror

Test with the mping command: –

Start receiver first mping -r -c 100

–

Start sender mping -s -c 100

–

Use the -a <multicastip> flag to set the multicast address to be used by mping

Customer network teams seem to usually prefer to use unicast TCP for IP heartbeating instead of multicast. Single or Dual Enterprise Storage

Multi-homed nodes can set the network to private with CAA and it will not be used for heartbeating: clmgr modify network <network> PUBLIC=private lscluster -i

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.trgd/ha_trgd_test_multicast.htm http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.admngd/clmgr_cmd.htm © Copyright IBM Corporation 2015

PowerHA 7.1 with dual node single/dual site Björn Rodén @ IBM Technical University in Cannes, October 2015

Repository Disk •

The cluster repository disk is used as the central repository for the cluster configuration data. •

PowerHA cluster

HA1 LPAR

HA2 LPAR

When CAA is configured with repos_loss mode set to assert and CAA loses access to the repository disk, the system automatically shuts down.

• •

Access from all nodes and paths. Start with ~10GB for up to 32 nodes (min=512MB, max=460 GB, thin provisioning is supported). Direct access by CAA only, raw disk I/O. Define a spare for the repos disk.

• • –

• •

LVM Mirror

repo Single or Dual Enterprise Storage

Verify the disk reserve attribute is set to no_reserve

Do not manually write to the repos disk ! Check repos disk status /usr/es/sbin/cluster/utilities/clmgr query repository /usr/lib/cluster/clras lsrepos /usr/lib/cluster/clras dumprepos /usr/lib/cluster/clras dumprepos -r <reposdisk> /usr/lib/cluster/clras dpcomm_status If IP heartbeating fails, cluster nodes will keep alive if the repository disk is accessible from all nodes.

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.clusteraware/claware_repository.htm https://www.ibm.com/developerworks/community/blogs/6eaa2884-e28a-4e0a-a1587931abe2da4f/entry/powerha_caa_repository_disk_management © Copyright IBM Corporation 2015

PowerHA 7.1 with dual node single/dual site Björn Rodén @ IBM Technical University in Cannes, October 2015

Storage Framework •

Fibre Channel adapters with target mode support only (verfify actual dd setting with lsattr with -P flag) – – –

PowerHA cluster

HA1 LPAR

HA2 LPAR

•

All physical FC adapters WWPNs zoned –

One Fabric supported with SFWcomm

–

For dual Fabric, it is supposed to work, if it do not work with your implementation and system software levels, please open a PMR with IBM Support

TM-ZONE •

LPM do not migrate SFWcomm configuration –

LVM Mirror

It is recommended that SAN communication be reconfigured after LPM is performed

•

Using datalink layer communication over VLAN between AIX cluster node and VIOS with the physical FC adapters

•

Check SFWcomm status – – –

Single or Dual Enterprise Storage

On fcsX tme=yes On fscsiX dyntrk=yes & fc_err_recov=fast_fail Enable the new settings (reboot)

lscluster -i /usr/lib/cluster/clras sfwinfo -a /usr/lib/cluster/clras sancomm_status

If the IP hearbeat and repository disk are not sufficient to meet heartbeat requirements, also enable SFWcomm.

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.clusteraware/claware_comm_setup.htm http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.concepts/ha_concepts_ex_san.htm © Copyright IBM Corporation 2015

PowerHA IP heartbeating over VIOS SEA Björn Rodén @ IBM Technical University in Cannes, October 2015

• The heartbeat function is used as a reliable means of monitoring an adapter's state over a long period of time. • When heartbeating is broken, a decision has to be made as to whether the local adapter has gone bad, or the neighbour (or something between them) has a problem. • The local node only needs to take action if the local adapter is the problem; if its own adapter is good, then we assume it is still reachable by other clients regardless of the neighbour's state (the neighbour is responsible for acting on its local adapters failures). • This decision (local vs remote bad) is made based on whether any network traffic can be seen on the local adapter, using the inbound byte count of the interface (without netmon.cf enabled). • Where Virtual Ethernet is involved, this test becomes unreliable since there is no way to distinguish whether inbound traffic came in from the VIO server's connection to the outside world, or just from a neighbouring VIO client, new (VIOS 2.2.3.4) poll_uplink feature is enabled which can detect a physical adapter failure on the VIOS side and propagate the cluster node. • Configure netmon.cf for Virtual Ethernet and single adapter PowerHA cluster node network adapters. Consierations regarding multiple IP heartbeat networks over virtual Ethernet 1.

For dual node single site clusters, one IP network is ordinarily sufficient, if backed by dual VIOS SEA as per PowerVM Virtualization Best Practice – the base/boot IP and service IPs can be on the same routable subnet. 2. Using additional Virtual Ethernet over same VIOS Shared Ethernet Adapter do not improve redundancy 3. Using additional Virtual Ethernet over different Shared Ethernet Adapter on same VIOS do not improve redundancy 4. Using additional Virtual Ethernet over different Shared Ethernet Adapter on different VIOS might improve redundancy (also depending on network switch and routing layer) • With separate hypervisor virtual switches for each • With partition link aggregation in network backup interface mode Can use a single PowerHA/CAA IP heartbeat network also for SRIOV/dedicated adapters • With dual SRIOV ports from separate Ethernet adapters assigned to each cluster node (in separate servers), each port connected to a separate network switch and configured with link aggregation in Network Interface Backup mode. © Copyright IBM Corporation 2015

PowerHA cluster heartbeat settings Björn Rodén @ IBM Technical University in Cannes, October 2015

• The heartbeat function is configured to use specific paths between nodes. These paths allow heartbeats to monitor the health of all PowerHA networks, network interfaces, and nodes. – Node Failure Detection Timeout •

The time in seconds for the health management layer to wait before sending a message that the node has failed. The valid values are in the range 10 - 600.

– Node Failure Detection Grace Time •

The time in seconds for the node to wait before declaring that a node has actually failed. The valid values are in the range 5- 600. This function starts after the Node Failure Detection Timeout field.

# clctrl -tune -L | egrep "node_down_delay|node_timeout|SCOPE" NAME DEF MIN MAX UNIT SCOPE node_down_delay 10000 5000 600000 milliseconds c n node_timeout 20000 10000 600000 milliseconds c n

– Link Failure Detection Timeout •

The time in seconds for the health management layer to wait before sending a message that the links between the sites have failed. A link failure can cause the cluster to fallover to another link and continue to function. If all the links in the cluster are not responding, a message is sent identifying that site is in an offline state.

– Site Heartbeat Cycle •

The heartbeat between sites in the cluster. The valid values are in the range 1 - 10. This value represents the ratio of site heartbeats to local heartbeats. For example, if this value is 10 and the Local Heartbeat Cycle value is 10 a heartbeat is sent every 1 second between the sites.

– Netmon •

If netmon.cf is configured: When netmon needs to stimulate the network to ensure adapter function, it sends ICMP ECHO requests to each IP address. After the last failed ICMP ECHO REPLY, there is currently a 5s trigger time before network is declared down. Open PMR to request “Tunable FDR iFix bundle" for netmon.cf.

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.admngd/ha_admin_config_hacmp_heartbeat.htm https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds1/clctrl.htm http://www-01.ibm.com/support/docview.wss?uid=isg1IV44428 © Copyright IBM Corporation 2015

Cluster Topology Configuration – netmon.cf facility 1/2 Björn Rodén @ IBM Technical University in Cannes, October 2015

With this feature network link and network switch failure can be detected by the cluster node, also several hops away by pinging to remote nodes.

•

For single adapter PowerHA cluster node network adapters, use the netmon.cf configuration file: – /usr/es/sbin/cluster/netmon.cf When netmon needs to stimulate the network to ensure adapter function, it sends ICMP ECHO requests to each IP address. After the last failed ICMP ECHO REPLY, there is currently only a 5s trigger time before network is declared down. Specify remote hosts that are not in the cluster configuration and that can be accessed from PowerHA interfaces, and who reply consistently to ICMP ECHO without delay, and use multiple hosts not only one. Up to 32 different targets can be provided for each interface, if *any* given target is pingable, the adapter will be considered up (ICMP ECHO).

!REQD <owner> <target> Parameters: ---------!REQD : An explicit string; it *must* be at the beginning of the line (no leading spaces). <owner> : The interface this line is intended to be used by; that is, the code monitoring the adapter specified here will determine its own up/down status by whether it can ping any of the targets (below) specified in these lines. The owner can be specified as a hostname, IP address, or interface name. In the case of hostname or IP address, it *must* refer to the boot name/IP (no service aliases). In the case of a hostname, it must be resolvable to an IP address or the line will be ignored. The string "!ALL" will specify all adapters. <target> : The IP address or hostname you want the owner to try to ping. As with normal netmon.cf entries, a hostname target must be resolvable to an IP address in order to be usable.

Open PMR and request “Tunable FDR iFix bundle" for netmon.cf heatbeat tunables options (a design change request has been accepted MR1202141628)

http://www-01.ibm.com/support/docview.wss?uid=isg1IZ01332 © Copyright IBM Corporation 2015

Cluster Topology Configuration – netmon.cf facility 1/2 Björn Rodén @ IBM Technical University in Cannes, October 2015

# lscluster –i | egrep "Node|Interface num|IPv4 ADDRESS" Node CL01 Interface number 1, en0 IPv4 ADDRESS: 10.10.226.231 broadcast 10.10.226.255 netmask 255.255.255.0 IPv4 ADDRESS: 10.10.226.233 broadcast 10.10.226.255 netmask 255.255.255.0 Interface number 2, en1 IPv4 ADDRESS: 10.10.227.85 broadcast 10.10.227.255 netmask 255.255.255.1 Interface number 3, en2 IPv4 ADDRESS: 10.10.229.232 broadcast 10.10.229.255 netmask 255.255.255.0 Node CL02 Interface number 1, en0 IPv4 ADDRESS: 10.10.226.232 broadcast 10.10.226.255 netmask 255.255.255.0 Interface number 2, en1 IPv4 ADDRESS: 10.10.227.86 broadcast 10.10.227.255 netmask 255.255.255.0 Interface number 3, en3 IPv4 ADDRESS: 10.10.229.233 broadcast 10.10.229.255 netmask 255.255.255.0

# cat !REQD !REQD !REQD

/usr/es/sbin/cluster/netmon.cf en0 10.10.226.1 en1 10.10.227.1 en2 10.10.229.1

All IP-based networks which are configured, will by default be used for IP heartbeating. To disable heartbeating on a IP-network set the Network Attribute to “private” from the default “public”. http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.admngd/ha_admin_steps_change_ipbased.htm © Copyright IBM Corporation 2015

Basic PowerHA cluster functionality verification Björn Rodén @ IBM Technical University in Cannes, October 2015

Verify PowerHA cluster functionality – – –

After system functionality verification (file systems, users, network, backup, etc) Before or after cluster application server verification (start/stop/monitor integration hardening) Before end-to-end application resiliency verification (environment/enterprise wide failure scenarios)

Procedure

Actions --- EXAMPLES

Reboot both NODE1 & NODE2 and restart HA on both RG stop on NODE1 w/RG on NODE1 RG start on NODE1 RG stop on NODE1 w/RG on NODE1 RG start on NODE2 RG stop on NODE2 w/RG on NODE2 RG move from NODE2 to NODE1 w/RG on NODE2 RG move from NODE1 to NODE2 w/RG on NODE1 IP Failure test NODE1 Reintegrate NODE1 IP Failure test NODE2 Reintegrate NODE2 IP Failure test NODE1&NODE2 Reintegrate NODE1 & NODE2 Stop of PowerHA on NODE1 w/ migration to NODE2 Re-start PowerHA on NODE1 to reintegrate Stop of PowerHA on NODE2 w/ migration to NODE1 Re-start PowerHA on NODE2 to reintegrate SAN Availability Test on NODE1 Reintegrate NODE1 SAN SAN Availability Test on NODE2 Reintegrate NODE2 SAN HMC Power Off of NODE1 w/ RG on NODE1 HMC Activate of NODE1 & Restart Power-HA on NODE1 HMC Power Off of NODE2 w/ RG on NODE2 HMC Activate of NODE2 & Re-start Power-HA on NODE2 Reboot both NODE1 & NODE2 and restart HA on both

shutdown –F + chsysstate clRGmove -d clRGmove -u clRGmove -d clRGmove -u clRGmove -d clRGmove -m clRGmove -m ifconfig en# down w/ RG on NODE1 ifconfig en# up on NODE1 ifconfig en# down w/ RG on NODE2 ifconfig en# up on NODE2 ifconfig en# down on NODE1 & NODE2 ifconfig en# down on NODE1 & NODE2 cl_clstop

Excepted outcome Actual outcome

cl_clstop SAN Admin or unmap VSCSI/VFC SAN Admin or unmap VSCSI/VFC SAN Admin or unmap VSCSI/VFC SAN Admin or unmap VSCSI/VFC chsysstate chsysstate chsysstate chsysstate shutdown –F + chsysstate © Copyright IBM Corporation 2015

Björn Rodén @ IBM Technical University in Cannes, October 2015

Identify Points of Failure

Redundancy and Single Points of Failure (SPOF) Björn Rodén @ IBM Technical University in Cannes, October 2015

Björn Rodén

ISP (external) Enterprise environment FW/IPS Your major goal throughout the Data Centre environmentprocess is to eliminate single planning Routers Server points ofServer failure Server and verify redundancy. Site environment

Storage

Find the

SPOF

Storage

MA N WA N SAN

UPS Gen .

A single point of failure exists when a Switches critical Service function is provided by a Application single component. Network Middleware Operating System & Servers System Software

Area Network If thatLocal component fails, the Service has no Logical/Virtual Machine Storage Storage Area Network other way of providing that function, and Physical Machine the application or service dependent on Network Switches Storage that component becomes unavailable.

Kernel stack

Hypervisor

http://publib.boulder.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.powerha.plangd/ha_plan_over_ppg.htm Hardware (cores, cache, nest)

Storage

Eliminating SPOF by using redundant components Björn Rodén @ IBM Technical University in Cannes, October 2015 Cluster components

To eliminate as single point of failure

PowerHA SystemMirror supports

Nodes Power sources

Use multiple nodes Use multiple circuits or uninterruptible power supplies

Up to 16. As many as needed.

Networks

Use multiple networks to connect nodes

Up to 48.

Network interfaces, devices, and labels

Use redundant network adapters

Up to 256.

TCP/IP subsystems

Use networks to connect adjoining nodes and clients

As many as needed.

Disk adapters Controllers Disks

Use redundant disk adapters As many as needed. Use redundant disk controllers As many as needed. Use redundant hardware and disk mirroring, striping, or As many as needed. both

Applications

Assign a node for application takeover, to configure an Flexible configuration policies for high availability within a application monitor, and to configure clusters with nodes site and between sites. at more than one site.

Sites

Use more than one site for disaster recovery.

Up to two sites.

Resource groups

Use resource groups to specify how a set of entities should perform.

Up to 64 per cluster.

Cluster resources

Use multiple cluster resources.

Up to 128 for the clinfo daemon (more can exist).

Virtual I/O Server (VIOS)

Use redundant VIOS

As many as needed.

HMC Managed System hosting a cluster node

Use redundant HMC Use separate managed systems for each cluster node

Up to 2. Up to 16.

Cluster repository disk

Use RAID protection

One active repository disk per site that has the ability to replace the disk after a failure. You must have a spare disk that is available to replace the failed repository disk in the live cluster.

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.plangd/ha_plan_eliminate_spf.htm © Copyright IBM Corporation 2015

Cluster partitioning, aka node isolation or “split brain” 1/2 Björn Rodén @ IBM Technical University in Cannes, October 2015

• Cluster partitioning, aka node isolation or “split brain”, is a failure situation where more than one server acts as a primary. – Partitioning occurs when a cluster node stops receiving all interconnecting heartbeat traffic from its peer-node, and assumes that the peer-node has failed. – Due to the lack of synchronization, a split brain situation is problematic and can cause undesirable behaviour, such as data corruption. – Once the peer-node is determined to be down due to lack of heartbeats, both nodes on each side of the cluster attempt to take over resources (if so configured) from a node that is actually still active and running. – When the interconnection is restored and hearbeats resume, the cluster will merge and at this point, the cluster manager identify that a partitioning has occurred, and the cluster node with the highest node number will stop itself immediately. – During partitioning, if both nodes have acquired its respective peer-nodes resource groups and have had applications running with users connected and updating data for the same application on both nodes separately, data integrity is lost.

Cluster partitioning, aka node isolation or “split brain” 2/2 Björn Rodén @ IBM Technical University in Cannes, October 2015

Common approaches regarding cluster partitioning: – Maximize independent interconnects between locations/sites •

Use multiple IP and non-IP interconnects for cluster node heartbeats, with all physical links provided separately, and well isolated from failure at the same time, such as: – Dual IP-networks (LAN), each over separate physical adapters and network switches, and interconnection between cluster node sites. – Dual non-IP-networks (SAN), each over separate physical adapters and network switches, and interconnection between cluster node sites. – Consider using a third network interconnect for heartbeat only between nodes, such as if primary interconnections between nodes/sites use DWDM, use a non-landbased or VPN over ISP connection.

– Use third site as tie breaker • •

Using “Tie-Breaker” concept, where a third site disk or node is used to determine surviving partition. Optimally also use separate physical interconnect from each cluster node site to the third site.

•

For PowerHA refer to: –

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.admngd/ha_admin_mergesplit_policy_713.htm

– Classify node-failure as site-down event and/or start secondary by operator • •

Active site declares itself down and expect that secondary site will take over the failed services, secondary site takes over services if communication is lost to previous active site. Active site declares itself down, and secondary site is started by operator.

– Accept as-is •

Decide that the risk for partitioning occurring is unlikely, the cost for redundancy is too high, and accepting longer downtime relying on backup restore in case of data inconsistency.

NOTE: External access to cluster nodes can still be available, even if site interconnects fail between the cluster nodes. © Copyright IBM Corporation 2015

Migrating to PowerHA 7.1.3 Björn Rodén @ IBM Edge 2015 May 11-15 The Venetian Las Vegas, Nevada

Migration process to PowerHA 7.1.3 from 6.1 Björn Rodén @ IBM Technical University in Cannes, October 2015

• Validate required design changes to current implementation – Plan service windows after migration procedure validation on test systems, review for updates: •

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_mig61.htm

• Verify current PowerHA 6.1 availability functionality – Run cluster verification and make sure no errors are reported, validate swap capability and LAN/SAN

• Verify PowerHA 7.1 preconditions, heartbeat networks and SPOFs • AIX upgrade during rolling upgrade or separate service window – Upgrade all nodes in the cluster to AIX 6.1 TL9 SP1 or AIX 7.1 TL3 SP1 or higher •

Recommend: AIX 7.1 TL3 SP5 or higher – with lower SP also APARs IV58849, IV59761, IV60736, IV65472, IV66180, IV66606

– Verify install additional CAA filesets, bos.cluster.rte, bos.ahafs, bos.clvm.enh, devices.common.IBM.storfwork.rte, service packs, and the MIG2 package: • •

For AIX 6.1 https://aix.software.ibm.com/aix/ifixes/PHA_Migration/MIG2_6195.tar For AIX 7.1 https://aix.software.ibm.com/aix/ifixes/PHA_Migration/MIG2_7135.tar

• Migrate the PowerHA 6.1 cluster – New install and configure •

Design and install PowerHA cluster from scratch.

– Rolling migration •

You can upgrade a PowerHA cluster while keeping your applications running and available, during the upgrade process, a new version of the software is installed on each cluster node while the remaining nodes continue to run the earlier version.

– Offline upgrade •

This type of migration involves bringing down the entire PowerHA cluster, reconfiguring the active cluster to fit, installing the new PowerHA and restarting cluster services one node at a time.

– Snapshot upgrade •

This type of migration involves bringing down the entire PowerHA cluster, reconfiguring the snapshot configuration, installing the new PowerHA and restarting cluster services one node at a time.

• Verify cluster and high availability functionality – Cluster system functionality tests, component failure tests, failure scenario tests © Copyright IBM Corporation 2015

Todo before migration Björn Rodén @ IBM Technical University in Cannes, October 2015

•

Software levels for currency – – –

• • • •

Upgrade AIX and RSCT to supporting levels and ensure that the same level of cluster software (including PTFs and APARs) are on all nodes before beginning a migration Ensure that the PowerHA cluster software is committed (not applied) When performing a rolling migration, all nodes in the cluster must be upgraded to the new base release before applying any updates for that release

Run cluster verification and make sure no errors are reported Take a snapshot of the cluster configuration Backup and mksysb Use the /usr/sbin/clmigcheck tool 7.1

AIX 6.1 TL6+ AIX 7.1

7.1.1

AIX 6.1 TL7 SP2 AIX 7.1 TL1 SP2

RSCT 3.1.2.0 or higher for both AIX 6.1 and 7.1

7.1.2

AIX 6.1 TL8 SP1 AIX 7.1 TL2 SP1

RSCT 3.1.2.0 or higher for both AIX 6.1 and 7.1

AIX 6.1 TL9 SP1 AIX 7.1 TL3 SP1

RSCT 3.1.2.0 or higher for both AIX 6.1 and 7.1, should be 3.2.0.5 or higher, see RSCT note in red

7.1.3

AIX 6.1 RSCT 3.1.0.0 or higher AIX 7.1 RSCT 3.1.0.0

The "Communication Path to Node" on the PowerHA cluster nodes must be set to an IP-address mapping to the hostname. All cluster node hostnames must be resolved locally using the /etc/hosts file (IP address and label), use netsvc.conf, irs.conf or NSORDER in /etc/environment to set the order. Pre-7.1.3: After you have synchronized the initial cluster configuration, it is not supported to change the hostname or IP resolution of the hostname.

RSCT: Starting in rsct.core.rmc 3.1.5.0 (and continuing into the 3.2.0.0 release), a memory leak in CAA-specific code paths of the IBM.ConfigRM subsystem may lead to library calls failing which can cause ConfigRM to believe the CAA domain is being shut down, causing it to go through offline processing of the RSCT domain, including stopping cthags. Time to failure after a new boot is estimated to be between 4 and 8 months. AIX 7.1 TL3 SP5 has rsct.core.rmc to 3.2.0.5 level and is not affected by the APAR: http://www-01.ibm.com/support/docview.wss?uid=isg1IV69760 http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_required_aix.htm © Copyright IBM Corporation 2015

Todo before migration Björn Rodén @ IBM Technical University in Cannes, October 2015

•

Review migration advise documentation for updates – –

•

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_mig61.htm http://www-01.ibm.com/support/docview.wss?uid=isg3T1022047

Verify cluster conditions and settings – – – –

Use clstat to review the cluster state and to make certain that the cluster is in a stable state Review the /etc/hosts file on each node to make certain it is correct Review the /etc/netsvc.conf (equiv) file on each node to make certain it is correct Review the /usr/es/sbin/cluster/netmon.cf file on each node to make certain it is correct •

–

• •

Configurations with IPAT via replacement or hardware address takeover (MAC address) Configurations with heartbeat via IP aliasing Configurations with non-IP networking, such as RS232, TMSCSI/SSA, DISKHB or MNDHB Configurations which use other than Ethernet for network communication, such as FDDI, ATM, X25, TokenRing Note that clmigcheck doesn't flag an error if DISKHB network is found and PowerHA migration utility automatically takes care of removing that network

SAN storage for Repository Disk and Target Mode – –

•

After AIX Version 6.1.6, or later is installed, enter the fully qualified host name of every node in the cluster in the /etc/cluster/rhosts file

Take a snapshot of the cluster configuration and save off customized scripts, such as start, stop, monitor and event script files Remove configurations which can’t be migrated – – – – –

•

If VIOS and AIX levels support, use the poll_uplink feature instead for link status propagation over Virtual Ethernet

The repository is stored on a disk that must be SAN attached and zoned to be shared by every node in the cluster and only the nodes in the cluster – and not part of a volume group SAN zoning of FC adapters WWPN for Target Mode communication

Multicast IP address for the monitoring technology (optional) – – –

You can explicitly specify multicast addresses, or one will be assigned by CAA Ensure that multicast communication is functional in your network topology before migration Note that from PowerHA 7.1.3 unicast is default © Copyright IBM Corporation 2015

clmigcheck tool (1/2) Björn Rodén @ IBM Technical University in Cannes, October 2015

clmigcheck tool is part of base AIX from 6.1 TL6 or 7.1 (/usr/sbin/clmigcheck) – An interactive tool that verifies the current cluster configuration, checks for unsupported elements, and collects additional information required for migration (node configuration in /var/clmigcheck/clmigcheck.txt) – Saves migration check to file /tmp/clmigcheck/clmigcheck.log – Run this command on all cluster nodes, one node at a time, before installing PowerHA 7.1.3 – When the clmigcheck command is run on the last node of the cluster before installing PowerHA 7.1.3, the CAA infrastructure will be started (check with lscluster -m command).

----------[PowerHA System Mirror Migration Check] ------------Please select one of the following options: 1 = Check ODM configuration. 2 = Check snapshot configuration. 3 = Enter repository disk and multicast IP addresses.

2015 clmigcheck enhanced • Dry run mode (option -V)of to walk through "x" without install: Select one the above, to migration, exit or "h" for help: • For AIX 6.1 https://aix.software.ibm.com/aix/ifixes/PHA_Migration/MIG2_6195.tar • For AIX 7.1 https://aix.software.ibm.com/aix/ifixes/PHA_Migration/MIG2_7135.tar • Many more checks added to fail migration with clear guidance and bug fixes Check for AIX UPDATES to clmigcheck ! © Copyright IBM Corporation 2015

clmigcheck tool (2/2) Björn Rodén @ IBM Technical University in Cannes, October 2015

• Download – For AIX 6.1 https://aix.software.ibm.com/aix/ifixes/PHA_Migration/MIG2_6195.tar – For AIX 7.1 https://aix.software.ibm.com/aix/ifixes/PHA_Migration/MIG2_7135.tar

• The tarball contains three separate epkgs, one each for CAA, RSCT, and PowerHA: – The CAA and RSCT epkgs should be installed prior to the PowerHA migration and require a reboot. •

Because they both require a reboot, it is recommended to install both CAA and RSCT epkgs and then perform a single reboot (using shutdown -r).

– The PowerHA epkg includes a fix to cluster manager and hence would require Cluster manager to be stopped and started. •

To avoid additional downtime, it is recommended that this fix package be installed as part of the PowerHA migration procedure, immediately after upgrading to PowerHA SystemMirror 7.1.3 with Service Pack 3.

• To install an epkg, enter: – emgr -e <epkg>

• The epkgs contained in this tarball are: – MIG2_7135.151023.epkg.Z (CAA) – PHA2_7133.150813.epkg.Z (PowerHA) – rsctHA7a1.150911.epkg.Z (RSCT)

From Knowledgecentre on 7.1.3 Your mileage may vary

Rolling Migration Overview Steps 1/2 Björn Rodén @ IBM Technical University in Cannes, October 2015

Review migration advise documentation for updates – –

2. 3.

Stop cluster services on one node (move resource group as needed) Upgrade AIX (if needed) and reboot, and verify: – –

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_mig61.htm http://www-01.ibm.com/support/docview.wss?uid=isg3T1022047

/etc/hosts, /etc/netsvc.conf, /etc/cluster/rhosts (/usr/es/sbin/cluster/netmon.cf or poll_uplink on VIOS and node), That CAA related entries were successfully added in the files /etc/inetd.conf, /etc/services, and the /etc/initab.conf file, run the following command: egrep "caa|clusterconf" /etc/services /etc/inetd.conf /etc/inittab

On the first node, check the Object Data Manager (ODM) configuration with clmigcheck option two (2) – The following message is displayed: The ODM has no unsupported elements. – Note: If the cluster cannot be migrated, an error message is displayed. Remove all the unsupported elements from the PowerHA SystemMirror configuration. After these elements are removed you must verify and synchronize the changes.

5. 6.

Run clmigcheck option four (4) and enter the shared disks for only the first node. Install PowerHA SystemMirror 7.1.3 on the first node – First the base filesets from one directory and then corresponding service packs from another directory, if in the same installation directory it might affect the order of installation and cause errors.

7. 8.

Using SMIT, start cluster services on the first node in the cluster. Complete the following steps for each remaining node in the cluster: – From the command line, run the clmigcheck command. – Note: When you run the /usr/sbin/clmigcheck command on the last node in the cluster, the Cluster Aware AIX (CAA) cluster is created. Type lscluster –m to verify that the CAA cluster is created on all the nodes in the cluster. If the CAA cluster was not created, contact IBM support to resolve the problem, and then complete the migration.

Using SMIT, start cluster services on the current node before you complete the migration steps for the remaining nodes in the cluster. – Note: Bring cluster services online on all nodes in a cluster to complete the migration process.

Rolling Migration Overview Steps 2/2

The core process below worked on several 7.1.3 service pack and AIX 7.1 levels (with some tweaks due to bugs on occasion)

Björn Rodén @ IBM Technical University in Cannes, October 2015

• Stop cluster services on one node (move rg as needed) • Upgrade AIX (if needed) and reboot – Also install additional CAA filesets, bos.cluster and bos.ahafs

• Verify /etc/hosts, /etc/netsvc.conf, /etc/cluster/rhosts (and if used /usr/es/sbin/cluster/netmon.cf or poll_uplink) • Update /etc/cluster/rhosts (if needed) – Enter cluster node hostname IP addresses. Only one IP address per line.

• Refresh -s clcomd • Run clmigcheck (option1, then option 3) • Upgrade PowerHA – Install base level install images and complete upgrade procedures – Then comeback and apply latest SPs on top of it. Can be done non-disruptively.

• Review the /tmp/clconvert.log file • Restart cluster services (move rg back if needed) • Repeat steps above for each node (minus the additional options on clmigcheck)

Key IT Availability Metrics

Björn Rodén @ IBM Edge 2015 May 11-15 The Venetian Las Vegas, Nevada

What are your key Availability Requirements? Björn Rodén @ IBM Technical University in Cannes, October 2015

Recovery Time Objective (RTO) How long time can you afford to be without your systems?

Recovery Point Objective (RPO) How much data can you afford to recreate or lose?

Maximum Time To Restart/Recover (MTTR) How long time until services are restored for the users?

Degree of Availability (Coverage Requirement) Annual percentage of a given time period when the business service should be available?

Notes on Degree of Availability Björn Rodén @ IBM Technical University in Cannes, October 2015

• IT service availability can be measured in percentage of a given when the business service is available for it’s intended purpose

time period

– Usually expressed with a number of nines (9) over a year (rounded): • 99% => 88 hours/year • 99.9% => 9 hours/year • 99.95% => 4 1/2 hours/year • 99.99% => 52 min/year < 1h • 99.999% => 5 min/year • 99.9999% => ½ min/year

• IT System vs. IT Service (ripple effect) – e.g. IT service dependent on five IT systems, if all target levels are met but not at the same time: • PROBABILITY((99.9*99.9*99.5*99.5*99.0)/1005) => 97.82% or 191-192h/period • MINIMUM(99.9*99.9*99.5*99.5*99.0) => 99.00% or 88h/period

• Determine the time period for the degree of availability – Are time for planned maintenance excluded during the year? •

Such as planned service windows and/or fixed number of days per month/quarter

– How many hours are used per year • Calendar year hours – 8760 h for 365 days non-leap years – 8784 h for 366 days leap years

• Decided amount of time per year (global coverage with 24 time zones, add one day) – 365 days (non-leap), then if global coverage add 24h d/y=366 or 8784 h – 366 days (leap), then if global coverage add 24h d/y=367 or 8808 h

Common Availability and Disaster requirements Björn Rodén @ IBM Technical University in Cannes, October 2015

Disaster Tolerance

High Availability • • • • • • • • •

RPO – zero (or near zero) data loss RTO – measured in minutes at the most NRO – zero PRO – zero from UPS & generator Coverage Requirement (e.g. 24x7 / 24x365) Degree of Availability (e.g. 99.9% or ~9h/year) No single point-of-failure (SPOF) – System level Geographic affinity (Metro distance) Automatic failover/continuance/recovery to redundant components including application components – up to in-flight transaction integrity

• RPO – near zero data loss (may require manual recovery of orphaned data) • RTO/NRO – measured in hours, days, weeks • PRO – depend on generator fuel storage • Maximum Tolerable Period of Degraded Operations • Maximum Time To Restart/Recover (MTTR) • Business Process Recovery Objective (BPRO) • No single point-of-failure (SPOF) – DC level • Geographic dispersion (Global distance) • Declaring disaster is a management decision • Rotating site swap or periodic site swap • Full or Partial swap

Timeline Checkpoint in Time

RPO

Outage

Minimum Service Delivery

System repair

Service Delivery at 100%

New Business RTO

Your Recovery Objectives - Example

Ducks & Docs Björn Rodén @ IBM Edge 2015 May 11-15 The Venetian Las Vegas, Nevada

Get the ducks in a row Björn Rodén @ IBM Technical University in Cannes, October 2015

• Know why – Business and regulatory requirements – Services, Risks, Costs – Key Performance Indicators (KPIs)

• Understand how – Architect, Design, Plan

• Can implement – Build, verify, inception, monitor, maintain, skill-up

• Will govern – – – – –

Service and Availability management Change, Incident and problem management Security and Performance management Capacity planning Migrate, replace and decommission

Further reading Björn Rodén @ IBM Technical University in Cannes, October 2015

• PowerHA for AIX • • •

http://www-03.ibm.com/systems/power/software/availability/aix/index.html https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/61ad9cf2-c6a3-4d2c-b779-61ff0266d32a/page/High%20Availability https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000001611&ps=25

• PowerHA migration • • •

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_mig61.htm http://www.ibm.com/support/docview.wss?uid=isg3T1022047 http://www.ibm.com/developerworks/aix/library/au-aix-powerha-cluster-migration/

• PowerHA latest fixes •

https://www-304.ibm.com/webapp/set2/flrt/liteTable?prodKey=hacmp

•

http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/Other+software/PowerHAClusterManager&release=7.1.3&platform=AIX&function=all

• PowerHA for AIX Version Compatibility Matrix •

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101347

• PowerHA Hardware Support Matrix •

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105638

• PowerHA 7.1 Knowledge Center • • • •

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.navigation/powerha_pdf.htm http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_offline_61to710.htm http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_upgrade_snapshot_61to71x.htm http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_rolling_migration_61to710.htm

• What's new in PowerHA 7.1 •

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.powerha.navigation/powerha_whatsnew.htm

• PowerHA 7.1.3 and 7.2 Release Notes • •

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS5241 http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appname=gpateam&supplier=897&letternum=ENUS215-389

• PowerHA 7.1.3 Announcement letter •

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS213-416

• PowerHA SystemMirror for AIX 7.1.3 Enhancements •

http://www.redbooks.ibm.com/abstracts/tips1097.html

• PowerHA SNMP monitoring •

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.admngd/ha_admin_snmpdv3.htm?lang=en

• PowerHA on Linkedin •

Thank you – Tack ! Björn Rodén @ IBM Technical University in Cannes, October 2015

Growing your IBM skills – a new model for training Björn Rodén @ IBM Technical University in Cannes, October 2015

Meet the authorized IBM Global Training Providers in the Edge Solution Showcase Global Skills Initiative

• Access to training in more cities local to you, where and when you need it, and in the format you want •

Use IBM Training Search to locate training classes near to you

• Demanding a high standard of quality / see the paths to success •

Learn about the New IBM Training Model and see how IBM is driving quality

•

Check Training Paths and Certifications to find the course that is right for you

• Academic Initiative works with colleges and universities to introduce real-world technology into the classroom, giving students the handson experience valued by employers in today’s marketplace • www.ibm.com/training

IBM Systems Lab Services and Training

Björn Rodén @ IBM Edge 2015 May 11-15 The Venetian Las Vegas, Nevada