Simplified access to the NVIDIA CUDA toolkit on SUSE Linux for HPC

Simplified access to the NVIDIA CUDA toolkit on SUSE Linux for HPC

Overview

The High-Performance Computing industry is rapidly embracing the use of AI and ML technology in addition to legacy parallel computing. Heterogeneous Computing, the use of both CPUs and accelerators like graphics processing units (GPUs), has become increasingly more common and GPUs from NVIDIA are the most popular accelerators used today for AI/ML workloads.

To get the full advantage of NVIDIA GPUs, you need to use the CUDA parallel computing platform and programming toolkit. The CUDA Toolkit includes GPU-accelerated libraries, a compiler, development tools and the CUDA runtime. 

To get the full advantage of NVIDIA GPUs, you need to use NVIDIA CUDA, which is a general purpose parallel computing platform and programming model for NVIDIA GPUs. The NVIDIA CUDA Toolkit includes GPU-accelerated libraries, a compiler, development tools and the CUDA runtime.

CUDA supports the SUSE Linux operating system distributions (both SUSE Enterprise and OpenSUSE) and NVIDIA provides a repository with the necessary packages to easily install the CUDA Toolkit and NVIDIA drivers on SUSE.

To simplify installation of NVIDIA CUDA Toolkit on SUSE Linux Enterprise for High Performance Computing (SLE HPC) 15, we have included a new SUSE Module, NVIDIA Compute Module 15. This Module adds the NVIDIA CUDA network repository to your SLE HPC system. You can select it at installation time or activate it post installation. This module is available for use with all SLE HPC 15 Service Packs.

Note that the NVIDIA Compute Module 15 is currently only available for the SLE HPC 15 product.

Post-Installation Process via Yast

Installing via Yast

  1. Start Yast and select System Extensions
  1. After YaST checks the registration for the system, a list of modules that are installed or available is displayed.

Click on the box to select the NVIDIA Compute Module 15 X86-64

Notice that a URL for the EULA is included in the Details section. Please comply with the NVIDIA EULA terms.

  1. Information on the EULA for the CUDA drivers is displayed.

Agree and click Next

  1. You must trust the GnuPG key for the CUDA repository.

Click Trust

  1. You will be given one more confirmation screen

Click Accept

  1. After adding the repository, you can install the CUDA drivers.

Start Yast and select Software Management” then search for cuda

Select the cuda meta package and press Accept

  1. A large number of packages will be installed

Press Continue to proceed

That’s it!

You are now ready to start using the CUDA toolkit to harness the power of NVDIA GPUs.

Summry

Managing heterogeneous computing environments has become increasingly important for HPC and AI/ML administrators. The NVIDIA Compute Module is one way we are working to make using these technologies easier to use.

Posted in High Performance Computing, Information Technology, Open Source | Tagged , , , , , , , | Leave a comment

Announcing the SUSE Linux for Raspberry Pi LinkedIn group

Introduction

We have supported SUSE Linux on various models of Raspberry Pi systems since 2018. Although we have seen a variety of use cases for SUSE Linux on Raspberry Pi devices, we really did not have a good forum for people to share technical information for using SUSE Linux on the Raspberry Pi.

To address this limitation, I have created a LinkedIn group to foster sharing of technical information about using SUSE Linux on Raspberry Pi and other Single Board Computers (SBC) based on Arm processors. This new group can be found at https://cutt.ly/rpisuse

A brief history of the Raspberry Pi

Raspberry Pi computers are inexpensive, widely available, and have a variety of I/O devices. It is not surprising that over 30 million have been sold since the first Raspberry Pi was introduced in 2012. Raspberry Pi were originally intended to support teaching of basic computer science in schools and in developing countries. They are now being used for a huge variety of workloads including robotics and weather monitoring.

The Raspberry Pi Foundation provides the Raspberry Pi OS (formerly Raspbian). But there are a wide range of other operating systems available for the RPI, including SUSE Linux Enterprise Server (SLES).  

SUSE enabled the Raspberry Pi to promote the new SLES for Arm product at SUSECon in 2016. SLES for Arm only supports 64bit processors and the then new Raspberry Pi 3 was capable of running a 64bit operating system. We decided it would be fun to give everyone at SUSECon their very own Raspberry Pi with a snazzy case.

Why SUSE Linux on a Raspberry Pi?

SUSE is well known as a premium Linux Distribution that focuses on delivering a secure, stable, and supported Linux operating system. Customers who value security, stability, and support depend on SUSE Linux to run their enterprise workloads on a variety of hardware platforms including Arm, Power, X86, and IBM z (mainframe).

Many of these customers recognized that the low cost, portable, Raspberry Pi systems would be ideal for applications such as industrial automation but needed an enterprise operating system that could ensure security, stability, and support. Early customers like the industrial company Knorr-Bremse pushed SUSE to support the Raspberry Pi and built an industrial monitoring solution based on SUSE Linux and the Raspberry Pi.

In additional to industrial monitoring, we have seen a wide variety of uses for SUSE Linux Enterprise Server (SLES) running on Raspberry Pi including warehouse picking systems, environmental monitoring, digital signage, and network security monitoring. The low cost Raspberry Pi hardware platform coupled with a rock solid SUSE Linux operating system have thousands of potential uses for edge applications.

Other Single Board Computers enabled for SUSE Linux

One of the advantages of the Arm ecosystem is the broad range of processors and single board computer platforms available from vendors such as Gateworks, NXP, Marvell, Mellanox/NVIDIA, Rockchip, Socionext, and Xilinx. But the diversity can also be a challenge because many of the Arm ODMs are not as mature in providing device driver enablement and boot enablement in upstream Linux.

Arm is working to improve this situation but there are unfortunately many of the smaller Arm based systems that are challenging to use with a standard Linux distro like SUSE Linux.  You should check with your hardware ODM to see if SUSE Linux will run on their servers.

How to get started

SUSE Linux is a commercial product and requires a subscription to fully install on a Raspberry Pi.

But, you can get free, one year Developer subscription on the SUSE developer subscription page https://www.suse.com/subscriptions/developer/.  You should select the “Arm AArch64” option for subscriptions for the Raspberry Pi.

You will need a 64bit Raspberry Pi such as:

  • Raspberry Pi 3 Model A+
  • Raspberry Pi 3 Model B
  • Raspberry Pi 3 Model B+
  • Raspberry Pi 4 (frame buffer graphics only)
  • Raspberry Pi Compute Module 3+

The Raspberry Pi Quick Start provides a step by step guide to installing SLES for Arm using the Raspberry Pi Just Enough OS(JEOS) image available on the SUSE Download page. You download the SLES15-SP2-JeOS.aarch64-15.2-RaspberryPi-GM.raw.xz image.

The JEOS image is a very minimal image without a graphics desktop. Additional packages can be installed on this base system including lightweight Xdisplay managers like icewm.

You should also read the SLES 15 SP2 Release Notes  that contains important updates for SLES for Arm and other platforms.

Conclusion

We really hope that the LinkedIn group will help to accelerate the use of SUSE Linux on the Raspberry Pi by providing a place where solution developers can share their experiences and get answers to their questions.

The SUSE Linux for Raspberry Pi User Group is a moderated group and will require authorization to join. The intent the group is to share technical information; job posts and other irrelevant posts are be permitted.

Please join me there today and have a lot of fun!

Posted in ARM Processors, Edge Computing, SUSE Linux | Leave a comment

SUSE Enterprise Storage delivers best CephFS benchmark on Arm

Since its introduction in 2006, Ceph has been used predominantly for volume-intensive use cases where performance is not the most critical aspect. Because the technology has many attractive capabilities, there has been a desire to extend the use of Ceph into areas such as High-Performance Computing. Ceph deployments in HPC environments are usually as an active archival (tier-2 storage) behind another parallel file system that is acting as the primary storage tier.

We have seen many significant enhancements in Ceph technology. Contributions to the open source project by SUSE and others has led to new features, considerably improved efficiency, and made it possible for Ceph-based software defined storage solutions to provide a complete end-to-end storage platform for an increasing number of workloads.

Traditionally, HPC data storage has been very specialised to achieve the required performance. With the recent improvements in Ceph software and hardware and in network technology, we now see the feasibility of using Ceph for tier-1 HPC storage.

The IO500 10-node benchmark is an opportunity for SUSE to showcase our production-ready software defined storage solutions. In contrast to some other higher ranked systems, this SUSE Enterprise Storage Ceph-based solution that we benchmarked was production-ready, with all security and data protection mechanisms in place.

This was a team effort with Arm, Ampere, Broadcom, Mellanox, Micron and SUSE to build a cost-effective, highly-performant system available for deployment.

The benchmark

The IO500 benchmark of IO performance combines scores for several bandwidth intensive and metadata intensive workloads to generate a composite score. This score is then used to rank various storage solutions. Note that the current IO500 benchmark does not account for cost or production readiness.

There are two lists for IO500 based upon the number of client nodes. The first list is for unlimited configurations and the second list is for what is termed the “10-node challenge”. ( More IO500 details) We chose the “10-node challenge” for this benchmark attempt.

Our previous benchmark was also for the 10-node challenge and it had a score of 12.43 on a cluster we called “Tigershark”. We created a new cluster this year, called “Hammerhead”, continuing with the shark theme.

The solution

There have been no major updates to the software stack used for the 2019 IO500 benchmark. The operating system was SLES 15 Service Pack 1 with SUSE Enterprise Storage version 6, based on the Ceph version “Nautilus”.

Our goal was to beat our score from last year of 12.43 that was done (11/19 IO500 list) using an entirely different hardware platform and to validate the performance of an Arm-based server. We believe that any system we benchmark should emulate a production system that a customer may deploy, therefore all appropriate Ceph security and data protection features were enabled during the benchmark.

The hardware we used this year is substantially different even though it looks very similar. We used a Lenovo HR350A system, based on the Ampere Computing  eMAG processor. The eMAG is a single socket 32 core ARM v8 architecture processor.

The benchmark was performed on a cluster of ten nodes as the data and metadata servers. Each server has 128GB of RAM and four Micron 7300PRO 3.84TB NVMe SSDs. These NVMe SSDs are designed for workloads that demand high throughput and low latency while staying within a capital and power consumption budget. Broadcom provided the HBA adapters in each storage node.

The cluster was interconnected with Mellanox’s Ethernet Storage Fabric (ENF) networking solution. The Mellanox technology provides a high-performance networking solution to eliminate data communication bottlenecks associated with transferring large amounts of data and is designed for easy scale-out deployments. ENF includes a ConnectX-6 100GbE NIC in each node connected by a SN3700 Spectrum 100GbE switch.

Link to the technical reference architecture

Cluster configuration

SLES Configuration

Each node was set up to disable CPU mitigations.

Network configuration is set up to use JUMBO frames with an MTU of 9000. We also increased the device buffers on the NIC and changed a number of sysctl parameters.

We used the tuned “throughput-performance” profile for all systems.

The last change was to alter the IO scheduler on each of the NVMEs and set it to “none”.

ip link set eth0 mtu 9000 
setpci -s 0000:01:00.0 68.w=5950 
ethtool -G eth0 rx 8192 tx 8192 

sysctl -w net.ipv4.tcp_timestamps=0 
sysctl -w net.ipv4.tcp_sack=0 
sysctl -w net.core.netdev_max_backlog=250000 
sysctl -w net.core.somaxconn=2048 
sysctl -w net.ipv4.tcp_low_latency=1 
sysctl -w net.ipv4.tcp_rmem="10240 87380 2147483647" 
sysctl -w net.ipv4.tcp_wmem="10240 87380 2147483647" 
sysctl -w net.core.rmem_max=2147483647 
sysctl -w net.core.wmem_max=2147483647 

systemctl enable tuned 
systemctl start tuned 
tuned-adm profile throughput-performance 

echo "none" >/sys/block/nvme0n1/queue/scheduler

CEPH Configuration

For the SUSE Enterprise Storage (SES) configuration, we deployed four OSD processes to each NVME device. This meant that each server was running with 16 OSD processes against four physical devices.

We ran a metadata service on every Arm node in the cluster which meant we had 12 active metadata services running.

We increased the number of PGs to 4096 for the metadata and data pools to ensure we had an even distribution of data. This is in line with the recommended number of PGs per OSD of between 50-100.

We also set the data protection of the pools to be 2X as we did in our last set of benchmarks. This is to make sure that any data that is written is protected.

As mentioned previously, we also left authentication turned on for the cluster to emulate what we would see in a production system.

The ceph.conf configuration used:

[global] 
fsid = 4cdde39a-bbec-4ef2-bebb-00182a976f52 
mon_initial_members = amp-mon1, amp-mon2, amp-mon3 
mon_host = 172.16.227.62, 172.16.227.63, 172.16.227.61 
auth_cluster_required = cephx 
auth_service_required = cephx 
auth_client_required = cephx 
public_network = 172.16.227.0/24 
cluster_network = 172.16.220.0/24 
ms_bind_msgr2 = false 

# enable old ceph health format in the json output. This fixes the 
# ceph_exporter. This option will only stay until the prometheus plugin takes 
# over 
mon_health_preluminous_compat = true 
mon health preluminous compat warning = false 

rbd default features = 3 

debug ms=0 
debug mds=0 
debug osd=0 
debug optracker=0 
debug auth=0 
debug asok=0 
debug bluestore=0 
debug bluefs=0 
debug bdev=0 
debug kstore=0 
debug rocksdb=0 
debug eventtrace=0 
debug default=0 
debug rados=0 
debug client=0 
debug perfcounter=0 
debug finisher=0

[osd]
osd_op_num_threads_per_shard = 4 prefer_deferred_size_ssd=0 

[mds]
mds_cache_memory_limit = 21474836480  mds_log_max_segments = 256  mds_bal_split_bite = 4

The Results

This benchmark effort achieved a score of 15.6. This is the best CephFS IO500 benchmark on an Arm-based platform to date. This score was an improvement of 25% over last year’s results on the “Hammerhead” platform. The score put this configuration in the 27th place in the IO500 10-node challenge, two places above our previous benchmark.

During the benchmark, we saw that the Ceph client performance metrics could easily hit peaks in excess of 16GBytes/s for write performance with large file writes. Because we were using production-ready settings for 2X data protection policy, this means that the Ceph nodes were achieving 3GB/s I/O performance.

One of the most significant findings during this testing was the power consumption, or rather the lack of it. We used ipmitool to measure the power consumption on a 30 second average. The worst case 30 second average was only 152 watts, significantly less than we saw with last year’s benchmark.

In addition to the performance improvement and power savings, we believe this cluster would cost up to 40% less than last year’s configuration.

Conclusion

The purpose-built storage solutions used in HPC come at a significant cost. With the data volume increasing exponentially for HPC and AI/ML workloads, there is considerable pressure on IT departments to optimise their expenses. This benchmark demonstrates that storage solutions based on innovative software and hardware technology can provide additional choices for companies struggling to support the amount of data needed in HPC environments.

The partnership with Ampere, Arm, Broadcom, Mellanox, Micron and SUSE yielded a new, highly-performant, power efficient platform – a great storage solution for any business or institution seeking a cost-effective, low power alternative.

Posted in ARM Processors, High Performance Computing, Information Technology, Open Source, Software Defined Storage, SUSE Linux | Leave a comment

What’s new in SUSE Linux for Arm 15 Service Pack 2

15SP2ArmSUSE Linux Enterprise Server (SLES) 15 Service Pack 2 delivers support for new 64bit Arm processors and enhancements to previous Arm support. SUSE uses a “Refresh” and “Consolidation” approach to Service Pack releases: Every “Even” release (e.g., SP0, SP2,…) is a “Refresh” release that will include the latest stable Linux kernel. For SLES 15 SP2, we are using the 5.3 Linux kernel as the base with backports from later kernels as needed.

SUSE’s uses an “upstream first” approach to hardware enablement. That means that SUSE will not use “out of tree” or proprietary Board Support Packages is to enable new hardware, SUSE will only use drivers that has been enabled in upstream Linux. SUSE does work with the community to get new hardware support accepted upstream, but our “upstream first” approach reduces the risk of regression in a later Linux release.

Not all device drivers for new hardware is available upstream at the time SUSE ships a new release. In those cases, SUSE does as much enablement as possible in the current Service Pack, and implements additional drivers in later releases.

New Arm Systems on a chip (SoC) support in SP2*:

  • Ampere Altra
  • AWS Graviton2
  • Broadcom BCM2711 (for RPI 4)
  • Fujitsu A64FX
  • NXP LS1028A (no graphics driver available yet)

*Note: Please check with your specific hardware vendor regarding SUSE support for your specific server. Due to the rapidly evolution of Arm systems, not all Arm based servers have undergone the same degree of hardware testing.

New Arm servers enabled in SP2:

  • Raspberry Pi 4 (no accelerated graphics)
  • Raspberry Pi Compute Module 3B+
  • Raspberry Pi 3A+
  • NVIDIA TegraX1, NVIDIA TegraX2
  • Fujitsu FX700 (SUSE “YES” certified)

Other Arm enhancements

  • Support for up to 480 vCPU
  • Add rpi_cpufreq for Raspberry Pi to dynamically change frequency resulting in lower energy use and heat generation when idle
  • Add Arm V8.2 48-bit IPA for increased memory addressability
  • Enable Armv8.5 Speculation Barrier (SB) instruction to enhance security
  • Enable ARMv8.1-VHE: Virtualization Host Extensions for KVM optimization
  • Remove support for pre-production Marvell ThunderX2 processors
  • USB enabled for ipmitool to simplify flashing firmware on 64bit Arm systems such as HPE Apollo 70
  • Enable ARMv8.4 Unaligned atomic instructions and Single-copy atomicity of loads/stores
  • Improved U-Boot bootloader to support Btrfs filesystem offering additional flexibility for partitioning, scripting and recovery. (Tech-Preview)
  • Improved Installer experience on Raspberry Pi by ensuring all firmware, boot loader, and device tree packages are installed when using DVD media
  • QuickStart for Raspberry Pi updated

Last thoughts

There are a number of other encouraging events in the news about Arm servers:

These announcements underscore that Arm processor-based servers are finally starting to reaching critical mass for data center workloads. SUSE is proud to be part of the Arm server revolution.

Learn more about SLES 15 SP2:

Posted in ARM Processors, Cloud, Information Technology, Internet of Things, SUSE Linux | Tagged , , , , | Leave a comment

What’s new for High Performance Computing in SLE 15 Service Pack 2

15SP2HPC

SUSE Linux Enterprise for High Performance Computing (SLE HPC) 15 Service Pack 2 has a lot of new capabilities for HPC on-premises and in the Cloud.

SUSE uses a “Refresh” and “Consolidation” approach to Service Pack releases: Every “Even” release (e.g., SP0, SP2, …) is a “Refresh” release that will include the latest stable Linux kernel. For SLES 15 SP2, we are using the 5.3 Linux kernel as the base with backports from later kernels as needed. Updating to a new kernel every two releases allows SUSE to provide our customers with the Linux features and enhancements.

SUSE Linux Enterprise for HPC

When we introduced the HPC Module to SUSE Linux early in 2017, we laid out a strategy to make High Performance Computing adoption easier by providing a several fully supported HPC packages to our SUSE Linux customers.

These packages have been built and tested by SUSE and are provided at no additional cost with the SUSE Linux support subscription.

SUSE provides the HPC Module for customers using the X86-64 and Arm hardware platform. Except for a few hardware specific packages, all the packages are supported on both platforms. If you haven’t tried the HPC module yet, here are the  instructions on how to access it. The HPC Module is also included in HPC images available on Clouds like Microsoft Azure.

HPC Module updates

  • Added adios1.13.1
  • boost to 1.71.0
  • conman to version 0.3.0
  • cpuid to version 20180519
  • fftw3 to version 3.3.8
  • genders to version 1.27.3
  • gsl to version 2.6
  • hdf5 to version 1.10.5
  • hwloc to version 2.1.0
  • hypre to version 2.18.2
  • imb to version 2019.3
  • luafilesystem to version 1.7.0
  • lua-lmod to version 8.2.5
  • lua-luaposix to version 34.1.1
  • memkind to version 1.9.0
  • mpich to 3.3.2
  • mvapich2 to version 2.3.3
  • mumps to version 5.2.1
  • netcdf to version 4.7.3
  • netcdf-cxx4 to version 4.3.1
  • netcdf-fortran to 4.5.2
  • python-numpy to version 1.17.3
  • python-scipy to version 1.3.3
  • openblas to 0.3.7
  • Added openmpi3 3.1.4
  • papi to version 5.7.0
  • petsc to version 3.12.2
  • Added PMIx version 3.1.5
  • scalapack to 2.1
  • scotch to version 6.0.9
  • slurm to 20.0.2
  • trilinos to version 12.14.1

HPC relevant base system packages

  • libfabric 1.8 to enable AWS
  • rdma-core 24.0 to enable AWS

PackageHub

PackageHub is a SUSE curated repository for community supported, open-source packages. A number of AI/ML packages were added to PackageHub for SP2.  These packages can be installed using zypper. Tips for using PackageHub.

  • armnn 20.02
  • caffe  1.0
  • charliecloud 0.15
  • clustershell 1.8.2
  • numpy 1.17.3
  • pytorch
  • robinhood 3.1.5
  • singularity 3.5.2
  • tensorflow2 2.1.0
  • theano 1.0.4
  • warewolf 3.8.1

New HPC and ML/AI Systems

SLE HPC 15 SP2 enabled a number of servers for HPC and AI/ML workloads including:

  • NVIDIA TegraX1
  • NVIDIA TegraX2
  • Fujitsu FX700 – (also SUSE “YES” certified)
  • AWS Graviton2 instances in AWS Cloud

Other HPC changes

  • Add Elastic Fabric Adapter (EFA) driver to enable HPC on AWS
  • Add the Live Patching Extension to the SLE-HPC Product
  • Improve clustduct cluster distribution tool
  • Remove unneeded package ‘ohpc’

Summary

The HPC ecosystem continues to expand and transform to include simulation, data analytics, and machine learning. SUSE HPC will continue to grow with the needs of our customers.

Posted in ARM Processors, Cloud, High Performance Computing, Information Technology, SUSE Linux, Uncategorized | Tagged , , , | Leave a comment

Understanding SUSE Sub-capacity pricing for IBM Power servers

SUSE_value_money_scale

 

Executive Summary

SUSE recently updated Terms and Conditions for SUSE products to clarify the SUSE pricing policies for IBM Power systems and to accommodate Sub-capacity pricing on IBM Power servers.

Sub-capacity pricing overview

IBM Power servers have long been known for vertical scalability (up to 192 cores and 64TB of RAM) and for efficient PowerVM virtualization infrastructure. As a result, many customers use IBM Power servers to consolidate multiple workloads on a single server.

PowerVM virtualization can guarantee that a particular workload can only run on a subset of the processors on a particular system.  For example, you could have a 192 core IBM Power server with 32 cores running a DB2 database with AIX and the other 160 cores could be running SAP HANA on SUSE Linux for SAP Applications.

Software subscriptions traditionally have been priced based on the amount of CPU capacity available to run the software, fewer cores generally mean lower software cost. This is known as Sub-capacity pricing because the customer is only paying for software based on a subset of the total hardware capacity.

Most IBM software is sold based on Sub-capacity pricing, so IBM Power customers expect to pay for software licenses or subscriptions based on the amount of processor capacity available to each workload.

Sub-capacity pricing for SUSE products on IBM Power

SUSE products started on X86 servers, where prices were traditionally based on the number of processor sockets as a measure of CPU capacity. Server consolidation is less prevalent in the X86 market and Sub-capacity pricing for software is less common.  As a result, the price of most SUSE subscriptions is based on the number of processor sockets available in the server, known as Full-capacity pricing.

In view of the unique capabilities of the IBM Power platform, we have changed the Terms and Conditions for SUSE product subscriptions to accommodate Sub-capacity pricing on Power. This will allow customers to partition their IBM Power servers with PowerVM virtualization and only pay for the capacity available to that SUSE product on a particular server.

The charge metric of SUSE product subscriptions on IBM Power has not been changed and is still measured in “socket pairs”. Because IBM PowerVM allocates processor capacity in terms of processor cores, there needs to be a way to convert the “cores” allocation in PowerVM to the equivalent number of socket pairs for SUSE subscription purposes.

Because SUSE subscriptions are sold in increments of socket pairs, to use Sub-capacity pricing on Power while still using socket pairs, you need to calculate the Socket Pair Equivalent that represents the CPU capacity that is made available to the SUSE workload.

Unfortunately, there is another complication in that IBM Power servers have different numbers of cores per socket, and thus per socket pair. For example, an IBM E980 server with hardware feature #EFP1 has 8 cores per socket, or 16 cores per socket pair. The same hardware ordered with feature #EFP3 has 12 cores per socket or 24 cores per socket pair.  You must know the number of cores per socket in the server before you can calculate the Socket Pair Equivalent for SUSE subscription purposes. The number of cores per socket on an IBM Power server is tied to the processor speed. Faster speed = Fewer cores per socket.

subcap0

Conditions for using Sub-capacity pricing for SUSE products on IBM Power

  • The SUSE product must use socket pairs as the charge metric.
    • SUSE Linux Enterprise Server (SLES) for Power, SLES for SAP Applications on Power, SUSE Manager Server for Power and SUSE Manager Lifecycle for Power are all included.
  • This is only applicable to POWER8 and POWER9 (and later generations) with four or more sockets.
    • E850, E880, E870, E950, and E980 are all eligible for Sub-capacity pricing.
  • PowerVM virtualization must be used to restrict the amount of capacity available to the SUSE products being used
    • Acceptable methodologies: Dedicated processor partitions (Dedicated LPAR), Dynamic LPAR, Single or Multiple Shared Processor Pools
    • Refer to the IBM standard on Sub-capacity pricing: https://www.ibm.com/software/passportadvantage/subcaplicensing.html
    • The Integrated Facility for Linux (IFL) processors does not automatically limit Linux workloads to only run on IFL processors and are not relevant to Sub-capacity pricing.
  • Customers and Sellers must account for the different number of cores per socket when calculating the Socket Pair Equivalent
    • Number of cores available to SUSE Linux ÷ by the number of cores per socket = Socket Pair Equivalent
    • Round up to nearest socket pair
  • The customer must purchase subscriptions for the maximum number of socket pairs that are ever available to run SUSE workloads.
    • Customer is responsible for purchasing additional subscriptions if the processor capacity is increased through changes to pools or activation of dark processors or other changes to the server capacity.

Example 1: Calculating Socket Pair Equivalent for IBM Power E950

In this example, the E950 has 10 cores per socket = 20 cores per socket pair.

subcap1

Example 2: Calculating Socket Pair Equivalent for IBM Power E950 with Rounding

In this example, the E950 has 10 cores per socket = 20 cores per socket pair.

subcap2

You can see that if the calculations end up with a fractional value, (that is, the number of cores is not an integer number of sockets) you must round up to the next nearest integer.

Example 3: Calculating Socket Pair Equivalent for IBM Power E950 lower core count and rounding

In this example, the E950 has 8 cores per socket = 16 cores per socket pair.

subcap3

As in the previous example, you can see that if the calculations end up with a fractional value, (that is, the number of cores is not an integer number of sockets) you must round up to the next nearest integer.

Example 4: Calculating Socket Pair Equivalent for IBM Power E950 with Dynamic LPAR

In this example, the E950 has 10 cores per socket = 20 cores per socket pair and the system administrator is using Dynamic LPAR (DLPAR) to dynamically change the number of cores available to SLES for SAP.

subcap4

Before the DLPAR operation, the SLES for SAP LPAR/VM had 18 cores of capacity which would round up to 1 socket pair of capacity but after the DLPAR operation, the SLES for SAP LPAR/VM is over 2 sockets of capacity and thus requires 2 SLES for SAP subscriptions.

The customer must provide enough SUSE subscriptions to cover the maximum capacity ever used.

Example 5: Calculating Socket Pair Equivalent for IBM Power E980 Server Consolidation

In this example, the E980 has 10 cores per socket = 20 cores per socket pair.

subcapp52

Once again, we see that if the number of cores assigned to the SUSE product is more than 2 socket pairs, so 3 SLES for Power subscriptions are needed for this configuration.

Example 6: Calculating Socket Pair Equivalent for IBM Power E980 with multiple SUSE products

In this example, the E980 has 12 cores per socket = 24 cores per socket pair.

subcapp62

There are two different SUSE products installed on this system. The Socket Pair Equivalent calculation must be performed for each SUSE product installed on a physical server.

Summary

Server consolidation is a common practice for IBM Power servers and Sub-capacity pricing is important to supporting consolidation. Using the Socket Pair Equivalent approach allows customers to leverage Sub-capacity pricing for SUSE products on IBM Power servers and to leverage the capabilities of PowerVM while maintaining compliance with SUSE terms and conditions.

Reference: Cores / Socket for IBM POWER9 Enterprise servers

POWER9_Sub-Capacity_Reference

Reference: Cores / Socket for IBM POWER8 Enterprise servers

subcapp8

Posted in AIX & Power Systems Blogroll, SAP HANA, SUSE Linux, Uncategorized | Tagged , , , , , , | Leave a comment

Using IBM POWER9 PowerVM Virtual Persistent Memory for SAP HANA with SUSE Linux

PowerVM_Persistent_memory_concept2

Introduction

SAP HANA uses in-memory database technology that allows much faster access to data than was ever possible with hard disk technology on a conventional database – access times of 5 nanoseconds versus 5 milliseconds. SAP HANA customers can also use the same database for real-time analysis and decision-making that is used for transaction processing.

The combination of faster access speeds and better access for analytics has resulted in strong customer demand for SAP HANA. There are already more than 1600 customers using SAP HANA on Power since it became available in 2015.

One of the drawbacks of in-memory databases is the amount of time required to load the data from disk into memory after a restart. In one test with an 8TB data file representing 16TB database, it took only six minutes to shut down SAP HANA, but took over 40 minutes to completely load that database back into memory. Although the system was up and responsive much more quickly, it took a long time for all data to be staged back into memory.

SAP has implemented features such as the Fast Restart Option to reduce the database load time, but system providers have enhancements to help address the problem. Intel introduced Optane DC memory technology where the data in memory is preserved even when powered down. Optane DC is positioned as a new tier of storage between DRAM and flash storage with more capacity at a lower price than DRAM but with less performance. Optane DC memory can deliver very fast restarts of SAP HANA but introduces new operational management complexity associated with adding a fourth tier of storage.

IBM PowerVM Virtual Persistent Memory

In October 2019, IBM announced Virtual Persistent Memory (vPMEM) with PowerVM. Virtual Persistent Memory isn’t a new type of physical memory but is a way to use the PowerVM hypervisor to create persistent memory volumes out of the DRAM that is already installed on the system. vPMEM is included at no additional charge with PowerVM.

The data in memory is persistent as long as the physical Power server is not powered off. By maintaining data persistence across application and partition restarts, it allows customers to leverage fast restart of a workload using persistent memory. An added benefit is that there is no difference in application performance when using vPMEM because the underlying DRAM technology is the same as for the non-persistent memory.

Although vPMEM does not preserve memory across a server power down, most IBM Power customers seldom power off their systems because of the many reliability features that are built into Power hardware. vPMEM provides for fast restart in the vast majority of planned maintenance and unplanned outages without compromising the performance of HANA during normal use.

sap_hana_memory_access­

Prerequisites

vPMEM has several prerequisites:

  • POWER9 processor-based systems
  • Hardware Management Console (HMC) V9R1 M940 or later
  • Firmware level FW940 or later
    • E980 system firmware FW940
    • L922 system firmware FW940
  • PowerVM level V3.1.1
  • SUSE Linux 15 for SAP Applications 15 Service Pack 1 with latest maintenance updates to bring the kernel up to 4.12.14-197.26-default or later
  • SAP HANA 2.0 SPS 03 Revision 35 (2.00.035)
  • SAP HANA 2.0 SPS 04 (adds new features for memory management)
  • SAP Note 2618154 SAP HANA Persistent Memory – Release Information
  • SAP Note 2700084 FAQ: SAP HANA Persistent Memory
  • SAP HANA Administration Guide – Administration Guide: Persistent Memory

IBM has provided the following documents:

 

SUSE has three technical support bulletins that you should review before implementing vPMEM

  • TID 7024333  “Activation of multiple namespaces simultaneously may lead to an activation failure”
  • TID 7024330 “vPMEM memory backed namespaces configured as a dump target for kdump/fadump takes a long time to save dump files”
  • TID 7024300 “Hot plugging/unplugging of pmem memory having a size that is not in multiples of a specific size can lead to kernel panics”

Summery

Virtual Persistent Memory is the latest tool in a long line of IBM and SUSE innovations to help customers get the most out of their SAP HANA on Power environments.

You should strongly consider using vPMEM when running SAP HANA on POWER9 systems.

Posted in AIX & Power Systems Blogroll, Information Technology, SAP HANA, SUSE Linux | Tagged , , , , , , | 1 Comment

SUSE Linux Essentials – Where are the compilers? Understanding the Development Tools Module

Binoculars_SUSE_Gecko_view_explore

Compilers on SUSE Linux

If you are new to SUSE Linux, you might have wondered why the C compiler on the system is so old. For example, on a SUSE Linux Enterprise Server (SLES) 12 Service Pack (SP) 3 system running on X86-64, the default gcc is version 4.8-6.189! Why isn’t the C compiler a newer version, like gcc version 8?

A SUSE Linux system can have multiple versions of the gcc compiler. The first type of compiler, the one used to compile the version of SUSE Linux that you are running, is known as the “System Compiler”. The “System Compiler” usually does not change throughout the life of the SLES version because changing it would greatly complicate creating patches to maintain the operating system. For example, gcc 4.8 is the SLES 12 “System Compiler” used to compile all SLES 12 releases including all Service Packs.  gcc 7-3.3.22 is the “System Compiler” for SLES 15.

The other type of compilers available on SLES are known as a “Toolchain Compilers”. These are the primary compilers for application development and are periodically updated with patches and new stable compiler versions as they become available. Usually there is more than one version available at any point in time. Most developers want to use newer compilers for application development because they provide additional optimization and functionality.

Installing the newer “Toolchain” compilers is easy, but first you must understand a little about the SUSE Linux Modules concept.

SUSE Linux Modules Introduction

SUSE introduced the concept of operating system Modules in SLES 12. SLES Modules are groups of packages with similar use cases and support status that are grouped together into a Module and delivered via a unique repository. SUSE introduced the Module concept to allow delivery of the latest technology in areas of rapid innovation without having to wait for the next SLES Service Pack. SUSE fully maintains and supports the Modules through your SUSE Linux subscription.

With the introduction of SLES 15, Modules have become even more important because the entire SUSE Linux OS is packaged as modules. The SLES 15 installation media consists of a unified Installer and a minimal system for deploying, updating, and registering SUSE Linux Enterprise Server or other SUSE products. Even the ls and ps commands are part of “Basesystem Module 15” module in SLES 15.

During deployment you can add functionality by selecting modules and extensions to be installed. The availability of certain modules or extensions depends on the product you chose in the first step of this installation. For example, in SLES 15, the HPC Module is only available if you selected the SUSE Linux Enterprise for HPC product during the initial installation.

INstall_select_Modules_15_Inverted

SUSE Linux Modules enable you to install just the set of packages required for the machine’s purpose, making the system lean, fast, and more secure. This modular packaging approach also makes it easy to provide tailor-made images for container and cloud environments. Modules can be added or removed at any time during the lifecycle of the system, allowing you to easily adjust the system to changing requirements. See the Modules Quick Start Guide for more information.

Please note that just activating a Module does not install the packages from that Module—you need to do that separately after activating a Module.  To use a package from a Module generally requires two steps: 1) activating the Module and 2) installing the desired packages.  Some Modules are pre-activated based on the product you installed. For example, the HPC Module is automatically activated when you install SUSE Linux for High Performance Computing 15.

The Development Tools Module is the SLES 15 Module that includes the most recent compilers such as gcc, debuggers, and other development tools (over 500 packages for SLES 15). The Toolchain Module provided access to newer compilers for SLES 12.

Activating the Development Tools Module using YaST

  1. Start YaST.                                                    # Requires root privileges
  2. Click Add System Extensions or Modules.

Install_Software_Management

  1. Select Development Tools Module.

Install_select_Development_tools

  1. Press Next to start the activation process for the Development Tools Module

Installing a package (gcc8) from the Development Tools Module

Once the Development Tools Module is activated, you can install packages from the Module. This example uses gcc8.

  1. Start YaST and click Software then Software management

Yast_Software Management

  1. Type gcc8 in the search field and click Search.

YaST_Search_gcc8

  1. Select gcc8 and click Accept.

The gcc8 compiler will now be installed.

YaST_Gcc_8_installed

And that’s it! You have successfully activated the Development Tools Module and installed gcc8.

Activating the Development Tools Module on SLES 15 using the command line

Now let’s take a different approach and use the command line to perform the same tasks. You will need to have root privileges to perform most of these steps either by su to root or using sudo.

  1. Activate the Development Tools Module.
sudo SUSEConnect -p sle-module-development-tools/15/x86_64

CMD_Out_SUSEConnect_activate_Devtools

  1. Verify that the Development Tools Module is activated.
sudo SUSEConnect –l

CMD_Out_suseconnect-l

In this example, the BaseSystem, Development Tools, Desktop Applications, and Web and Scripting Modules are activated. The other Modules and Extensions are available for activation.

  1. Check the status of gcc on this system.
zypper search gcc

CMD_Out_zypper_search_gcc.png

You can see that the system C compiler is gcc7 and is installed (i)  and gcc8 is available for installation.

  1. Install the gcc8 compiler.
sudo zypper install gcc8      # Must be root

CMD_Out_zypper+install_gcc8

  1. Verify that the gcc8 compiler was installed

zypper search gcc8

CMD_Out_zypper_search_gcc8

  1. Check the gcc8 version.
gcc-8 –version

CMD_Out_gcc8--version

Summary

You should now understand how to activate a SUSE Module and install a package from the module. This same approach can be applied to the other SUSE Modules, such as the HPC Module. Modules are a key concept of SUSE product packaging, particularly for SLES 15.

Posted in High Performance Computing, SUSE Linux, Uncategorized | Tagged , , , , , , | Leave a comment

The ibmvnic driver with SR-IOV is now supported by SLES 12 and SLES 15 on IBM POWER9 processor-based systems

green_networkSUSE has worked with IBM to enable support for the ibmvnic driver with PowerVM on POWER9 systems.

The ibmvnic driver enables PowerVM Single Root I/O Virtualizations (SR-IOV) for improved network capabilities including reduced systems processor utilization, decreased network latency, and enforcement of network Quality of Service.

Supported SLES Levels:

  • SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)
  • SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4)
  • SUSE Linux Enterprise Server 15

Supported Hardware:

  • All IBM POWER9 processor-based systems using PowerVM virtualization

IBM and SUSE strongly recommend customers apply the latest maintenance updates to SLES, PowerVM, and the latest firmware level for the POWER9 hardware before using ibmvnic.

Customers should not attempt installation of SLES 12 SP3 over an ibmvnic network.

Please NOTE: ibmvnic is not supported on POWER8 systems. Ibmvnic is a technology preview on POWER8 based systems.

Full details are in the SUSE TID at https://www.suse.com/support/kb/doc/?id=7023703

Posted in AIX & Power Systems Blogroll, Information Technology, SAP HANA, SUSE Linux | Leave a comment

SUSE Linux for Arm is now available for all customers

green register 2

SUSE Arm products available to all

Subscriptions for SUSE Linux Enterprise Server for Arm and SUSE Manager Lifecycle for Arm are now available directly to customers through the Corporate price list or through the SUSE Shop https://www.suse.com/shop/

Previously, SUSE subscriptions for the Arm hardware platforms were only available to SUSE Partners due to the relative immaturity of the Arm server platform. Now that we have delivered four releases of SUSE Linux Enterprise Server for Arm and have customers running SUSE Linux on Arm servers as diverse as the tiny Raspberry Pi and the high performance HPE Apollo 70 servers, we are now ready to sell subscriptions directly to customers.

Diversity of Arm: An Advantage and Challenge

As you can see from the chart below, SUSE has enabled a number of different Arm System-On-a-Chip (SOC) processors. One of the major advantages of the Arm platform compared to other platforms is the huge diversity of processors and processor features.

Each Arm licensee has great freedom to create their own System-On-a-Chip (SoC) engineered with the I/O, memory bandwidth, and number of processor cores needed for a specific workload. The number of processor cores varies from 1 processor to 48 processors per core. A single Arm SoC can have multiple varieties, such as the Marvel OcteonTX that is available in with 1 to 24 cores.

sles15_arm_enablement

Beyond socket-based pricing

The ability to create Arm SoCs with just the right I/O and number of processor cores needed for a particular workload is one of the largest advantages of the Arm platform, but it is also a pricing challenge for software vendors such as SUSE.

Software pricing should:

  • Reflect the value the customer receives from the software
  • Be as simple and easy to understand as possible

Both goals are essential, but sometimes difficult to achieve. For example, the value of a high availability solution that reduces system outages might be much more valuable to an airline than it would be to a small retail shop due to the difference in the number of people and revenue affected by an outage.

Software vendors have traditionally priced software based on the amount of processing capability available to the application (or Operating System). The underlying concept is that the more capacity, the higher the value, and therefore the higher the price.

For example, on a UNIX hardware platform such as IBM Power, IBM software cost is based on the number of processor cores and the relative performance of each core-something IBM calls the Processor Value Unit (PVU). Oracle has a similar approach called the Core Factor.  The underlying concept is simple: the bigger the system, the greater the value to the customer and thus, the more expensive the software. Unfortunately, both of these approaches are relatively complex.

By contrast, software pricing in the commodity Linux server market has traditionally used a much simpler measure of capacity: processor sockets. The more sockets in a server, the higher the software costs. Most SUSE software subscriptions are based on increments of 1-2 sockets of processing capacity. For example, subscriptions for SUSE Linux Enterprise Server on X86 servers have a U.S. list price of $799 per socket pair. If you have a four-socket server, then you would need two 1-2 socket subscriptions for a total list price of $1,598.

The traditional socket-based pricing goes back many years, when each socket might have only a single processor core. Although this pricing model is very simple, it doesn’t fit well to the diversity of systems based on Arm SoCs where a single socket can hold a single, low performance Arm core or 48 (or more) high performance cores.

Hybrid pricing for SUSE products on Arm

SUSE needed a pricing approach for SUSE Linux and other products for the Arm platform that could be used for very small systems such as the Raspberry Pi and for huge systems such as the HPE Apollo 70. The pricing approach also needed to be simple to understand. Ultimately, we decided to use a model that has core-based pricing for lower end Arm hardware and socket-based pricing for everything else.

The new pricing for SUSE Linux Enterprise for Arm is tied to the number of processor cores in a server. Servers with less than 16 cores are priced based on the number of groups of 4 processor cores. Each group of 4 cores, up to 15 cores, requires a 4-core group subscription that is stackable to a maximum 4 subscriptions. The number of cores is rounded up to the nearest group of 4 cores, therefore a server with 10 cores would require three 4-core group subscriptions.

Servers with 16 or more cores use the traditional 1-2 socket-based pricing.

This hybrid pricing model applies to SUSE products for Arm that have traditionally used socket-based pricing, such as the SUSE Linux Enterprise Server (SLES) for Arm and SUSE Manager Lifecycle. Other SUSE products such as SUSE Enterprise Storage uses server-based pricing and do not use this hybrid pricing.

Hybrid pricing examples for SUSE products on Arm

Raspberry Pi 3 Model B

The Raspberry Pi 3 Model B has 4 Broadcom processor cores. Because the total number of cores in the server is less than 16 cores, you use the 4-core group-based pricing.

arm pricing examples-rpi

NXP LS2085A

Systems based on the NXP LS2085A have 8 processor cores. Because the total number of cores in the server is less than 16 cores, you use the 4-core group-based pricing.

arm pricing examples-ls2085

Marvell OcteonTX

Systems based on the Marvell Octeon TX can have from 1 to 24 processor cores. In this example, we use a 14 core Octeon TX processor. Because the total number of cores in this server is less than 16 cores, you use the 4-core group-based pricing. This example also demonstrates how the number of cores is rounded up to the nearest integer multiple of 4.

arm pricing examples-octeontx

HPE Apollo 70

The HPE Apollo 70 is based on the Marvell ThunderX2 which can have from 16-32 processors per socket. In this example we use an HPE Apollo 70 dual socket server with 32 processor cores per socket for a total of 64 cores in this system. Because the server is greater than 15 cores, you use the 1-2 socket pricing.

arm pricing examples-apollo70

Summary

The diversity of the Arm server platform requires a more sophisticated pricing model than the simplistic socket-based approach. The new hybrid pricing model strikes a balance by using a core-based approach for smaller systems while retaining the traditional socket-based approach for larger servers.

Posted in ARM Processors, High Performance Computing, Internet of Things, SUSE Linux | Tagged , , , , | Leave a comment