SUSE Enterprise Storage delivers best CephFS benchmark on Arm

Since its introduction in 2006, Ceph has been used predominantly for volume-intensive use cases where performance is not the most critical aspect. Because the technology has many attractive capabilities, there has been a desire to extend the use of Ceph into areas such as High-Performance Computing. Ceph deployments in HPC environments are usually as an active archival (tier-2 storage) behind another parallel file system that is acting as the primary storage tier.

We have seen many significant enhancements in Ceph technology. Contributions to the open source project by SUSE and others has led to new features, considerably improved efficiency, and made it possible for Ceph-based software defined storage solutions to provide a complete end-to-end storage platform for an increasing number of workloads.

Traditionally, HPC data storage has been very specialised to achieve the required performance. With the recent improvements in Ceph software and hardware and in network technology, we now see the feasibility of using Ceph for tier-1 HPC storage.

The IO500 10-node benchmark is an opportunity for SUSE to showcase our production-ready software defined storage solutions. In contrast to some other higher ranked systems, this SUSE Enterprise Storage Ceph-based solution that we benchmarked was production-ready, with all security and data protection mechanisms in place.

This was a team effort with Arm, Ampere, Broadcom, Mellanox, Micron and SUSE to build a cost-effective, highly-performant system available for deployment.

The benchmark

The IO500 benchmark of IO performance combines scores for several bandwidth intensive and metadata intensive workloads to generate a composite score. This score is then used to rank various storage solutions. Note that the current IO500 benchmark does not account for cost or production readiness.

There are two lists for IO500 based upon the number of client nodes. The first list is for unlimited configurations and the second list is for what is termed the “10-node challenge”. ( More IO500 details) We chose the “10-node challenge” for this benchmark attempt.

Our previous benchmark was also for the 10-node challenge and it had a score of 12.43 on a cluster we called “Tigershark”. We created a new cluster this year, called “Hammerhead”, continuing with the shark theme.

The solution

There have been no major updates to the software stack used for the 2019 IO500 benchmark. The operating system was SLES 15 Service Pack 1 with SUSE Enterprise Storage version 6, based on the Ceph version “Nautilus”.

Our goal was to beat our score from last year of 12.43 that was done (11/19 IO500 list) using an entirely different hardware platform and to validate the performance of an Arm-based server. We believe that any system we benchmark should emulate a production system that a customer may deploy, therefore all appropriate Ceph security and data protection features were enabled during the benchmark.

The hardware we used this year is substantially different even though it looks very similar. We used a Lenovo HR350A system, based on the Ampere Computing  eMAG processor. The eMAG is a single socket 32 core ARM v8 architecture processor.

The benchmark was performed on a cluster of ten nodes as the data and metadata servers. Each server has 128GB of RAM and four Micron 7300PRO 3.84TB NVMe SSDs. These NVMe SSDs are designed for workloads that demand high throughput and low latency while staying within a capital and power consumption budget. Broadcom provided the HBA adapters in each storage node.

The cluster was interconnected with Mellanox’s Ethernet Storage Fabric (ENF) networking solution. The Mellanox technology provides a high-performance networking solution to eliminate data communication bottlenecks associated with transferring large amounts of data and is designed for easy scale-out deployments. ENF includes a ConnectX-6 100GbE NIC in each node connected by a SN3700 Spectrum 100GbE switch.

Link to the technical reference architecture

Cluster configuration

SLES Configuration

Each node was set up to disable CPU mitigations.

Network configuration is set up to use JUMBO frames with an MTU of 9000. We also increased the device buffers on the NIC and changed a number of sysctl parameters.

We used the tuned “throughput-performance” profile for all systems.

The last change was to alter the IO scheduler on each of the NVMEs and set it to “none”.

ip link set eth0 mtu 9000 
setpci -s 0000:01:00.0 68.w=5950 
ethtool -G eth0 rx 8192 tx 8192 

sysctl -w net.ipv4.tcp_timestamps=0 
sysctl -w net.ipv4.tcp_sack=0 
sysctl -w net.core.netdev_max_backlog=250000 
sysctl -w net.core.somaxconn=2048 
sysctl -w net.ipv4.tcp_low_latency=1 
sysctl -w net.ipv4.tcp_rmem="10240 87380 2147483647" 
sysctl -w net.ipv4.tcp_wmem="10240 87380 2147483647" 
sysctl -w net.core.rmem_max=2147483647 
sysctl -w net.core.wmem_max=2147483647 

systemctl enable tuned 
systemctl start tuned 
tuned-adm profile throughput-performance 

echo "none" >/sys/block/nvme0n1/queue/scheduler

CEPH Configuration

For the SUSE Enterprise Storage (SES) configuration, we deployed four OSD processes to each NVME device. This meant that each server was running with 16 OSD processes against four physical devices.

We ran a metadata service on every Arm node in the cluster which meant we had 12 active metadata services running.

We increased the number of PGs to 4096 for the metadata and data pools to ensure we had an even distribution of data. This is in line with the recommended number of PGs per OSD of between 50-100.

We also set the data protection of the pools to be 2X as we did in our last set of benchmarks. This is to make sure that any data that is written is protected.

As mentioned previously, we also left authentication turned on for the cluster to emulate what we would see in a production system.

The ceph.conf configuration used:

[global] 
fsid = 4cdde39a-bbec-4ef2-bebb-00182a976f52 
mon_initial_members = amp-mon1, amp-mon2, amp-mon3 
mon_host = 172.16.227.62, 172.16.227.63, 172.16.227.61 
auth_cluster_required = cephx 
auth_service_required = cephx 
auth_client_required = cephx 
public_network = 172.16.227.0/24 
cluster_network = 172.16.220.0/24 
ms_bind_msgr2 = false 

# enable old ceph health format in the json output. This fixes the 
# ceph_exporter. This option will only stay until the prometheus plugin takes 
# over 
mon_health_preluminous_compat = true 
mon health preluminous compat warning = false 

rbd default features = 3 

debug ms=0 
debug mds=0 
debug osd=0 
debug optracker=0 
debug auth=0 
debug asok=0 
debug bluestore=0 
debug bluefs=0 
debug bdev=0 
debug kstore=0 
debug rocksdb=0 
debug eventtrace=0 
debug default=0 
debug rados=0 
debug client=0 
debug perfcounter=0 
debug finisher=0

[osd]
osd_op_num_threads_per_shard = 4 prefer_deferred_size_ssd=0 

[mds]
mds_cache_memory_limit = 21474836480  mds_log_max_segments = 256  mds_bal_split_bite = 4

The Results

This benchmark effort achieved a score of 15.6. This is the best CephFS IO500 benchmark on an Arm-based platform to date. This score was an improvement of 25% over last year’s results on the “Hammerhead” platform. The score put this configuration in the 27th place in the IO500 10-node challenge, two places above our previous benchmark.

During the benchmark, we saw that the Ceph client performance metrics could easily hit peaks in excess of 16GBytes/s for write performance with large file writes. Because we were using production-ready settings for 2X data protection policy, this means that the Ceph nodes were achieving 3GB/s I/O performance.

One of the most significant findings during this testing was the power consumption, or rather the lack of it. We used ipmitool to measure the power consumption on a 30 second average. The worst case 30 second average was only 152 watts, significantly less than we saw with last year’s benchmark.

In addition to the performance improvement and power savings, we believe this cluster would cost up to 40% less than last year’s configuration.

Conclusion

The purpose-built storage solutions used in HPC come at a significant cost. With the data volume increasing exponentially for HPC and AI/ML workloads, there is considerable pressure on IT departments to optimise their expenses. This benchmark demonstrates that storage solutions based on innovative software and hardware technology can provide additional choices for companies struggling to support the amount of data needed in HPC environments.

The partnership with Ampere, Arm, Broadcom, Mellanox, Micron and SUSE yielded a new, highly-performant, power efficient platform – a great storage solution for any business or institution seeking a cost-effective, low power alternative.

Posted in ARM Processors, High Performance Computing, Information Technology, Open Source, Software Defined Storage, SUSE Linux | Leave a comment

What’s new in SUSE Linux for Arm 15 Service Pack 2

15SP2ArmSUSE Linux Enterprise Server (SLES) 15 Service Pack 2 delivers support for new 64bit Arm processors and enhancements to previous Arm support. SUSE uses a “Refresh” and “Consolidation” approach to Service Pack releases: Every “Even” release (e.g., SP0, SP2,…) is a “Refresh” release that will include the latest stable Linux kernel. For SLES 15 SP2, we are using the 5.3 Linux kernel as the base with backports from later kernels as needed.

SUSE’s uses an “upstream first” approach to hardware enablement. That means that SUSE will not use “out of tree” or proprietary Board Support Packages is to enable new hardware, SUSE will only use drivers that has been enabled in upstream Linux. SUSE does work with the community to get new hardware support accepted upstream, but our “upstream first” approach reduces the risk of regression in a later Linux release.

Not all device drivers for new hardware is available upstream at the time SUSE ships a new release. In those cases, SUSE does as much enablement as possible in the current Service Pack, and implements additional drivers in later releases.

New Arm Systems on a chip (SoC) support in SP2*:

  • Ampere Altra
  • AWS Graviton2
  • Broadcom BCM2711 (for RPI 4)
  • Fujitsu A64FX
  • NXP LS1028A (no graphics driver available yet)

*Note: Please check with your specific hardware vendor regarding SUSE support for your specific server. Due to the rapidly evolution of Arm systems, not all Arm based servers have undergone the same degree of hardware testing.

New Arm servers enabled in SP2:

  • Raspberry Pi 4 (no accelerated graphics)
  • Raspberry Pi Compute Module 3B+
  • Raspberry Pi 3A+
  • NVIDIA TegraX1, NVIDIA TegraX2
  • Fujitsu FX700 (SUSE “YES” certified)

Other Arm enhancements

  • Support for up to 480 vCPU
  • Add rpi_cpufreq for Raspberry Pi to dynamically change frequency resulting in lower energy use and heat generation when idle
  • Add Arm V8.2 48-bit IPA for increased memory addressability
  • Enable Armv8.5 Speculation Barrier (SB) instruction to enhance security
  • Enable ARMv8.1-VHE: Virtualization Host Extensions for KVM optimization
  • Remove support for pre-production Marvell ThunderX2 processors
  • USB enabled for ipmitool to simplify flashing firmware on 64bit Arm systems such as HPE Apollo 70
  • Enable ARMv8.4 Unaligned atomic instructions and Single-copy atomicity of loads/stores
  • Improved U-Boot bootloader to support Btrfs filesystem offering additional flexibility for partitioning, scripting and recovery. (Tech-Preview)
  • Improved Installer experience on Raspberry Pi by ensuring all firmware, boot loader, and device tree packages are installed when using DVD media
  • QuickStart for Raspberry Pi updated

Last thoughts

There are a number of other encouraging events in the news about Arm servers:

These announcements underscore that Arm processor-based servers are finally starting to reaching critical mass for data center workloads. SUSE is proud to be part of the Arm server revolution.

Learn more about SLES 15 SP2:

Posted in ARM Processors, Cloud, Information Technology, Internet of Things, SUSE Linux | Tagged , , , , | Leave a comment

What’s new for High Performance Computing in SLE 15 Service Pack 2

15SP2HPC

SUSE Linux Enterprise for High Performance Computing (SLE HPC) 15 Service Pack 2 has a lot of new capabilities for HPC on-premises and in the Cloud.

SUSE uses a “Refresh” and “Consolidation” approach to Service Pack releases: Every “Even” release (e.g., SP0, SP2, …) is a “Refresh” release that will include the latest stable Linux kernel. For SLES 15 SP2, we are using the 5.3 Linux kernel as the base with backports from later kernels as needed. Updating to a new kernel every two releases allows SUSE to provide our customers with the Linux features and enhancements.

SUSE Linux Enterprise for HPC

When we introduced the HPC Module to SUSE Linux early in 2017, we laid out a strategy to make High Performance Computing adoption easier by providing a several fully supported HPC packages to our SUSE Linux customers.

These packages have been built and tested by SUSE and are provided at no additional cost with the SUSE Linux support subscription.

SUSE provides the HPC Module for customers using the X86-64 and Arm hardware platform. Except for a few hardware specific packages, all the packages are supported on both platforms. If you haven’t tried the HPC module yet, here are the  instructions on how to access it. The HPC Module is also included in HPC images available on Clouds like Microsoft Azure.

HPC Module updates

  • Added adios1.13.1
  • boost to 1.71.0
  • conman to version 0.3.0
  • cpuid to version 20180519
  • fftw3 to version 3.3.8
  • genders to version 1.27.3
  • gsl to version 2.6
  • hdf5 to version 1.10.5
  • hwloc to version 2.1.0
  • hypre to version 2.18.2
  • imb to version 2019.3
  • luafilesystem to version 1.7.0
  • lua-lmod to version 8.2.5
  • lua-luaposix to version 34.1.1
  • memkind to version 1.9.0
  • mpich to 3.3.2
  • mvapich2 to version 2.3.3
  • mumps to version 5.2.1
  • netcdf to version 4.7.3
  • netcdf-cxx4 to version 4.3.1
  • netcdf-fortran to 4.5.2
  • python-numpy to version 1.17.3
  • python-scipy to version 1.3.3
  • openblas to 0.3.7
  • Added openmpi3 3.1.4
  • papi to version 5.7.0
  • petsc to version 3.12.2
  • Added PMIx version 3.1.5
  • scalapack to 2.1
  • scotch to version 6.0.9
  • slurm to 20.0.2
  • trilinos to version 12.14.1

HPC relevant base system packages

  • libfabric 1.8 to enable AWS
  • rdma-core 24.0 to enable AWS

PackageHub

PackageHub is a SUSE curated repository for community supported, open-source packages. A number of AI/ML packages were added to PackageHub for SP2.  These packages can be installed using zypper. Tips for using PackageHub.

  • armnn 20.02
  • caffe  1.0
  • charliecloud 0.15
  • clustershell 1.8.2
  • numpy 1.17.3
  • pytorch
  • robinhood 3.1.5
  • singularity 3.5.2
  • tensorflow2 2.1.0
  • theano 1.0.4
  • warewolf 3.8.1

New HPC and ML/AI Systems

SLE HPC 15 SP2 enabled a number of servers for HPC and AI/ML workloads including:

  • NVIDIA TegraX1
  • NVIDIA TegraX2
  • Fujitsu FX700 – (also SUSE “YES” certified)
  • AWS Graviton2 instances in AWS Cloud

Other HPC changes

  • Add Elastic Fabric Adapter (EFA) driver to enable HPC on AWS
  • Add the Live Patching Extension to the SLE-HPC Product
  • Improve clustduct cluster distribution tool
  • Remove unneeded package ‘ohpc’

Summary

The HPC ecosystem continues to expand and transform to include simulation, data analytics, and machine learning. SUSE HPC will continue to grow with the needs of our customers.

Posted in ARM Processors, Cloud, High Performance Computing, Information Technology, SUSE Linux, Uncategorized | Tagged , , , | Leave a comment

Understanding SUSE Sub-capacity pricing for IBM Power servers

SUSE_value_money_scale

 

Executive Summary

SUSE recently updated Terms and Conditions for SUSE products to clarify the SUSE pricing policies for IBM Power systems and to accommodate Sub-capacity pricing on IBM Power servers.

Sub-capacity pricing overview

IBM Power servers have long been known for vertical scalability (up to 192 cores and 64TB of RAM) and for efficient PowerVM virtualization infrastructure. As a result, many customers use IBM Power servers to consolidate multiple workloads on a single server.

PowerVM virtualization can guarantee that a particular workload can only run on a subset of the processors on a particular system.  For example, you could have a 192 core IBM Power server with 32 cores running a DB2 database with AIX and the other 160 cores could be running SAP HANA on SUSE Linux for SAP Applications.

Software subscriptions traditionally have been priced based on the amount of CPU capacity available to run the software, fewer cores generally mean lower software cost. This is known as Sub-capacity pricing because the customer is only paying for software based on a subset of the total hardware capacity.

Most IBM software is sold based on Sub-capacity pricing, so IBM Power customers expect to pay for software licenses or subscriptions based on the amount of processor capacity available to each workload.

Sub-capacity pricing for SUSE products on IBM Power

SUSE products started on X86 servers, where prices were traditionally based on the number of processor sockets as a measure of CPU capacity. Server consolidation is less prevalent in the X86 market and Sub-capacity pricing for software is less common.  As a result, the price of most SUSE subscriptions is based on the number of processor sockets available in the server, known as Full-capacity pricing.

In view of the unique capabilities of the IBM Power platform, we have changed the Terms and Conditions for SUSE product subscriptions to accommodate Sub-capacity pricing on Power. This will allow customers to partition their IBM Power servers with PowerVM virtualization and only pay for the capacity available to that SUSE product on a particular server.

The charge metric of SUSE product subscriptions on IBM Power has not been changed and is still measured in “socket pairs”. Because IBM PowerVM allocates processor capacity in terms of processor cores, there needs to be a way to convert the “cores” allocation in PowerVM to the equivalent number of socket pairs for SUSE subscription purposes.

Because SUSE subscriptions are sold in increments of socket pairs, to use Sub-capacity pricing on Power while still using socket pairs, you need to calculate the Socket Pair Equivalent that represents the CPU capacity that is made available to the SUSE workload.

Unfortunately, there is another complication in that IBM Power servers have different numbers of cores per socket, and thus per socket pair. For example, an IBM E980 server with hardware feature #EFP1 has 8 cores per socket, or 16 cores per socket pair. The same hardware ordered with feature #EFP3 has 12 cores per socket or 24 cores per socket pair.  You must know the number of cores per socket in the server before you can calculate the Socket Pair Equivalent for SUSE subscription purposes. The number of cores per socket on an IBM Power server is tied to the processor speed. Faster speed = Fewer cores per socket.

subcap0

Conditions for using Sub-capacity pricing for SUSE products on IBM Power

  • The SUSE product must use socket pairs as the charge metric.
    • SUSE Linux Enterprise Server (SLES) for Power, SLES for SAP Applications on Power, SUSE Manager Server for Power and SUSE Manager Lifecycle for Power are all included.
  • This is only applicable to POWER8 and POWER9 (and later generations) with four or more sockets.
    • E850, E880, E870, E950, and E980 are all eligible for Sub-capacity pricing.
  • PowerVM virtualization must be used to restrict the amount of capacity available to the SUSE products being used
    • Acceptable methodologies: Dedicated processor partitions (Dedicated LPAR), Dynamic LPAR, Single or Multiple Shared Processor Pools
    • Refer to the IBM standard on Sub-capacity pricing: https://www.ibm.com/software/passportadvantage/subcaplicensing.html
    • The Integrated Facility for Linux (IFL) processors does not automatically limit Linux workloads to only run on IFL processors and are not relevant to Sub-capacity pricing.
  • Customers and Sellers must account for the different number of cores per socket when calculating the Socket Pair Equivalent
    • Number of cores available to SUSE Linux ÷ by the number of cores per socket = Socket Pair Equivalent
    • Round up to nearest socket pair
  • The customer must purchase subscriptions for the maximum number of socket pairs that are ever available to run SUSE workloads.
    • Customer is responsible for purchasing additional subscriptions if the processor capacity is increased through changes to pools or activation of dark processors or other changes to the server capacity.

Example 1: Calculating Socket Pair Equivalent for IBM Power E950

In this example, the E950 has 10 cores per socket = 20 cores per socket pair.

subcap1

Example 2: Calculating Socket Pair Equivalent for IBM Power E950 with Rounding

In this example, the E950 has 10 cores per socket = 20 cores per socket pair.

subcap2

You can see that if the calculations end up with a fractional value, (that is, the number of cores is not an integer number of sockets) you must round up to the next nearest integer.

Example 3: Calculating Socket Pair Equivalent for IBM Power E950 lower core count and rounding

In this example, the E950 has 8 cores per socket = 16 cores per socket pair.

subcap3

As in the previous example, you can see that if the calculations end up with a fractional value, (that is, the number of cores is not an integer number of sockets) you must round up to the next nearest integer.

Example 4: Calculating Socket Pair Equivalent for IBM Power E950 with Dynamic LPAR

In this example, the E950 has 10 cores per socket = 20 cores per socket pair and the system administrator is using Dynamic LPAR (DLPAR) to dynamically change the number of cores available to SLES for SAP.

subcap4

Before the DLPAR operation, the SLES for SAP LPAR/VM had 18 cores of capacity which would round up to 1 socket pair of capacity but after the DLPAR operation, the SLES for SAP LPAR/VM is over 2 sockets of capacity and thus requires 2 SLES for SAP subscriptions.

The customer must provide enough SUSE subscriptions to cover the maximum capacity ever used.

Example 5: Calculating Socket Pair Equivalent for IBM Power E980 Server Consolidation

In this example, the E980 has 10 cores per socket = 20 cores per socket pair.

subcapp52

Once again, we see that if the number of cores assigned to the SUSE product is more than 2 socket pairs, so 3 SLES for Power subscriptions are needed for this configuration.

Example 6: Calculating Socket Pair Equivalent for IBM Power E980 with multiple SUSE products

In this example, the E980 has 12 cores per socket = 24 cores per socket pair.

subcapp62

There are two different SUSE products installed on this system. The Socket Pair Equivalent calculation must be performed for each SUSE product installed on a physical server.

Summary

Server consolidation is a common practice for IBM Power servers and Sub-capacity pricing is important to supporting consolidation. Using the Socket Pair Equivalent approach allows customers to leverage Sub-capacity pricing for SUSE products on IBM Power servers and to leverage the capabilities of PowerVM while maintaining compliance with SUSE terms and conditions.

Reference: Cores / Socket for IBM POWER9 Enterprise servers

POWER9_Sub-Capacity_Reference

Reference: Cores / Socket for IBM POWER8 Enterprise servers

subcapp8

Posted in AIX & Power Systems Blogroll, SAP HANA, SUSE Linux, Uncategorized | Tagged , , , , , , | Leave a comment

Using IBM POWER9 PowerVM Virtual Persistent Memory for SAP HANA with SUSE Linux

PowerVM_Persistent_memory_concept2

Introduction

SAP HANA uses in-memory database technology that allows much faster access to data than was ever possible with hard disk technology on a conventional database – access times of 5 nanoseconds versus 5 milliseconds. SAP HANA customers can also use the same database for real-time analysis and decision-making that is used for transaction processing.

The combination of faster access speeds and better access for analytics has resulted in strong customer demand for SAP HANA. There are already more than 1600 customers using SAP HANA on Power since it became available in 2015.

One of the drawbacks of in-memory databases is the amount of time required to load the data from disk into memory after a restart. In one test with an 8TB data file representing 16TB database, it took only six minutes to shut down SAP HANA, but took over 40 minutes to completely load that database back into memory. Although the system was up and responsive much more quickly, it took a long time for all data to be staged back into memory.

SAP has implemented features such as the Fast Restart Option to reduce the database load time, but system providers have enhancements to help address the problem. Intel introduced Optane DC memory technology where the data in memory is preserved even when powered down. Optane DC is positioned as a new tier of storage between DRAM and flash storage with more capacity at a lower price than DRAM but with less performance. Optane DC memory can deliver very fast restarts of SAP HANA but introduces new operational management complexity associated with adding a fourth tier of storage.

IBM PowerVM Virtual Persistent Memory

In October 2019, IBM announced Virtual Persistent Memory (vPMEM) with PowerVM. Virtual Persistent Memory isn’t a new type of physical memory but is a way to use the PowerVM hypervisor to create persistent memory volumes out of the DRAM that is already installed on the system. vPMEM is included at no additional charge with PowerVM.

The data in memory is persistent as long as the physical Power server is not powered off. By maintaining data persistence across application and partition restarts, it allows customers to leverage fast restart of a workload using persistent memory. An added benefit is that there is no difference in application performance when using vPMEM because the underlying DRAM technology is the same as for the non-persistent memory.

Although vPMEM does not preserve memory across a server power down, most IBM Power customers seldom power off their systems because of the many reliability features that are built into Power hardware. vPMEM provides for fast restart in the vast majority of planned maintenance and unplanned outages without compromising the performance of HANA during normal use.

sap_hana_memory_access­

Prerequisites

vPMEM has several prerequisites:

  • POWER9 processor-based systems
  • Hardware Management Console (HMC) V9R1 M940 or later
  • Firmware level FW940 or later
    • E980 system firmware FW940
    • L922 system firmware FW940
  • PowerVM level V3.1.1
  • SUSE Linux 15 for SAP Applications 15 Service Pack 1 with latest maintenance updates to bring the kernel up to 4.12.14-197.26-default or later
  • SAP HANA 2.0 SPS 03 Revision 35 (2.00.035)
  • SAP HANA 2.0 SPS 04 (adds new features for memory management)
  • SAP Note 2618154 SAP HANA Persistent Memory – Release Information
  • SAP Note 2700084 FAQ: SAP HANA Persistent Memory
  • SAP HANA Administration Guide – Administration Guide: Persistent Memory

IBM has provided the following documents:

 

SUSE has three technical support bulletins that you should review before implementing vPMEM

  • TID 7024333  “Activation of multiple namespaces simultaneously may lead to an activation failure”
  • TID 7024330 “vPMEM memory backed namespaces configured as a dump target for kdump/fadump takes a long time to save dump files”
  • TID 7024300 “Hot plugging/unplugging of pmem memory having a size that is not in multiples of a specific size can lead to kernel panics”

Summery

Virtual Persistent Memory is the latest tool in a long line of IBM and SUSE innovations to help customers get the most out of their SAP HANA on Power environments.

You should strongly consider using vPMEM when running SAP HANA on POWER9 systems.

Posted in AIX & Power Systems Blogroll, Information Technology, SAP HANA, SUSE Linux | Tagged , , , , , , | 1 Comment

SUSE Linux Essentials – Where are the compilers? Understanding the Development Tools Module

Binoculars_SUSE_Gecko_view_explore

Compilers on SUSE Linux

If you are new to SUSE Linux, you might have wondered why the C compiler on the system is so old. For example, on a SUSE Linux Enterprise Server (SLES) 12 Service Pack (SP) 3 system running on X86-64, the default gcc is version 4.8-6.189! Why isn’t the C compiler a newer version, like gcc version 8?

A SUSE Linux system can have multiple versions of the gcc compiler. The first type of compiler, the one used to compile the version of SUSE Linux that you are running, is known as the “System Compiler”. The “System Compiler” usually does not change throughout the life of the SLES version because changing it would greatly complicate creating patches to maintain the operating system. For example, gcc 4.8 is the SLES 12 “System Compiler” used to compile all SLES 12 releases including all Service Packs.  gcc 7-3.3.22 is the “System Compiler” for SLES 15.

The other type of compilers available on SLES are known as a “Toolchain Compilers”. These are the primary compilers for application development and are periodically updated with patches and new stable compiler versions as they become available. Usually there is more than one version available at any point in time. Most developers want to use newer compilers for application development because they provide additional optimization and functionality.

Installing the newer “Toolchain” compilers is easy, but first you must understand a little about the SUSE Linux Modules concept.

SUSE Linux Modules Introduction

SUSE introduced the concept of operating system Modules in SLES 12. SLES Modules are groups of packages with similar use cases and support status that are grouped together into a Module and delivered via a unique repository. SUSE introduced the Module concept to allow delivery of the latest technology in areas of rapid innovation without having to wait for the next SLES Service Pack. SUSE fully maintains and supports the Modules through your SUSE Linux subscription.

With the introduction of SLES 15, Modules have become even more important because the entire SUSE Linux OS is packaged as modules. The SLES 15 installation media consists of a unified Installer and a minimal system for deploying, updating, and registering SUSE Linux Enterprise Server or other SUSE products. Even the ls and ps commands are part of “Basesystem Module 15” module in SLES 15.

During deployment you can add functionality by selecting modules and extensions to be installed. The availability of certain modules or extensions depends on the product you chose in the first step of this installation. For example, in SLES 15, the HPC Module is only available if you selected the SUSE Linux Enterprise for HPC product during the initial installation.

INstall_select_Modules_15_Inverted

SUSE Linux Modules enable you to install just the set of packages required for the machine’s purpose, making the system lean, fast, and more secure. This modular packaging approach also makes it easy to provide tailor-made images for container and cloud environments. Modules can be added or removed at any time during the lifecycle of the system, allowing you to easily adjust the system to changing requirements. See the Modules Quick Start Guide for more information.

Please note that just activating a Module does not install the packages from that Module—you need to do that separately after activating a Module.  To use a package from a Module generally requires two steps: 1) activating the Module and 2) installing the desired packages.  Some Modules are pre-activated based on the product you installed. For example, the HPC Module is automatically activated when you install SUSE Linux for High Performance Computing 15.

The Development Tools Module is the SLES 15 Module that includes the most recent compilers such as gcc, debuggers, and other development tools (over 500 packages for SLES 15). The Toolchain Module provided access to newer compilers for SLES 12.

Activating the Development Tools Module using YaST

  1. Start YaST.                                                    # Requires root privileges
  2. Click Add System Extensions or Modules.

Install_Software_Management

  1. Select Development Tools Module.

Install_select_Development_tools

  1. Press Next to start the activation process for the Development Tools Module

Installing a package (gcc8) from the Development Tools Module

Once the Development Tools Module is activated, you can install packages from the Module. This example uses gcc8.

  1. Start YaST and click Software then Software management

Yast_Software Management

  1. Type gcc8 in the search field and click Search.

YaST_Search_gcc8

  1. Select gcc8 and click Accept.

The gcc8 compiler will now be installed.

YaST_Gcc_8_installed

And that’s it! You have successfully activated the Development Tools Module and installed gcc8.

Activating the Development Tools Module on SLES 15 using the command line

Now let’s take a different approach and use the command line to perform the same tasks. You will need to have root privileges to perform most of these steps either by su to root or using sudo.

  1. Activate the Development Tools Module.
sudo SUSEConnect -p sle-module-development-tools/15/x86_64

CMD_Out_SUSEConnect_activate_Devtools

  1. Verify that the Development Tools Module is activated.
sudo SUSEConnect –l

CMD_Out_suseconnect-l

In this example, the BaseSystem, Development Tools, Desktop Applications, and Web and Scripting Modules are activated. The other Modules and Extensions are available for activation.

  1. Check the status of gcc on this system.
zypper search gcc

CMD_Out_zypper_search_gcc.png

You can see that the system C compiler is gcc7 and is installed (i)  and gcc8 is available for installation.

  1. Install the gcc8 compiler.
sudo zypper install gcc8      # Must be root

CMD_Out_zypper+install_gcc8

  1. Verify that the gcc8 compiler was installed

zypper search gcc8

CMD_Out_zypper_search_gcc8

  1. Check the gcc8 version.
gcc-8 –version

CMD_Out_gcc8--version

Summary

You should now understand how to activate a SUSE Module and install a package from the module. This same approach can be applied to the other SUSE Modules, such as the HPC Module. Modules are a key concept of SUSE product packaging, particularly for SLES 15.

Posted in High Performance Computing, SUSE Linux, Uncategorized | Tagged , , , , , , | Leave a comment

The ibmvnic driver with SR-IOV is now supported by SLES 12 and SLES 15 on IBM POWER9 processor-based systems

green_networkSUSE has worked with IBM to enable support for the ibmvnic driver with PowerVM on POWER9 systems.

The ibmvnic driver enables PowerVM Single Root I/O Virtualizations (SR-IOV) for improved network capabilities including reduced systems processor utilization, decreased network latency, and enforcement of network Quality of Service.

Supported SLES Levels:

  • SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)
  • SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4)
  • SUSE Linux Enterprise Server 15

Supported Hardware:

  • All IBM POWER9 processor-based systems using PowerVM virtualization

IBM and SUSE strongly recommend customers apply the latest maintenance updates to SLES, PowerVM, and the latest firmware level for the POWER9 hardware before using ibmvnic.

Customers should not attempt installation of SLES 12 SP3 over an ibmvnic network.

Please NOTE: ibmvnic is not supported on POWER8 systems. Ibmvnic is a technology preview on POWER8 based systems.

Full details are in the SUSE TID at https://www.suse.com/support/kb/doc/?id=7023703

Posted in AIX & Power Systems Blogroll, Information Technology, SAP HANA, SUSE Linux | Leave a comment

SUSE Linux for Arm is now available for all customers

green register 2

SUSE Arm products available to all

Subscriptions for SUSE Linux Enterprise Server for Arm and SUSE Manager Lifecycle for Arm are now available directly to customers through the Corporate price list or through the SUSE Shop https://www.suse.com/shop/

Previously, SUSE subscriptions for the Arm hardware platforms were only available to SUSE Partners due to the relative immaturity of the Arm server platform. Now that we have delivered four releases of SUSE Linux Enterprise Server for Arm and have customers running SUSE Linux on Arm servers as diverse as the tiny Raspberry Pi and the high performance HPE Apollo 70 servers, we are now ready to sell subscriptions directly to customers.

Diversity of Arm: An Advantage and Challenge

As you can see from the chart below, SUSE has enabled a number of different Arm System-On-a-Chip (SOC) processors. One of the major advantages of the Arm platform compared to other platforms is the huge diversity of processors and processor features.

Each Arm licensee has great freedom to create their own System-On-a-Chip (SoC) engineered with the I/O, memory bandwidth, and number of processor cores needed for a specific workload. The number of processor cores varies from 1 processor to 48 processors per core. A single Arm SoC can have multiple varieties, such as the Marvel OcteonTX that is available in with 1 to 24 cores.

sles15_arm_enablement

Beyond socket-based pricing

The ability to create Arm SoCs with just the right I/O and number of processor cores needed for a particular workload is one of the largest advantages of the Arm platform, but it is also a pricing challenge for software vendors such as SUSE.

Software pricing should:

  • Reflect the value the customer receives from the software
  • Be as simple and easy to understand as possible

Both goals are essential, but sometimes difficult to achieve. For example, the value of a high availability solution that reduces system outages might be much more valuable to an airline than it would be to a small retail shop due to the difference in the number of people and revenue affected by an outage.

Software vendors have traditionally priced software based on the amount of processing capability available to the application (or Operating System). The underlying concept is that the more capacity, the higher the value, and therefore the higher the price.

For example, on a UNIX hardware platform such as IBM Power, IBM software cost is based on the number of processor cores and the relative performance of each core-something IBM calls the Processor Value Unit (PVU). Oracle has a similar approach called the Core Factor.  The underlying concept is simple: the bigger the system, the greater the value to the customer and thus, the more expensive the software. Unfortunately, both of these approaches are relatively complex.

By contrast, software pricing in the commodity Linux server market has traditionally used a much simpler measure of capacity: processor sockets. The more sockets in a server, the higher the software costs. Most SUSE software subscriptions are based on increments of 1-2 sockets of processing capacity. For example, subscriptions for SUSE Linux Enterprise Server on X86 servers have a U.S. list price of $799 per socket pair. If you have a four-socket server, then you would need two 1-2 socket subscriptions for a total list price of $1,598.

The traditional socket-based pricing goes back many years, when each socket might have only a single processor core. Although this pricing model is very simple, it doesn’t fit well to the diversity of systems based on Arm SoCs where a single socket can hold a single, low performance Arm core or 48 (or more) high performance cores.

Hybrid pricing for SUSE products on Arm

SUSE needed a pricing approach for SUSE Linux and other products for the Arm platform that could be used for very small systems such as the Raspberry Pi and for huge systems such as the HPE Apollo 70. The pricing approach also needed to be simple to understand. Ultimately, we decided to use a model that has core-based pricing for lower end Arm hardware and socket-based pricing for everything else.

The new pricing for SUSE Linux Enterprise for Arm is tied to the number of processor cores in a server. Servers with less than 16 cores are priced based on the number of groups of 4 processor cores. Each group of 4 cores, up to 15 cores, requires a 4-core group subscription that is stackable to a maximum 4 subscriptions. The number of cores is rounded up to the nearest group of 4 cores, therefore a server with 10 cores would require three 4-core group subscriptions.

Servers with 16 or more cores use the traditional 1-2 socket-based pricing.

This hybrid pricing model applies to SUSE products for Arm that have traditionally used socket-based pricing, such as the SUSE Linux Enterprise Server (SLES) for Arm and SUSE Manager Lifecycle. Other SUSE products such as SUSE Enterprise Storage uses server-based pricing and do not use this hybrid pricing.

Hybrid pricing examples for SUSE products on Arm

Raspberry Pi 3 Model B

The Raspberry Pi 3 Model B has 4 Broadcom processor cores. Because the total number of cores in the server is less than 16 cores, you use the 4-core group-based pricing.

arm pricing examples-rpi

NXP LS2085A

Systems based on the NXP LS2085A have 8 processor cores. Because the total number of cores in the server is less than 16 cores, you use the 4-core group-based pricing.

arm pricing examples-ls2085

Marvell OcteonTX

Systems based on the Marvell Octeon TX can have from 1 to 24 processor cores. In this example, we use a 14 core Octeon TX processor. Because the total number of cores in this server is less than 16 cores, you use the 4-core group-based pricing. This example also demonstrates how the number of cores is rounded up to the nearest integer multiple of 4.

arm pricing examples-octeontx

HPE Apollo 70

The HPE Apollo 70 is based on the Marvell ThunderX2 which can have from 16-32 processors per socket. In this example we use an HPE Apollo 70 dual socket server with 32 processor cores per socket for a total of 64 cores in this system. Because the server is greater than 15 cores, you use the 1-2 socket pricing.

arm pricing examples-apollo70

Summary

The diversity of the Arm server platform requires a more sophisticated pricing model than the simplistic socket-based approach. The new hybrid pricing model strikes a balance by using a core-based approach for smaller systems while retaining the traditional socket-based approach for larger servers.

Posted in ARM Processors, High Performance Computing, Internet of Things, SUSE Linux | Tagged , , , , | Leave a comment

Success Story: Using SUSE Linux for Arm with the Raspberry Pi to transform manufacturing

RaspberryPi_Locomotive.jpg

Knorr-Bremse, a long time SUSE customer, is deploying SUSE Linux Enterprise Server for Arm (SLES for Arm) to transform their manufacturing operations with a goal of improving productivity, reducing downtime, and improving factory floor operations. The solution success story can be found here: https://www.suse.com/c/success/knorr-bremse-ag/

Knorr-Bremse is a German manufacturer of a variety of rail and commercial vehicle components including brakes, door systems, and air conditioning systems. Knorr-Bremse employs over 27,000 workers around the world.

Knorr-Bremse had a problem. Many of their manufacturing machines have service lives of up to 30 years.  Some older machines are not instrumented for remote monitoring, resulting in manufacturing outages and other problems. Knorr-Bremse needed to make their manufacturing operations “smarter” to improve production efficiency and reduce downtime.

The Raspberry Pi single-board-computers were a logical choice for a monitoring hardware platform due to their low cost, easy availability, and the wide variety of I/O sensing devices that are available. Knorr-Bremse experimented with Raspberry Pi devices running the community supported Raspbian operating system, but they soon concluded that they needed an enterprise operating system as part of their solution.

Knorr-Bremse needed an operating system that was secure and reliable. Since the Raspberry Pi-based monitoring systems would be running inside of their corporate network, security was a big concern. Because these systems were key to improving manufacturing efficiency, the systems also had to be reliable. They needed the operating system to be commercially supported, so that they could receive fixes and assistance resolving any operating system problems that might arise. Finally, they wanted a commercially supported management platform to deliver updates to these systems.

Knorr-Bremse seized on the commercial support of SUSE Linux on Raspberry Pi single-board computers as a cost-efficient way to add monitoring to their manufacturing. The solution consists of SUSE Linux Enterprise Server for Arm (SLES for Arm) running on Raspberry Pi 3 Model B single-board computers with management services provided by SUSE Manager.

Their monitoring solution also includes a number of custom built I/O boards that attach to the Raspberry Pi I/O ports to sense different operational factors such as temperature and machine status. These Raspberry Pi-based systems are IoT gateways that gather information and send that information to a centralized manufacturing operations team. Some of the Raspberry Pi systems include touch screens that simplify communications from manufacturing workers.

Although Knorr-Bremse is early in the process of deploying this solution to their manufacturing plants, the initial results made them confident enough to release this success story.

SUSE has a number of manufacturing and government agency customers implementing monitoring solutions based on SLES for Arm on the Raspberry Pi systems. All of them leverage the security and reliability of SUSE Linux as well as the SUSE Manager to deliver a monitoring solution that can be distributed in a wide range of environments. SLES for Arm on the Raspberry Pi uses the same common code base used for SLES on X86, Power, and mainframe servers. SUSE provides a SD-Card image with customers and use to quickly install SLES on the Raspberry Pi.

Customers can experiment with SLES for Arm on Raspberry Pi devices by using a free 60 day trial version of SLES for Arm: https://www.suse.com/products/server/download/.

Information on how developers can obtain a free Developer License for SLES for Arm can also be found here: https://www.suse.com/c/free-developer-subscriptions-for-suse-linux-on-arm-based-systems-are-now-available/

SUSE looks forward to helping other customers solve difficult problems using SUSE Linux with Raspberry Pi and other small devices. The combination of enterprise Linux with extremely low cost hardware can enable solutions to problems that previously were impractical to address.

Posted in Uncategorized | Leave a comment

SUSE supports innovative High Performance and Edge Computing cross-industry initiative

Hands_pulling_digital_rope

As the creator of the first commercially supported enterprise Linux distribution, SUSE is no stranger to open source innovation or working with partners to achieve industry goals.

SUSE is proud to join the Forschungszentrum Jülich research institute, Fraunhofer Institute for Open Communication Systems, Huawei, Mellanox, and E4 in a new initiative to develop open ecosystems for High Performance and Edge Computing.  This initiative brings together end users and solution providers to accelerate adoption of new technologies for HPC and Edge computing workloads including deployments of 5G networks.

SUSE contributions to this initiative will leverage our experience delivering three versions of our commercially supported Linux distribution for Arm. SUSE supports a variety of 64-bit Arm System-on-a-Chip processors including those from Marvell (Cavium), NXP, Qualcomm, HiSilicon, Ampere, Mellanox, and Xilinx among others. SUSE Linux for Arm is already being used by customers for workloads such as the Catalyst UK High Performance Computing project with HPE. Multiple manufacturing customers are also using SUSE Linux on the Raspberry Pi for industrial IoT automation and monitoring.

SUSE as a key player in High Performance Computing where SUSE Linux is the underpinning for almost half of the TOP 100 HPC systems. SUSE has already delivered an initial set of HPC infrastructure packages such as slurm, openblas, openmpi, and fftw that are supported as part of the HPC Module, available for Arm-64 and X86-64 platforms.

This new initiative is expected to improve the maturity of ecosystems across multiple industries and simplify using new technologies such as Arm processor technology to deliver innovative High Performance and Edge Computing solutions. ­­

Press release:  http://www.fz-juelich.de/SharedDocs/Pressemitteilungen/UK/EN/2018/2018-10-10-hpc.html

Posted in ARM Processors, High Performance Computing, Internet of Things, Open Source | Tagged , , | Leave a comment