An Early Valentine’s Day present from Power Systems

IBM is providing Power Systems clients with an early Valentines Day present – a POWER7+ processor refresh for (most of) the Power Systems product line.  The POWER7+ processors made their debut in October 2012 in the “D” models of the Power 770 and Power 780 systems.

This announcement brings the POWER7+ processor advantages of higher clock speeds and built-in cryptographic and memory compression accelerators to the Express (low end) Power Systems products. The announcements today include POWER7+ versions of the Power 710, Power 720, Power 730 and Power 740 Express servers, a completely revamped Power 750 Express server. There is also a new entry to the Power Systems mid-range product line,  the IBM Power 760 server, that supports up to 48 cores.

The new POWER7+ processor based systems offer twice as much memory as their POWER7 predecessors – up to 2 Tb of memory in the new Power 760 model. Most of the Power clients I talk to tend to run out of memory before they run out of processor capacity, so the increase will certainly be appreciated. By the way, the new, higher capacity memory modules will not work in the older “C” model systems, so if you need the additional memory you will need to step up to the new systems.

All these new systems include new POWER7+ reliability features such as a self-healing capability for L3 cache and the ability to restart a processor without restarting the system. These features help avoid unplanned outages and improve the ability to provide concurrent firmware updates to avoid some planned outages.

When coupled with PowerVM enhancements that allows virtual machines to be sized as small as 1/20^th of a core, the POWER7+ “D” model systems will support up to twice as many virtual machines (aka LPAR) as their POWER7 based predecessors.

The new systems will be supported by AIX, IBM i, and Linux, except for the Linux-only 7R1, 7R2 and p24L systems. There is a statement of direction for AIX 5.3 support coming in the future for clients that have purchased the AIX 5.3 Service Extension. 

If the additional performance and doubled memory wasn’t enough, IBM has priced the 710 and 730 Express servers right on top of their X86 competition–making Power Systems even more attractive to clients that focus on cost of acquisition.

Now things really get interesting with a POWER7+ version of the extremely popular Power 750 Express server. The new “D” model isn’t just a refresh of the old model with POWER7+ chips – it is really a blend of the Express 750 and mid-range Power 770 technologies.

You’ll notice that the new Power 750 Express is in a bigger, 5U form factor. When you see all the new capabilities you’ll know why they needed a bigger box. Like the predecessor 750, the “D” model has 4 sockets with 8 processors per socket for 32 total cores running at 3.5 or 4.0Ghz. The new 750 can be equipped with up to 1Tb of memory and will support up to 640 virtual machines running AIX, IBM i or Linux.

As the TV commercials say: “But wait there’s more”. The Power 750 Express server sports the same kind of I/O you would expect in a Power 770. For starters, it has 6 PCIe Gen 2 slots and 6 disk bays. It also supports four different multifunction I/O cards, that can provide, for example, two 10Gb Converged Network Adapters and two 1/10Gb Ethernet ports.  The Power 750 also includes two GX++ ports, each with 20Gb bandwidth.  Finally, the Power 750 includes a split backplane allowing you to divide up the internal drives between two virtual machines; for example, dual VIOS servers. The Power 750 also comes with three year of maintenance in the purchase price.

With all of these changes, the 32 core Power 750 Express Server will rival the performance of the top of the line 64 core POWER5+ 595 system of a few years ago —at a fraction of the price.

IBM also is announcing a new model to the Power Systems family, the Power 760. The Power 760 is a mid-range server, a baby brother to the Power 770 server. The Power 760 is intended as a server consolidation platform with up to 48 cores of capacity. The processors are supported by up to 2 Tb of memory to allow for up to 960 virtual machines in a single 5U system.

The Power 760 shares the same 5U form factor as the Power 750 as well as the Power 770 inspired I/O configuration including the multifunction I/O cards, dual GX++ busses and the split backplane. It also includes the same reliability features as the Power 750 Express and three years maintenance of hardware maintenance in the purchase price.

The Power 760 also offers Capacity Upgrade on Demand (CUoD) for processors to provide for even more flexibility.  The CUoD feature allows clients to activate processors to respond to increased workloads without interruption. The CUoD capability is limited to permanent activation only and does not include CUoD for memory.

There are a number of other new products announced today including: POWER7+ updates to the Linux-only PowerLinux 7R1 and 7R2 systems, new I/O including a 16Gb fibre channel adapter. There is also a IBM i 7.1 Technology Refresh 6 (TR6) with lots of enhancements for IBM i and PowerHA SystemMirror for IBM i.  Finally, there is also a new Power Systems version of the PureApplications System for increased performance and workload density.

There’s no new AIX Technology Levels announced today in keeping with the AIX Release Strategy, but there are updates to AIX Solution Edition offerings for Cognos and SPSS.

There is much more detail in the announcement letters that are available on ibm.com/power. A Smarter Computing webcast highlighting the announcements is scheduled for 11 AM EST on Feb 5.

Jay

chromeaix@twitter

October 2012 Power Systems Announcements

Well it’s that time of year again, time for IBM to release the next set of hardware and software enhancements for Power Systems.

First up is a new processor, the POWER7+. The POWER7+ provides additional per core performance through faster clock speeds and lots more cache. But the POWER7+ chips also include accelerators that can shift some specialized processing workloads off of the POWER7 cores. There are three accelerators that are part of the POWER7+ chips: An analog-based random number generator that provides for more security when using cryptography, a memory compression/decompression accelerator that can potentially reduce processor utilization and improve performance when using Active Memory Expansion, and cryptographic offload engines that can potentially improve the performance and reduce the processor utilization when using IPSec and AIX Encrypting Filesystem.

The new POWER7+ processors have been released in new “D” models of the popular POWER 770 and POWER 780 systems. In addition to the new processors, both of these systems include additional features to improve the scalability, performance and reliability of these systems.  The POWER 780 can now contain a maximum of 128 cores.

In addition to these new POWER7+ processor-based systems, the existing POWER 795 has been enhanced with additional memory capability (up to 16TB), enhanced I/O and greater virtualization flexibility.

Significant enhancements to Power Systems Virtualization include: a minimum entitled capacity of 1/20^th of a core when using MicroPartitioning, up to 16 concurrent Live Partition Mobility operations and a significant performance improvement in Live Partition Mobility. There is also a new capability called Dynamic Platform Optimizer (DPO) that allows an administrator to defragment system resources without requiring an outage. System resources can become fragments over time, and DPO can eliminate this problem to insure high sustained performance. These enhancements require the 760 level of the firmware, the latest VIOS (2.2.2) and the latest HMC. The 760 firmware will be initially available on the new Power 770 and Power 780 systems and on the Power 795.

A new System Pools option is now available for the Power 780 and Power 795 that allows for sharing of Elastic Capacity on Demand resources between multiple systems. This is intended to reduce the impact of planned maintenance events by allowing workloads to be relocated to other systems during the planned maintenance without requiring significant amounts of idle resources.

Continuing on the list of virtualization features, PowerVM includes some usability enhancements and a new VIOS Performance Adviser that is intended to simplify tuning your VIOS systems. An earlier release of the VIOS Performance Advisor is available as-is on the IBM DeveloperWorks web site, but the new version is included with PowerVM as a supported tool.

The scalability and reliability of PowerVM Shared Storage Pools has been enhanced to provide support for up to 16 nodes in  a cluster, a redundant and resilient cluster repository and pool utilization and reporting tools.

AIX 6 Technology Level 8 and AIX 7 Technology Level 2 provide exploitation of the new POWER7+ accelerators and include new capabilities such as a LPAR to WPAR conversion option (AIX 7 TL2 only). AIX 6 TL8 picks up the Active System Optimizer functionality that was first delivered in 2011 in AIX 7 TL1. AIX also adds support for IPV6 networks for NFS V3 and enhanced IPV6 support for WPAR specific routing.

The AIX Enterprise Edition bundle is being changed to better reflect current client requirements. Due to the increased demand for cloud management tools, AIX EE will add PowerSC Standard Edition and the SmartCloud Entry for Power (including Director Storage Control). Two products, IBM Tivoli Monitoring for Energy Management and the Tivoli Application Dependency Discovery Manager are being dropped from the bundle due to limited client adoption.

PowerSC Express and Standard Editions have been enhanced to provide real time compliance checking and a new compliance profile for HIPAA (Healthcare Industry Portability Accountability  Act).

A new member of the PowerSC family has been introduced, PowerSC Trusted Surveyor. This new product is designed to simplify the security management of your PowerVM VLAN environment. PowerSC works by interrogating your HMCs to obtain the VLAN configuration. The administrator can define the relationships between VLANs and the LPARs they connect to, for example, classifying them into production, test, QA, etc. systems. Then, when you collect the information again the next day, PowerSC Trusted Surveyor will highlight any changes and identify any misconfiguration  (such as a “production” LPAR on a “test” VLAN).

Last but not least, PowerHA SystemMirror V7.1 Enterprise Edition has been introduced. PowerHA SystemMirror V7.1 Enterprise Edition brings the advantages of Cluster Aware AIX to disaster recovery environments. Features include: IBM Systems Director graphical management, robust, kernel based cluster communications, a multi-site setup wizard to simplify deployment. Additionally there is a HyperSwap option that, when used with IBM DS8000 storage, can provide continuous availability during storage planned maintenance or unplanned storage outages.

I hope you found this brief overview of the Power Systems October announcements useful. There is much more detail in the announcement letters that are available on ibm.com.

If you want to get an even more in-depth on these announcements and other topics, you should consider attending the Power Technical University events in Dublin or Las Vegas in October.

Jay

AIX 7 receives OSPP/EAL4+ Common Criteria Security Certification

The AIX 7 Operating System has received the new, Operating System Protection Profile Version 2.0 BSI-CC-PP-0067-2010 with the OSPP Extended Packages: General Purpose Cryptography, Integrity Verification, Virtualization,  Advanced Management, and Labeled Security at the EAL4+ level

The new OSPP Common Criteria certification replaced the previous Common Criteria certification such as the old Common Access Protection Profile (CAPP)

AIX has had a long history of receiving security certifications going back to the old C2 standard that AIX 4.2 was certified on back in 1997. As security certifications evolved to the Common Criteria standards, CAPP/EAL4+, successive releases of AIX including AIX 5.2, AIX 5.3 and AIX 6 all received certification.

In 2010 the Common Criteria organizations established the new OSPP standard as an evolution of the Common Criteria standard to address the fundamental changes in the industry as the result of the migration from single isolated systems to networked, multi-system environments http://www.atsec.com/us/news-operating-system-protection-profile-200.html

The Target of Evaluations (TOE) for this new AIX 7 certification includes an evaluation of the PowerVM Virtual I/O Server (VIOS).

Common Criteria certifications are based on a set of internationally recognized technical standards and configurations that are used to evaluate the security of Information Technology products and technology.

IT organizations typically use Common Criteria certification as one measure of the suitability of a particular computing technology for use in their business.

Obtaining a Common Criteria certification is a long and expensive process and this most recent certification is a testimony to the IBM commitment to AIX and Power Systems.

Jay

OSPP certificate for AIX 7 dated August 20, 2012

AIX 5.3 Service Extension

In 2004, the “Opportunity” rover landed on Mars. Janet Jackson had a “wardrobe malfunction”, Google launched gmail, and AIX Version 5.3 was released.

Eight years later, AIX Version 5.3 will officially go out of standard support at the end of April 2012.  AIX 5.3 will probably go down in history as one of the most widely adopted releases of AIX due to features such as support for POWER5, POWER6, MicroPartitions, Virtual I/O, 64 core scalability, SMT2, NFSV4 and a host of other features. But, like all good things, it too, must come to an end.

The reason it has to end is the “N-1″ policy for AIX releases. That means that there are generally only two releases of AIX in the market at any given time. This is done primarily because most independent software vendors (ISV) can only support two operating system levels at any point in time. If you have more than two AIX releases in the market, then ISV support would get fragmented across those releases. As a result, there are only two releases of AIX currently in the market – AIX 7 and AIX 6

Unfortunately there are a lot of clients out there still running AIX 5.3    It’s not that they didn’t have warning, IBM announced AIX 5.3 was going to be withdrawn from marketing back in the spring of 2010. But it is often difficult for clients to move to a new AIX release because they need to coordinate the move across their entire application ISV application stack.

ISVs often release a new version of their application every 12 to 18 months. Many clients don’t update to a later version of the application that often, so when it’s time to upgrade the AIX release, they also need to update to a later version of the ISV application. The need to upgrade the AIX OS and the ISV applications at the same time requires a lot of effort, so it often takes a long time to make the transition.

To help clients that could not make the transition to a later level of AIX from AIX 5.3 by the end of April 2012, we are making available a new service extension for AIX 5.3. This service extension will allow clients to continue to get support for AIX 5.3 even after the end of standard support. (Announcement letter here http://tinyurl.com/aix53extension )

This new AIX 5.3 service extension offering will:

• Provide usage support such as answers to configuration questions and problem determination
• Provide new fixes. This includes fixes for newly discovered security exposures. New fixes availability are subject to the standard “best commercial effort” restrictions. Fixes will be provided via interim fixes and, in some cases via Service Packs
• Be available for three years, May 2012 – 2015. Of course, plans are always “subject to change”
• This service extension is only available for AIX 5.3 Technology Level 12. Earlier AIX 5.3 Technology Levels will not be supported under this offering.
• New hardware toleration. As new hardware becomes available over the next two years, we will provide new hardware toleration when possible. This will not include new hardware that requires architectural changes.
• This service extension offering will be priced per core and will be subcapacity eligible. In other words, if you have 4 cores running AIX 5.3 on a 16 core server with AIX 7 running on the other 12 cores, you only need to purchase the AIX 5.3 service extension for the 4 cores running AIX 5.3
• Minimum term of six months. In most countries you will need to purchase this support for at least six months at a time. In the US, the minimum term is 90 days.
• Normal AIX Software Maintenance (SWMA) required. You will need to purchase this service extension in addition to normal AIX SWMA.

As you might have noticed, this service extension is significantly different from prior AIX release service extensions in three ways:

• New fixes will be delivered via Service Packs for the first two years
• Enablement for some new hardware will added during the first 2 years
• Announced with consistent prices and terms worldwide

For the first two years, 2012-2013 and 2013-2014, we intend to publish two Service Packs per year as part of this service extension.  These Service Packs will include cumulative fixes and new hardware enablement. Note that these Service Packs will only be available to clients that have purchased the service extension.  In addition to the Service Packs, clients will still be given Interim Fixes to address immediate issues. No service packs will be released the final year of the service extension, 2014-2014. In the last year, only interim fixes will be available and those will only be provided only as needed.

As noted above, we intend to provide support for some of the new hardware that is expected to be released after April 2012.  This new hardware support will not include support for any new hardware that requires architectural changes or nor will the support provide any exploitation of new hardware features.

The pricing and terms and conditions for the AIX 5.3 service extension will be very similar world wide, with variations for currency and other country specific requirements.

Some people have asked me about when they should use the AIX 5.3 service extension versus using the AIX 5.3 Workload Partitions product. The answer is pretty easy – if you intend to migrate to a later release of AIX, then use the service extension to bridge you to that point. If however, you believe that you will need to run AIX 5.3 indefinitely for a particular application, then the AIX 5.3 WPARs is the better choice.

Hopefully I have answered most of your questions about this new AIX 5.3 service extension offering. The intent of this offering is to help clients remain supported as they as they move up to later releases of AIX.

Jay

AIX Service Pack best practices

As discussed previously, IBM periodically delivers new function, new hardware support, and cumulative fixes for AIX in software updates called “Technology Levels”.  The name “Technology Level” was created in 2005 to replace the former name “Maintenance Level”. The new name indicated that a “Technology Level” wasn’t just a bundle of fixes, but may also include new functionality.  Technology Levels are now released once per year, and are a way for IBM to deliver cumulative fixes, functional enhancements and new hardware support between new releases/versions of the AIX operating system. Not all Technology Levels include functional enhancements, but all Technology Levels provide cumulative fixes and new hardware support.

Service Packs have a very similar purpose but there are some key differences. First, Service Packs are released about four times per year to deliver cumulative fixes and new hardware enablement on top of an existing Technology Level.  Second, Service Packs do not include functional enhancements. Since Service Packs do not include new function, they are viewed by many administrators as potentially less likely to introduce problems.

The ability to install individual fixes to AIX on a piecemeal basis, called “Selective Fixes”, was introduced in AIX Version 3.2 in 1992.  Selective fixes were a highly requested capability because with selective fixes, clients could pick and choose which fixes to install.

This seemed like a great idea; after all, shouldn’t installing only the fixes that you really needed reduce the risk of an update destabilizing your system? Well, unfortunately, the answer turned out to be “not so much”.

The challenge with selective fixes is how to manage the dependencies, also known as “requisites”, between different functional areas within AIX. For example, fixes to one area (such as NFS) might be dependent on a requisite fix being made in another area (such as the kernel).  The effect if this was that if you wanted to update NFS, you also had to also make sure you also made the requisite update to the kernel.

The knowledge about which fix requires which requisite fix comes from the developer. The developer has to code knowledge about co requisites into the individual fixes so that when you installed the NFS fix in the example above, the corresponding kernel fix was also installed.

Unfortunately, we observed that a number of the problems that clients were experiencing in the field were the result of hidden requisites between code areas. While it was relatively easy for a developer to understand and manage the requisites for code changes that she had introduced, it became difficult to manage the unintentional requisites that naturally occur in a complex environment like an operating system.

The irony is that when an administrator thought he was reducing risk by introducing fewer changes to the operating systems, he was in fact becoming a test pilot – a test pilot that was running a combination of components that had likely never been run before.

To avoid this issue, we came up with only one approach that is guaranteed to work: clients should install all the fixes at the same time. After all, IBM tests all fixes in a service pack (or Technology Level) as a unit so if clients install all the fixes (update_all) , then the client is using the same combination that IBM tested. This recommendation is part of the “IBM AIX Operating System Service Strategy Details and Best Practices” documented at http://www14.software.ibm.com/webapp/set2/sas/f/best/home.html

While IBM has not disabled the selective fix capability in AIX, we have made it more difficult to use by disabling the ability to download individual fixes from Fix Central – you must download the entire service pack. You can still extract and install individual fixes, but we don’t make that risky behavior easy.

In summary, IBM provides Service Packs for AIX because it is the best way for you to maintain the operating system. While it may be counter-intuitive, periodically installing all fixes in the Service Pack is the best way for you to maintain the AIX operating system between Technology Level updates and it is least risky way to update your AIX system.

Jay

Some “Whys” about AIX Technology Levels and Service Packs

Here are the answers to a couple of “Why” questions that I have gotten recently about how we roll out AIX fixes such as Technology Levels and Service Packs.

The first question:

“Why does a new Service Pack comes out at the same time as a new Technology Level?”

As you might imagine, testing a new AIX Technology Level is a very involved process that takes many months. As with any complex software, it is impossible to get to a point where all problems have been discovered and resolved. The process of releasing a new AIX Technology Level involves testing the code and evaluating the severity and impact of problems found to decide when the code is good enough to be released.

For example, a bug that caused a data integrity issue would be a severe enough problem that it would have to be fixed before the Technology Level could be released. In contrast, a typo in the man page would not be severe enough problem to hold up the release of the Technology Level.

As the testing of a Technology Level progresses, the product team tracks the severity of the new problems and the resolution of previously discovered problems against established quality criteria. Over time, the severity and number of new problems discovered during testing tapers off and most of the problems that have been discovered would have been resolved. At this point the quality of the Technology Level meets the quality criteria and the Technology Level is ready to be released to the distribution channels.

When I say “distribution channel”, I am talking about the two main ways that clients can obtain the new Technology Level: via Fix Central as downloadable update or as new bootable installation media that is physically replicated in multiple IBM distribution centers around the world.

It takes some period of time for new Technology Level to make it’s way through these distribution channels (such as time for the physical media replication), thus the new Technology Level has to be released to the distribution channels before it is actually available to the clients.

The key thing to understand is that we don’t stop testing the new Technology Level even though it has been released to distribution. In fact, there is ongoing testing of AIX going on all the time.

As we continue testing the Technology Level, we also test new fixes for the less severe bugs (such as the ealier example of  the typo in the man page), that we found before the Technology Level was released to distribution. Occasionally we also find new, more severe problems that need to be urgently need to be addressed. All of those fixes get included in the first Service Pack.

The near simultaneous release of the first Service Pack is built into our process of releasing a new AIX Technology Level.

By the way, this policy of releasing a fix at the same time as a major AIX update isn’t anything new. Before 2007 we would roll up all of these kind of fixes into something that we called  “The First PTF”. The First PTF would come out the same time as the underlying Maintenance Level. After 2007 we restructured AIX release support and did away with standalone PTFs which is why we now release these fixes as the first Service Pack.

Another “Why” question that has come up recently is:

“I’ve been running an Interim Fix for a problem for some time, but it wasn’t included in the latest Service Pack. Why not?” 

The key thing to understand is that Service Packs are not simply bundles of Interim Fixes. There is a lot more to releasing a Service Pack.

Let’s work though an example:

1. Let’s say that company XYZ has a problem in AIX
2. The admin for company XYZ calls IBM Support.
3. IBM Support investigates the problem and finds that there is a bug in the code.
4. The Support person crafts an Interim Fix that corrects the problem for that client.

The Interim Fix is specifically targeted at a particular AIX level in that particular client’s environment.  The point of that Interim Fix is to quickly address the problem for a particular client in that client’s specific environment.

The AIX architect responsible for that area will also analyze the bug and the Interim Fix provided as she works on a future Service Pack update. Remember that the Interim Fix for company XYZ that was delivered in step #4 above was tailored for that specific client’s environment. When it becomes time to roll that fix for that bug into the next Service Pack, the architect will look at the Interim Fix in the context of supporting all customers, not just company XYZ.

Sometimes the Interim Fix provided to company XYZ would not be appropriate for other customers. In that case, the architect will have to come up with a fix that would be more generally applicable. Because that process takes some time, that more comprehensive fix may not be delivered in the next Service Pack. Additionally timing and workload issues also affect when fixes get rolled up into a Service Pack. All of these factors may result in a fix not being delivered in the next Service Pack even though it had been made available to a client as an Interim Fix.

I hope this explanation of how we test and roll out AIX Technology Levels and Service Packs has been helpful. If you have other questions about AIX you can post a comment to this blog or drop an email to chromeaix@gmail.com and I might make it a topic for a future blog post.

Jay

Travel stories – Manila

In September of 2006 my colleague and I were on a four country conference tour of Asia. I would speak about AIX and virtualization and my colleague would cover Power Systems hardware.

The first two events in KL and Singapore went off without a hitch but we couldn’t go to the third event # in Thailand because the military had a coup the day I was to arrive. I didn’t know it until then, but I have a firm travel rule : “Never go to a place that has tanks in the streets”

Because we had already canceled one of the four events, we were really determined to do the fourth event, in Manila.

There was only one little problem, and that was that a category 5 typhoon was headed straight towards Manila.

The Manila team assured us that the conference would occur that Thursday morning as scheduled despite the weather. Despite my gut feel that this was a mistake,  we dutifully flew to Manila that Wednesday night.

Well the next morning the typhoon was starting to hit Manila in earnest and the winds were blowing at least 50 miles an hour. Our local country hosts called us and said “Err – we noticed that the wind picked up overnight so we are canceling the conference!”

You can imagine my thoughts. I said a few choice words and then we checked out of the hotel in an attempt to get out of the country before the typhoon struck. We made the drive to the airport in blinding rain and hurried to the counter. Unfortunately by that point the Manila airport had lost power and it was a surreal tableau of desperate people in the dim light of a few backup lights and lightning flashes. All the computers were powered down and the airport was shut down. Nobody was leaving.

After waiting in line for several hours, the airline (Singapore) finally decided to escort us to their lounge past the unpowered security checkpoint. But before we could leave, we had to check our luggage. The clerk HAND WROTE my baggage tag and I remember looking wistfully at that bag as I left the counter. I never expected to see that bag again.

The business class lounge didn’t have electricity but they did have candles, ice and booze. I proceeded to medicate myself in preparation for the typhoon.  I was really concerned about my safety and after the vodka loosened my tongue, I spent a good part of the time giving the local team a frank appraisal of their weather prognostication abilities via my cellphone.

The typhoon went directly over the airport and we watched the typhoon rip pieces off of airport buildings through the big picture windows in the lounge. The lounge was luckily on the leeward side of the storm so none of the lounge windows were broken.

Eventually the typhoon passed and the airport resumed operations. We boarded a flight to Hong Kong about 10:30 that night on an extremely lightly loaded Singapore Airlines plane.

As expected, my bag did not arrive with me (remember the hand filled out baggage tag?) I filed a missing luggage report but frankly thought it was a lost cause. The next day we managed to make the long trip home from Hong Kong, sans luggage.

Over the next few days, when I inquired about my missing bag Singapore Airlines could only tell me that they were searching for it but had no status or information on where it might be.

Three days later, out of the blue, I get a call from American Airlines in Austin – my bag had managed to make it home after all!

That was my first real trip to that part of Asia and it took five more years before I ever got the nerve to return to Manila.

Things I learned on that trip: Go with your gut feelings. Be flexible. Stay at the hotel if there is any doubt about your flight. Don’t worry, it’ll (probably) work out all right in the end and never, ever, ever check a bag.