advertisement

The experiences of migrating a large scale, high performance healthcare network

60 %
40 %
advertisement
Information about The experiences of migrating a large scale, high performance healthcare...

Published on July 10, 2008

Author: george.james

Source: slideshare.net

advertisement

The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare

The experiences of migrating a large scale, high performance healthcare network

Larry Williams

Corporate Manager, Partners HealthCare

In the next half hour… Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements Post database migration results Future milestones

Partners Healthcare System overview

Caché platform architecture & metrics

The need to migrate

Phased migration approach

Benchmark testing and results

Discoveries and production enhancements

Post database migration results

Future milestones

Partners Healthcare System Founded in 1994 Brigham & Women’s Hospital Massachusetts General Hospital Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million

Founded in 1994

Brigham & Women’s Hospital

Massachusetts General Hospital

Now includes:

Community physician network (1200 + 3500 MD’s) PCHi

3 community hospitals

2 rehab hospitals

3 specialty institutions

Enterprise-wide Information Systems

1100 employees

Annual budget FY05 approximately $160 million

Anchor Hospitals & Airport BWH MGH Logan Airport 10 km 6 km

Acute Care Hospitals MGH BWH Newton- Wellesley Community Physician Practices

Partners Domain Devices Internet 12,000 Printers 32,000 Desktops Firewall ~30,000 other devices 1,450 Servers Closely Managed Assumed Managed

Current Mixed-Mode Production Architecture

Enterprise Integration Over 30% are to and from Caché database Change from prior year Daily Average Est. Annual Transactions # of Interfaces 196 170 192 167 37% 4,659,035 1,330,962,017 2007 40% 3,399,211 1,240,712,044 2006 45% 2,431,917 887,649,802 2005 1,673,515 610,833,080 2004

Integration Components

Annual Database Growth Rate

Database Utilization Average Database References per day in Billions

The Need to Migrate - Availability Monthly Downtime Current State Business need

Additional Business Requirements Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover Reduce monthly maintenance needs, updates once or twice per year -------------------------------------------------------- Improve Performance 64 bit OS, more memory for cache Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance Increase Scalability 91 Terabytes available on EMC SAN DMX3 On-demand addition of processor cores

Increase availability and reliability

Decrease database risk from 5 single points of failure

More robust hardware and OS

Many less servers and OS instances to manage

Clustering and automated failover

Reduce monthly maintenance needs, updates once or twice per year

--------------------------------------------------------

Improve Performance

64 bit OS, more memory for cache

Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance

Increase Scalability

91 Terabytes available on EMC SAN DMX3

On-demand addition of processor cores

Caché Migration Decision Making Process Only considered first tier vendors and support (IBM, HP) HP assumed much more risk with Professional Services Existing HP business yields more leverage & visibility with regional office More headroom in HP configuration Price was not a distinguishing factor

Only considered first tier vendors and support (IBM, HP)

HP assumed much more risk with Professional Services

Existing HP business yields more leverage & visibility with regional office

More headroom in HP configuration

Price was not a distinguishing factor

Phased migration approach Proof of Concept (benchmark testing) Completed 10/15/07 Phase 1 – Database tier Completed 4/14/08 Phase 2 – Application tier Big Bang migration 12/14/08 (includes Cache 2008.1 UNIX platform upgrade) Phase 3 – Disaster Recovery Now implemented as part of Phase 2, leveraging HP Blade servers

Proof of Concept (benchmark testing)

Completed 10/15/07

Phase 1 – Database tier

Completed 4/14/08

Phase 2 – Application tier

Big Bang migration 12/14/08 (includes Cache 2008.1 UNIX platform upgrade)

Phase 3 – Disaster Recovery

Now implemented as part of Phase 2, leveraging HP Blade servers

UNIX DataBase Tier Benchmark Environment

UNIX Application Tier Benchmark Environment

Benchmark Load Testing Results Goals Simulate current Production user counts & transaction loads Verify support for load increases up to 3x Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers, 10 script ‘players’) Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen LMR functions) Conclusions Able to simulate production load, 1.5x and 3x load 2 HP rx8640 can handle growth projections 0.66 0.15 0.32 LMR avg Caché app time (in sec.) 40,000 40,000 11,806 LMR transactions (5 min. period) 135,000 30,000 35,000 Database Global Refs / sec. Benchmark full script load Benchmark “paced” script load Production peak (8/21, 11:20 am) Metric

Goals

Simulate current Production user counts & transaction loads

Verify support for load increases up to 3x

Benchmark Environment

Isolated LAN, new DMX3 SAN

20 new Windows blade servers (10 app servers, 10 script ‘players’)

Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps)

2 batch jobs (screensaver simulation, NullGen LMR functions)

Conclusions

Able to simulate production load, 1.5x and 3x load

2 HP rx8640 can handle growth projections

Design and Configuration Considerations Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2 reserved) Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration Maintain same data distribution across 5 DB instances Journal synch bottleneck the biggest issue High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance) Determine 1 ECP channel per instance optimal Additional channels did not improve throughput, still have only 1 Journal Deamon

Database configuration simulation testing

1 to 5 Caché database instances were assessed

1 vs. 5 ECP channels per Caché instance were assessed

Number of active cores were accessed (4 active, 2 reserved)

Results and unexpected discoveries

Identify 5 Caché database instance as optimal design configuration

Maintain same data distribution across 5 DB instances

Journal synch bottleneck the biggest issue

High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance)

Determine 1 ECP channel per instance optimal

Additional channels did not improve throughput, still have only 1 Journal Deamon

Benchmark Discoveries Led to Production Improvements (the rules have changed) References to Undefined globals using $Data and $Get  These commands require network round trip Use of $increment Each call to $I requires network round trip Excessive use of Cache locks Forces more than 1 round trip Use of large strings Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client. Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch.  Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions.

References to Undefined globals using $Data and $Get 

These commands require network round trip

Use of $increment

Each call to $I requires network round trip

Excessive use of Cache locks

Forces more than 1 round trip

Use of large strings

Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client.

Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch.  Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions.

Post UNIX DB migration - 50% Reduction in Outlier Transaction

Monthly Average Caché Web Transaction Time

Unprecedented Growth - The LMR

Phased/Parallel Migration Approach

Parallel Migration Testing and Certification Effort DEV VC/m DEV QA PROD QA auto copy on promote promote check-out & check-in Developer test Analyst cert Production release promote auto copy on check-in Caché Environment 5.0 (WIN) 2008.1 (UNIX)

Application Models Old New Browser client Web server Cache Cache VB client .Net server Cache Cache .Net client Browser client Web server Cache Web Services Browser client .Net client Scalability/Connection pooling, robustness/error handling, Vism Managed Provider Vism.ocx (MServices) Managed Provider Cache Web Services WebLink

Old New

The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare

Add a comment

Related pages

Application Migration - Cisco - Cisco Systems, Inc

... to a large-scale application migration, Cisco IT initiated ... Network and Data Center Services at Cisco, ... user experience before migrating.
Read more

Chapter 2 - Managing the Migration Process

Chapter 2 - Managing the Migration ... team is ASP performance. To ensure good user experiences, ... to users on the network. For a large-scale ...
Read more

Architecting Microsoft SQL Server on VMware vSphere - Best ...

Microsoft Word - Architecting Microsoft SQL Server on VMware vSphere - Best Practices Guide.docx Author: nevenchen Created Date: 3/8/2016 7:52:57 PM ...
Read more

Migrating to Amazon Aurora: The View from the Other Side ...

Migrating to Amazon Aurora: The ... One of the benefits of moving to a higher performance system is ... In operation we now have enterprise scale with ...
Read more

Migrating Enterprise Applications to the Cloud | CloudTimes

Migrating enterprise applications to the cloud ... The virtualization setup is manual and migrating large ... Network performance and ...
Read more

IBM z Systems - Migrate to IBM: Migration path

Migrate to IBM z Systems. Migrating to IBM z ... Mainframes typically run large-scale, ... secure and available high performance platform for ...
Read more

Multiprotocol Label Switching - Wikipedia, the free ...

Multiprotocol Label Switching ... and ATM attractive for deploying large-scale networks. ... proposed to allow high performance traffic forwarding and ...
Read more

The 5 Benefits of Migrating to Cloud - Insight ON

... security and value are impressive advantages of migrating from physical ... Scale and save. Seasonal ... network and data protection software are ...
Read more