Published on September 17, 2014
Copyright © 2012 Splunk, Inc. Major Hospital Company
About Founded in 1968 On Fortune’s “Most Admired Healthcare Companies” list Ranked on “The World’s Most Ethical Companies” list Computerworld “Top 100 best places to work” 2
IT&S 2014 3
Connected Devices in Hospitals Improve workflow and allows for efficiency in hospitals Used every day to document patient care activities Requires accurate data transfer to comply with Meaningful Use provisions of HITECH Act Connected devices lead to better healthcare outcomes thru better quality management “The FDA assures devices are accurate – we assure accuracy is maintained through to the patient record” 4
Rolling Out Mobile Vital Signs Monitoring Technology 5000 wireless devices across 165 hospitals 65,000 patients monitored 17,000 nurses using devices 90% Vitals Signs taken electronically The plan: Vitals automatically posted in electronic medical record “The size and implications of this project were larger than anything IT&S had ever attempted.” 5
MDI Key Challenges 6 Troubleshooting to address nurse complaints about devices Changing nurse behavior to ensure effective use of devices Compliance with meaningful use requirements of healthcare act
Why Splunk? (MDI) Need unified view of 5000 devices across 56 servers Fill gap between existing monitoring systems and out of box application monitoring Create consolidated view of application We Needed a Platform 7
Solving the Issue of Broken Devices Widespread complaints of devices failing Variation in complaints across facilities/servers Minimal onsite support Nurses will not troubleshoot devices—devices work or they sit Nurses can be averse to and threatened by new technology Splunk rapidly identified it was user error, not a device issue “Splunk rapidly identified it was user error, not a device issue.” 8
Splunk ad-hoc Search Showed • Only 30% of vitals were being taken via electronic method • Devices were unusable due to inadequate charge • Wireless coverage was inconsistent • Devices responded slowly to loss of signal “A combination of end user behaviors and technical issues were responsible for poor adoption of the technology, threatening a $7 million investment.” 9
Vitals Tracking-The Data Device Logs EHR Reports AD Lookups Application DB Logs alone lack the context to provide meaning to the data “Logs alone lack the context to provide meaning to the data.” 10
Driving Success with Splunk (MDI) • Avoided $3 million incremental Bonus – Using the same data spend on new batteries • Reduces time to resolve and correct issues by viewing all dashboard data at once • Enabled change in nurse behavior with real-time clinician dashboards • Cross-enterprise view of application performance • System problem alerting • Automated service and support records “We were able to very quickly determine that there was nothing wrong with the batteries at all, the problem was simply that they weren’t being charged.” 11
Splunk inflationary epoch! Expanding Splunk to become a major application management resource Expand to other teams (Security, Networking & Infrastructure) Splunk is becoming the cornerstone for management of clinical applications and a trusted single pane of glass for the enterprise “Splunk is our enterprise management platform for clinical informatics.” 12
Health Information Exchange (HIE) HIE is one of the most important aspects of Meaningful Use Stage II At the point we began work on its own HIE, the failure rate of Regional Health Information Organizations (RHIOs – state, regional, and community-based HIE initiatives) was around 30%. After over a year in planning, vendor selection, and build/development; the finished product was ready for testing Testing did not go well… 13
HIE Key Challenges (WE) being the largest healthcare organization in the US, once again managed to overwhelm all aspects of the vendors solution – 6 Million transactional messages per day – Some messages can be in excess of 40,000 lines of text – So many transactions and events that logs were being overwritten every 6-8 hours If something happened to crash the system after 5PM, by the time the testing team got back to the office the next morning; the cause of the system crash had likely been lost forever 14
Enter Splunk (HIE) It took almost a week to comb through 45 servers, all different application subsystems, for logs and to classify those logs Due to the “unique” (unstable) circumstance of the application, deploying universal forwarders to those systems was strictly forbidden – Two servers were commandeered (slightly illegitimately) from a past initiative – Splunk was installed on commandeered servers , actively monitoring and parsing out “junk data” and PHI for over 300 logs via UNC – This was a substantial leap forward, logs were no longer lost! 15
Why Splunk? (HIE) Splunk released the DBX application – Directly connected to the vast number of databases contained within this HIE solution – We now have a truly complete whole system view in real-time This holistic view of HIE data empowered our team to see problems in real-time and begin doing something about it! – Used correlated WMI and database data to compare 4 key system health metrics Incoming Rate Completed Rate FIFO Queue Stuck Messages 17
HIE Data Analysis Tool (DAT) Plagued with integration issues there was an immediate need to get access to a tool that allowed users to: – Research failed message transactions – Investigate slow end-user response times If a clinician can access data from the source system and not access the same data in this tool, adoption will be nonexistent – Without clinician adoption of the HIE solution, tens of millions of dollars will be wasted 18
HIE DAT Key Challenges HIE solution has its own tool built in, making it the default choice for the task. – There is a major flaw with it though, it doesn’t work. No timeline for a fix Without a data analysis tool for constant monitoring of system response times and failed message transactions the reliability of the entire solution was in jeopardy – In depth auditing would be unattainable – Data integrity could not be assured – Meaningful Use attestation would be inconceivable 19
Enter Splunk (HIE DAT) In a world where vendors can’t… Splunk can and does, quickly! – Splunk DB Connect app leveraged to pass real-time database queries to various databases – Uses Vendor supplied Queries This data was made searchable via a series of dashboards, letting the individual doing the research drill down to individual message ID’s with 0 impact to the production databases Development time for this solution, planning to production was 6 hours with a 30 days historical backload – Comparable solution from internal DBA group? 3-6 Months 20
Driving Success with Splunk (HIE) Splunk was paramount to the success of HIE – Splunk empowered rapid root-cause determination – Facilitated application health metric monitoring Alerting of trends that were actionable – Splunk empowered rapid development Reduction in DAT development from a minimum of 3 months to 6 hours – 8,000% Reduction in effort – $23,700 cost avoidance The failure of this application could have led to millions of dollars in lost reimbursements and millions more lost in man hours and development work 21
Payment Card Industry (PCI) Compliance The Payment Card Industry Data Security Standard (PCI DSS) is a set of requirements designed to ensure that ALL companies that process, store or transmit credit card information maintain a secure environment. – Penalty $5,000 to $100,000 per month for PCI compliance violations 22
PCI Compliance Key Challenges RSA enVision solution is expensive – Roughly $200K/year in support costs RSA enVision solution requires hard coded inputs – Will not automatically adjust reporting as new methods are added Requires Vendor PS to make supported modifications to add additional inputs Initial setup took 6 months development 23
Enter Splunk PCI Compliance Clone of RSA tool created in 2 weeks, with the following enhancements: – Completely automated – Real-time data – Designed to be dynamic – Added UI access – Drillable data model that takes end-user context to intelligently filter data 24
Driving Success with Splunk (PCI) Annual cost avoidance of ~$200K Byproducts: – Real-time IVR dashboards tracking revenue – Real-time transaction failure notification – Real-time potential fraud notifications 25
Executive Dashboard Team of 3 using C# building custom solution for over a year – Significant strides – Incomplete Created complete POC solution in Splunk with 24 hours effort 27
Doctor Access Login Time Monitoring Problem: Doc’s login in times to internal systems via Citrix were taking to long and there was no way for IT to monitor and measure the time it takes (Doctor Experience is CIO level issue) Solution: Splunk now measures the exact experience that Doc’s are seeing on their screen – Citrix XenDesktop, Citrix Broker, Juniper, Netscaler, SSO 29
Splunk Growth Plans Licensing: – Increase License by 430% in the next 2 weeks – Increase License by an additional 256% by Jan 2015 Hardware: – Purchase dedicated storage solution for Splunk in excess of 3M IOPS by 1st quarter 2015 (1PB) – Purchase additional 4 double spec’d indexers by 1st quarter 2015 30
Enter SkyNet Coming Q2 2015 31
... to assist job seekers in finding employment with our largest employers. ... in Nashville; Business Studio; Major ... Companies; Map Room; Economic ...
... s leading hospital companies and two major ... Leading U.S. Hospital Companies and Hospital ... LifePoint Hospitals, Inc., based in Nashville, ...
List of companies based in Nashville, Tennessee ... Many significant companies are based in Nashville, ... Lifepoint Hospitals
Search or browse our list of Hospital Management companies in Nashville, Tennessee by category. ... Manta has 17 companies under Hospital Management in ...
This list of major hospitals companies includes the largest ... so if you're thinking of working in the hospitals industry you might want to look to these ...
About Our Company; Share; About Our Company. ... Nashville-based HCA was one of the nation's first hospital companies.
Benesse Oncology Center for Plastic & Cosmetic Surgery Hospital Major Sleep Center Renovo Orthopaedics ... Experience the Major Difference. fb twitter ...
15 Largest For-Profit Hospital Operators . ... Companies are listed in order of the number of ... Hospital Corporation of America (Nashville, ...
Here is a list of Nashville's major companies, businesses ... We've complied a list of some of largest Nashville employers and while ...