Xsede for-nlhpc

50 %
50 %
Information about Xsede for-nlhpc
Technology

Published on March 3, 2014

Author: jtownsil

Source: slideshare.net

Description

International Symposium NLHPC 2013: Innovation at the frontier of HPC

Title: XSEDE: an ecosystem of advanced digital services accelerating scientific discovery

Abstract:
The XSEDE program (Extreme Science and Engineering Discovery Environment) has recently entered its third year of operation. In this talk we will discuss the vision, mission and goals of this project and some of the distinguishing characteristics of the program. This will be accompanied by a review of current status and look ahead at where the program is headed over the next several years.

November 17, 2013 XSEDE: an ecosystem of advanced digital services accelerating scientific discovery John Towns PI and Project Director, XSEDE Director, Collaborative eScience Programs, NCSA jtowns@ncsa.illinois.edu

XSEDE – accelerating scientific discovery XSEDE aspires to be the place to go to access digital research services. Accelerate scientific discovery by enhancing the productivity of researchers, engineers, and scholars through the use of advanced digital services and infrastructure. 2

Motivation for XSEDE: • Scientific advancement across multiple disciplines requires a variety of resources and services • XSEDE is about increased productivity of the community and providing expanded capabilities – leads to more science – is sometimes the difference between a feasible project and an impractical one – lowers barriers to adoption • XSEDE provides a comprehensive eScience infrastructure composed of expertly managed and evolving advanced heterogeneous digital resources and services integrated into a general-purpose infrastructure 3

Innovation: proactively looking to expand scope of capabilities • Striking a balance between providing stable, reliable services and fostering innovation – both in what we are doing and how we do it • Campus Bridging use cases are mostly for capabilities we have not traditionally supported • Novel and Innovative Projects team is seeking out new communities and identifying new capabilities necessary to support them • Architecture design processes explicitly support innovation by the project – more importantly, facilitate innovation by the community Our goal is to deliver new capabilities – and thus new science – faster 4

XSEDE Factoids: high order bits • 5 year, $121M project – plus $9M, 5 year Technology Investigation Service • separate award from NSF – option for additional 5 years of funding upon major review after PY3 • No funding for major hardware – coordination, support and creating a national/international eScience infrastructure – coordinate allocations, training and documentation for >$100M of concurrent project awards from NSF • ~140 FTE (~250 individuals) across 19 partner institutions 5

XSEDE’s Strategic Goals • Deepen and extend the use of the XSEDE ecosystem – deepen use of XSEDE by existing researchers – extend use of XSEDE to new communities – prepare the current and next generation via education, training, and outreach – raise the general awareness of the value of advanced digital services • Advance the XSEDE infrastructure – create an open and evolving infrastructure – enhance the array of technical expertise and support services offered • Sustain the XSEDE infrastructure – sustain a reliable and secure infrastructure – provide excellent user support services – operate an effective and innovative virtual organization 6

What is XSEDE? • An ecosystem of advanced digital services – support a growing portfolio of resources and services • advanced computing, high-end visualization, data analysis, and other resources and services • interoperability with other infrastructures • A virtual organization providing – dynamic distributed infrastructure – support services, and technical expertise to enable researchers engineers and scholars • addressing the most important and challenging problems facing the nation and world • A project funded by the National Science Foundation 7

Total Research Funding Supported by XSEDE in FY2013 US$700 million in research supported by XSEDE in FY2013 8

What do you mean by “Advanced Digital Services?” • Often use the terms “resources” and “services” – these should be interpreted very broadly – most are likely not operated by XSEDE • Examples of resources – compute engines: HPC, HTC (high throughput computing), campus, departmental, research group, project, … – data: simulation output, input files, instrument data, repositories, public databases, private databases, … – instruments: telescopes, beam lines, sensor nets, shake tables, microscopes, … – infrastructure: local networks, wide-area networks, … • Examples of services – – – – – – 9 collaboration: wikis, forums, telepresence, … data: data transport, data management, sharing, curation, provenance, … access/used: authentication, authorization, accounting, … coordination: meta-queuing, … support: helpdesk, consulting, ECSS, training, … And many more: education, outreach, community building, …

XSEDE offers a variety of resources • Leading-edge distributed memory systems • Very large shared memory systems • High throughput systems, including Open Science Grid (OSG) • Visualization engines • Accelerators like GPUs and Xeon PHIs Many scientific problems have components that call for use of more than one architecture. 10

Approach to Other Infrastructures: Active Interactions • OSG is a significant CI in the US – Level 2 Service Provider in XSEDE – the nation’s premier high-throughput computing infrastructure • complement traditional HPC resources inherited from TeraGrid – ties to CI (eScience infrastructure) providers internationally • PRACE is a significant HPC CI in Europe – PRACE represents both large scale HPC and distributed resources • subsumed DEISA in 2011 – joint Summer School series – working on joint call for collaborations support later this calendar year • EGI is a significant HTC CI in Europe – initiating organizational benchmarking effort – identifying collaborating research teams spanning XSEDE-EGI • HPC Wales – Champions programs, Science Gateways – training content 11

Some Unexpected Challenges: XSEDE is a socio-technical ecosystem • Highly distributed organization – challenges in managing a project that involves staff at 19 partner institutions • A completely virtual organization – breaking new ground from an organizational structure and management point of view • Highly distributed engineering project – developing new methodologies to adapt traditional practices to the unusual context of XSEDE 12

XSEDE Software Engineering Processes Need for new capabilities Requirements, Use Cases, Qualities Architecture Design, Documentation A&D, S&SE Focus Architecture Review Bug fixes, routine enhancements documented architecture Detailed Engineering Plans Design, Development, Integration SD&I Focus Integration Test production-ready software Operations enhancements Acceptance Test User Documentation User Experience Production Support User s 13 Operations Focus Pilot s supported software and services (TEOS, ECSS) Service Providers

A Few Highlights from Past Year Transitioning from a project in “start-up” mode to regular delivery of value to a broad community of researchers • Facilitated broad range of ground-breaking research • Added powerful new resources and transitioned users smoothly – integrated/coordinated documentation • Initiated development of undergraduate and graduate certificate and degree programs • Campus Champions reached new heights – 200 Champions at 147 institutions • Delivered new or improved software, services and capabilities on a regular basis – new POPS user interface, new GridFTP version, xdusage utility, new RT ticket system, UNICORE client and server, Globus Online services 14

Current Status: excellent production operations, improving new services • Recent annual review held in June – very strong review from panel – either doing well or on the right track to address issues • Clear objectives of the panel – make sure XSEDE is successful in major review next year – position XSEDE as a leader in the community • beyond the scope of an NSF project – develop sustainability for XSEDE for the long term 15

Objectives for Coming Year+ Accelerating the realization of the XSEDE vision • Deliver new or improved software, services and capabilities on a regular basis – XSEDE Wide Area Filesystem; Global Federated Filesystem; enhanced single sign-on; science gateway APIs; Canonical Use Case components • Campus Bridging will promote "XSEDE Compatible" cluster build tools and use of Globus Online and GFFS for data movement and access • Incorporate the third cadre of under-represented students into the XSEDE Scholars program • Expand Champions Program to include Regional, Student, and Domain Champions • Redesign and implement a new allocations request system • Complete baseline architecture and expanded set of defined Use Cases • Develop joint activities with industry • Further develop relationships with other resource, service and infrastructure providers 16

What keeps me up at night? - #3 Sustainability of funding (how apropos) • XSEDE in year 3 of 5 year project – Can we secure an additional 5 years of funding? • How do we sustain after 10 years? – Can we avoid disruptive nature of re-compete while still deriving best value? – Will NSF even continue to fund such an activity? • Is NSF definition of “sustainable” that they no longer fund it? 17

Call for participation to be announced before SC13!

Questions?

Add a comment

Related presentations