OpenMPI OpenIB

75 %
25 %
Information about OpenMPI OpenIB

Published on January 3, 2008

Author: UpBeat


Open MPI:  Open MPI Overview and Current Status Tim Woodall - LANL Galen Shipman - LANL/UNM Overview:  Overview Point-to-Point Architecture OpenIB Implementation Results Future Work Point-to-Point Architecture:  Point-to-Point Architecture Component Architecture: “Plug-ins” for different capabilities (e.g. different networks) Tunable run-time parameters Three component frameworks: Point-to-point messaging layer (PML) implements MPI semantics Byte Transfer Layer (BTL) abstracts network interfaces Memory Pool (mpool) provides for memory management/registration PML Framework:  PML Framework Single PML manages multiple BTL modules Maintains set of BTLs on a per-peer basis Message fragmentation and scheduling Implements MPI semantics Synchronous / buffered / ready / normal sends Persistent requests / Request completion Eager/Rendezvous protocol Eager send of short messages Configurable threshold (short vs. long) Multiple long protocols PML Protocols:  PML Protocols Send / receive pipeline to / from pre-registered buffers (non-contiguous data) MPI_Alloc_mem support Red/black tree of memory registrations BTL associated with registration is used by scheduler Xfer of contiguous data with 1 RDMA (after match) “Leave pinned” run-time parameter Registration on first-use MRU cache (configurable size) of registrations Bandwidth equivalent to pre-registered buffers (MPI_Alloc_mem) PML Protocols (Continued):  PML Protocols (Continued) Dynamic memory registration/deregistration Fragment message and build pipeline of RDMA requests Overlap [de-]registration with RDMA Bandwidth 97% of pre-registered memory at large message sizes (8Mbytes) Performance impacted by bus type/bandwidth BTL Framework:  BTL Framework MPI agnostic Provides simple API to upper layers Tagged send/receive primitives One-sided put/get operations Access to data type engine for zero copy data transfer BTL modules natively support commodity networks: Current (self, shared memory, myrinet GM/MX, Infiniband mvapi/OpenIB, Portals, TCP) Planned (LAPI, Quadrics Elan4) OpenIB BTL:  OpenIB BTL BTL module initialization Resources allocation Connection management Small message Xfer Large message Xfer OpenIB Issues Future Work BTL module initialization:  BTL module initialization A separate BTL module is initialized for each port on each HCA The PML schedules across these BTL modules just as any other interconnect When multiple BTL modules exist peers establish QP connections by matching subnets Resource Allocation:  Resource Allocation SRQ Scalability :  SRQ Scalability K- multiplier based on number of nodes Connection management:  Connection management Addressing information is exchanged dynamically via an OOB channel This greatly improves scalability but at the cost of increased first message latency Connections are established with peers in the same subnet (local subnet routing only) Small Message Xfer:  Small Message Xfer Maintain list of pre-registered fragments for send and recv List grows dynamically in chunks (more efficient to register) Small messages are copied to/from pre-registered buffers Recv descriptors are posted as needed based on min/max thresholds Small Message Performance:  Small Message Performance * Send/Recv based protocol Large Message Xfer:  Large Message Xfer RDMA Write and RDMA Read are both supported RDMA Read provides better performance than RDMA Write - control messages are reduced RDMA pipeline protocol performance highly dependent on I/O Bus performance Results:  Results OpenMPI/OpenIB - All Results:  Results OpenMPI/OpenIB - All - Log Results:  Results OpenMPI/OpenIB - Eager limit Results:  Results Combined Results Results:  Results Combined Results - Log OpenIB Opportunities:  OpenIB Opportunities User level notification of VM activity Caching of memory registrations can be dangerous Need the ability to detect VM changes that effect memory registrations (such as sbrk and munmap) Reliable Multicast for collectives SRQ performance, 2/10 usec penalty, but who’s counting? Future Work:  Future Work Small message RDMA (using working set of peers) - optional Dynamic connection management using Unreliable Datagrams Dynamic connection teardown - optional Source Code Access:  Source Code Access Subversion repository Download client from: v1.2.1 or later Check out with: svn co ompi Anonymous, read-only access Questions?:  Questions? Tim Woodall Email: Phone: 505-665-5224 Galen Shipman Email: Hardware Specs:  Hardware Specs Dual Intel Xeon 3.2 GHz 1024 KB Cache 2 Gbytes memory Bus: Intel Corp. E7525/E7520/E7320 PCI Express Mellanox Technologies MT25208 InfiniHost III Ex 288 Port Voltaire switch

Add a comment

Related presentations

Related pages

Open MPI: Open Source High Performance Computing

Open MPI is therefore able to combine the expertise, technologies, and resources from all across the High Performance Computing community in order to ...
Read more

Frequently Asked Questions - Open MPI: Open Source High ...

Frequently Asked Questions | Home | Support | FAQ | all just the FAQ About Publications Open MPI Team FAQ Rollup/ALL: General information ...
Read more

Infiniband HOWTO: OpenMPI - Debian OFED Infiniband packaging

6. OpenMPI. The section describes how to configure OpenMPI to use Infiniband. 6.1 Configure IPoIB. OpenMPI uses IPoIB for job startup and tear-down. You ...
Read more

1.140. openib - CentOS Project

1.140. openib. 1.140.1. RHBA-2010:0292: bug fix and enhancement update. ... * when running openmpi jobs, many warnings would present related to udapl.
Read more

Open MPI - HLRS Platforms

Open MPI is an Message Passing Interface (MPI) library project combining technologies and resources from several other projects (FT-MPI, LA-MPI, LAM/MPI ...
Read more

loop_spawn fails with openib BTL · Issue #490 · open-mpi ...

I get a failure in both 1.8 latest and trunk. I think this has been around for months. With 1.8, I get the following: mpirun --mca btl_openib_if_include ...
Read more

1.140. openib

1.140. openib. 1.140.1. RHBA-2010:0292: bug fix and enhancement update. ... * when running openmpi jobs, many warnings would present related to udapl.
Read more

[Rocks-Discuss] OpenMPI and Infiniband - ...

[Rocks-Discuss] OpenMPI and Infiniband ... > > > > > > The next problem I had was when running openmpi with openib support: > >> libibverbs: Warning: ...
Read more

FAQ: Building Open MPI

--with-openib(=DIR) and --with-openib-libdir=DIR--with-mx(=DIR) ... /usr/local/openmpi-1.10.2 on all four machines in some other common location, ...
Read more

5.25. openmpi - CentOS Project

When upgrading openmpi using yum, the following warning may be returned: cannot open `/tmp/openmpi-upgrade-version.*' for reading: ... Prev5.24. openib; Up;
Read more