HPSS IU slides

67 %
33 %
Information about HPSS IU slides

Published on November 21, 2007

Author: Laurence

Source: authorstream.com

IU's TeraGrid-accessible HPSS archival storage system:  IU's TeraGrid-accessible HPSS archival storage system Overview about HPSS Feeds and speeds, and particulars of IU's distributed HPSS environment Moving data to and from IU's HPSS via the TeraGrid and gridftp Hands on exercises with gridftp and HPSS HPSS Background:  HPSS Background IU installed HPSS system in 1998 HPSS is distributed between Indianapolis and Bloomington campuses. Over 1.8 petabytes of storage available and can scale to 4.2 petabytes In aggregate, HPSS can handle as much as 2GB/s of data flowing in or out. HPSS Design:  HPSS Design IUB Subsystem IUPUI Subsystem Research Network Research Network FC SAN FC SAN HPSS Core Servers HPSS Performance:  HPSS Performance 100MB/sec single thread over gigabit IU HPSS capable of >1GB/sec in parallel over multiple 10 gigabit 150TB of DDN disk for cache 40 IBM Jaguar drives - 100MB/sec transfers Storing Data:  Minimum file size should be one MB Prefer data to be at 10MB or larger Small files should be aggregated Maximum file size is 10TB By default, data is stored on tape at each campus Storing Data Class of Service:  Class of Service Each file has class of service (COS) COS defines maximum file size number of copies stripe width With GridFTP, COS is chosen automatically by file size User can override by appending ,,<COS> to the filename COS Listing:  COS Listing COS 1 - default; max file size 10GB COS 2 - max file size 40GB COS 3 - max file size 10TB COS 4 - max file size 10TB 16 way to disk 4 way to tape Single copy COS available Quotas:  Quotas Default is 1TB Can request up to 10TB Double these numbers if single-copy COS used Accessing MDSS:  Accessing MDSS Data goes to disk cache first Data copied to tape ASAP Copy sent to other campus by default Data purged from disk cache as space needed Data staged back to disk cache as needed Accessing MDSS:  Accessing MDSS Fastest Methods hsi (hsi.mdss.iu.edu) gridftp (gridftp.mdss.iu.edu) pftp_client (ftp.mdss.iu.edu) kerberized ftp (ftp.mdss.iu.edu) Accessing MDSS:  Accessing MDSS Convenient Methods sftp (sftp.mdss.iu.edu) https (www.mdss.iu.edu) Samba (smb.mdss.iu.eu) Hpssfs hsi example:  hsi example intrigue:~ seiffert$ hsi Principal: seiffert [seiffert]Password: Username: seiffert UID: 1529 CC: 1529 Copies: 1 [hsi.3.3 Wed Jan 17 11:31:07 EST 2007] /hpss/s/e/seiffert->put myfile.tar put ’myfile.tar' : '/hpss/s/e/seiffert/myfile.tar’\ ( 4051006 bytes, 4198.5 KBS (cos=1)) /hpss/s/e/seiffert->get myfile.tar get '/Users/seiffert/myfile.tar' :\ '/hpss/s/e/seiffert/myfile.tar' (2007/01/14 12:09:36\ 4051006 bytes, 8275.6 KBS ) /hpss/s/e/seiffert-> sftp example:  sftp example intrigue:~ seiffert$ sftp sftp.mdss.iu.edu Connecting to sftp.mdss.iu.edu... seiffert@sftp.mdss.iu.edu's password: sftp> put welcome.wmv Uploading welcome.wmv to /ssh\-chroot/cos1/hpss/s/e/seiffert/welcome.wmv welcome.wmv 100% 3956KB 3.9MB/s 00:01 sftp> get welcome.wmv Fetching /ssh-chroot/cos1/hpss/s/e/seiffert/welcome.wmv to welcome.wmv /ssh-chroot/cos1/hpss/s/e/seiffert/welcome.wmv \ 100% 3956KB 3.9MB/s 00:00 IU HPSS from TeraGrid:  IU HPSS from TeraGrid Accessible by either: gridftp.archive.iu.teragrid.org gridftp.hpss.iu.teragrid.org Need to add TeraGrid certificate to IU Big Red System uberftp and globus-url-copy tested and supported tgcp is not tested or supported globus-url-copy Example:  globus-url-copy Example Simple example to and from globus-url-copy -rp gsiftp://my.local.site/myfile.tar \ gsiftp://gridftp.archive.iu.teragrid.org/myfile.tar globus-url-copy -rp \ gsiftp://gridftp.archive.iu.teragrid.org/myfile.tar \ gsiftp://my.local.site/myfile.tar Parallel transfer to and from globus-url-copy -rp -stripe -bs 8388608 \ gsiftp://my.local.site/bigfile.tar \ gsiftp://gridftp.archive.iu.teragrid.org:2814/bigfile.tar,,4 globus-url-copy -rp -stripe \ gsiftp://gridftp.archive.iu.teragrid.org:2814/bigfile.tar \ gsiftp://my.local.site/ uberftp example:  uberftp example Simple Example uberftp> put myfile.tar myfile.tar uberftp> get myfile.tar Parallel Example uberftp> open -P 2814 gridftp.archive.iu.teragrid.org uberftp> lopen my.local.site uberftp> parallel 4 uberftp> blksize 8388608 uberftp> put bigfile.tar bigfile.tar,,4

Add a comment

Related presentations

Related pages

IU's TeraGridaccessible HPSS archival storage system ...

... (sftp.mdss.iu.edu) https (www.mdss.iu.edu) ... (displayed as a Flash slide show) on PowerShow.com ... IU's TeraGridaccessible HPSS archival storage ...
Read more

High Performance Storage System Overview

IU Big Red LBNL NERSC (top 500 #7) LLNL ASC Purple NCEP Power 575 ... Copyright © IBM 2009 Slide Number 10 HPSS tape architecture HPSS ...
Read more

Implementation and experience with Big Red (a 30.7 TFLOPS ...

Implementation and experience with Big Red (a 30.7 TFLOPS IBM BladeCenter cluster), the Data Capacitor, and HPSS (UITS Brownbag)
Read more

PPT – The N'I' HPSS Complaints System PowerPoint ...

Chart and Diagram Slides for PowerPoint - Beautifully designed chart and diagram s for ... IU's TeraGridaccessible HPSS archival storage system ...
Read more

Presentation Title - ASIS

Indiana University Data Publishing Service Stacy Kowalczyk April 9, 2010
Read more

Presentation "Overview of IU activities in supercomputing ...

Overview of IU activities in supercomputing, grids, and computational biology Dr. Craig A. Stewart stewart@iu.edu Director, Research and Academic Computing,
Read more

Presentation "Data Publishing Service Indiana University ...

Slide 4: Current Data ... derived data creation creation of data documentation End of research research outputs preparing data for preservation IU ...
Read more

PPT - IU TeraGrid Gateway Support PowerPoint Presentation

IU TeraGrid Gateway Support. Marlon Pierce Community Grids Lab Indiana University. Personnel. Marlon Pierce: project leader Sangmi Lee Pallickara ...
Read more