Tuan Q. Phan - WESST - Getting Started on the Computational Social Sciences

50 %
50 %
Information about Tuan Q. Phan - WESST - Getting Started on the Computational Social Sciences

Published on August 16, 2017

Author: ALSETNUS

Source: slideshare.net

1. The slides are available at https://www.dropbox.com/s/73eyxqir5wicrsa/mu sangVfinal.pptx?dl=0 For Windows users, you need install Xshell and Xftp at https://www.netsarang.com/download/free_licens e.html For Mac/Linux users, no extra software necessary

2. Getting Started on the Computational Social Sciences Xuesong LU and Tuan Q. PHAN

3. Introduction – Computational Social Science cluster • SoC Strategic Initiative funded by Dean Mohan • “Musang” cluster • Problem statement: • Efficient and managed infrastructure of social scientists • Private cloud and security* • High performance computing • Storage and software/tools • Ideal for medium-large (terabyte) datasets * Admins prevent hacking and security breach. Usage is responsibility of PI and users

4. Application and workflow 1. PI register (NUS affiliate) 1. Yearly subscription model 2. Free for SoC faculty/PI 3. Free for now to non-SoC 2. Assigned a group name: PI_projectname 3. Collaborators register as member of group 4. Access via shared group folder

5. Why Using a Cluster? The tip of NUS email graph Nodes: 5000 users Edges: >10000 email links Plotting time: 2.5 hours

6. Why Using a Cluster? A tip of NUS email graph

7. Why Using a Cluster? Plotting in parallel

8. Why Using a Cluster? Plotting in parallel 0 0.5 1 1.5 2 2.5 3 1 2 3 4 5 6 7 8 9 10 PlottingTime(hour) Plots

9. Overview 1. The Musang cluster 2. Command line shell 3. Using Musang: An illustrated example

10. Overview 1. The Musang cluster 2. Command line shell 3. Using Musang: An illustrated example

11. Musang: A Cluster for Big Data

12. Road Map R, Python, SQL 1 Access Node 4 Computing Nodes 2 HPC Nodes 144 CPU Cores ~200TB Storage Hadoop, Spark A few Access Nodes ~100 Computing Nodes ~10 HPC Nodes ~2000 CPU Cores Several PB Storage

13. The Architecture comp001 comp002 comp003 comp004 hpc01 hpc02 shared storageaccess

14. comp001 comp002 comp003 comp004 The Architecture hpc01 hpc02 shared storage access

15. Accessing Musang Cluster SSH connection: <ssh username@musang01.d1.comp.nus.edu.sg>

16. Accessing Musang Cluster SSH connection: <ssh username@musang01.d1.comp.nus.edu.sg>

17. Accessing Musang Cluster SSH connection: <ssh username@musang01.d1.comp.nus.edu.sg>

18. Accessing Musang Cluster SSH connection: <ssh username@musang01.d1.comp.nus.edu.sg>

19. Overview 1. The Musang cluster 2. Command line shell 3. Using Musang: An illustrated example

20. What is “Shell”? “A UI for access to an operating system's services”

21. Popular Shell Interfaces

22. Xshell Configuration

23. Xshell Configuration

24. Xshell Configuration

25. Xshell Configuration

26. Xshell Configuration

27. Two-Factor Authentication Google Authenticator VIP Access

28. Learning the Shell http://linuxcommand.org

29. Navigation File system organization Your home directory: /NAS/home01/username Your group directory: /NAS/project01/PI_groupname

30. Navigation: pwd Print your working directory

31. Navigation: pwd Print your working directory

32. Navigation: ls List the files in the working directory

33. Navigation: ls List the files in the working directory

34. Navigation: cd Change your working directory

35. Navigation: cd Change your working directory

36. Navigation: cd Change your working directory

37. Navigation: cd Change your working directory

38. Manipulating Files: cp Copy files and directories

39. Manipulating Files: cp Copy files and directories

40. Manipulating Files: cp Copy files and directories

41. Manipulating Files: mv Move or rename files and directories

42. Manipulating Files: mv Move or rename files and directories

43. Manipulating Files: mv Move or rename files and directories

44. Manipulating Files: mv Move or rename files and directories

45. Manipulating Files: rm Remove files and directories

46. Manipulating Files: rm Remove files and directories Be careful with ‘rm’! Once you delete something with ‘rm’, it's gone!

47. Manipulating Files: mkdir Create directories

48. Manipulating Files: mkdir Create directories

49. Revealing the Mystery: which Determine the location of a command

50. Revealing the Mystery: which Determine the location of a command

51. Revealing the Mystery: --help Display a command’s syntax and options

52. Revealing the Mystery: --help Display a command’s syntax and options

53. Revealing the Mystery: man Display a command’s documentation

54. Revealing the Mystery: man Display a command’s documentation

55. Reading Files: cat Read the contents of files

56. Reading Files: cat Read the contents of files

57. Reading Files: head, tail Read the first and last part of files

58. Reading Files: head, tail Read the first and last part of files

59. Filtering: grep Search for matching a string or regular expression

60. Filtering: grep Search for matching a string or regular expression

61. Pipelines: | A sequence of processes chained together

62. Pipelines: | A sequence of processes chained together

63. Editing Files: nano Edit the contents of files

64. Editing Files: nano Edit the contents of files

65. Transfering Files: scp Transfer files between Musang and your laptop

66. Transfering Files: scp Transfer files between Musang and your laptop

67. Transfering Files: scp Transfer files between Musang and your laptop

68. Transfering Files: Using GUI Transfer files between Musang and your laptop

69. Transfering Files: Using GUI Transfer files between Musang and your laptop

70. Summary of Shell Commands Ø Nevigation: pwd, ls, cd Ø Manipulating files: cp, mv, rm, mkdir Ø Command details: which, help, man Ø Reading files: cat, head, tail, grep Ø Pipelines: | Ø Editing files: nano Ø Transfer files: scp, GUI

71. Advanced Shell Commands Ø Executing in background: screen Ø Reading files: less, sort, uniq, awk/gwak Ø Editing files: vi/vim

72. Learning the Shell http://linuxcommand.org You will be an expert of Linux Shell!

73. Overview 1. The Musang cluster 2. Command line shell 3. Using Musang: An illustrated example

74. Module Environment Maintain dynamic user environment

75. Module Environment Maintain dynamic user environment

76. Job Scheduler: Slurm Submit your job with Slurm batch sbatch / squeue / sacct / sinfo https://slurm.schedmd.com/tutorials.html

77. Job Scheduler: Slurm Submit your job with Slurm batch sbatch test_job.sh

78. Notes If you got ‘Permission denied’ error with srun, most probably your script is not executable. To solve the problem, just type ‘chmod 755 your_script’.

79. Getting Started on the Computational Social Sciences Thanks for the attention!

Add a comment