EnginFrame on AWS

EnginFrame Getting Started on AWS

This CloudFormation template is designed to help you get started with an HPC portal and cluster on AWS in less than one hour. Here are some of the things it does on your behalf:

  1. Create a Virtual Private Cloud (VPC), with security groups and an Application Load Balancer to manage access to your HPC environment from a IP range of your choice
  2. Create an Elastic File System (in the regions that support it) to store your job data and application binaries
  3. Create a Directory Service (in the regions that support it) for user authentication
  4. Configure and run the CfnCluster template to create an elastic cluster with a job scheduler of choice
  5. Install EnginFrame on the head node of the cluster, integrated with all of the above

This automation template and deployment architecture has been sized for clusters of up to 32 nodes and runs in your own AWS Account. You can sign up here, if you don't have one already.

Now choose your preferred AWS Region and enjoy your HPC cluster!

Terms & Conditions

Your use of AWS services are subject to your applicable agreement(s) with AWS, including the AWS Service Terms. Your use of EnginFrame is subject to the EnginFrame end user license agreement, which can be found here (for USA resident) and here (for non-USA resident).

Region
eu-west-1EU (Ireland)Agree and Continue
us-east-1US East (N. Virginia)   Agree and Continue
us-west-2US West (Oregon)Agree and Continue

Details and instructions

  • Before you begin you need an AWS Identity and Access Management (IAM) account with sufficient permissions to log-on in the AWS Web Console and deploy the CloudFormation template. This account must have the right level of permissions to create all the resources, including an EC2 IAM Role. This level of permissions is typically granted to an IAM user with the AdministratorAccess managed policy.
  • These are some of the questions you'll be asked during the installation process:
FieldDescriptionSample values / rules
Access Froma Classless Inter-Domain Routing (CIDR) used to restrict TCP/IP access to EnginFramePublic Internet: 0.0.0.0/0
Private network: 192.168.100.0/24
Specific IP: 54.239.25.208/32
Domain Admin PasswordPassword for the Active Directory administrator user, to be created into AWS Directory ServiceMust be at least 8 characters with at least three of these attributes: lowercase, uppercase, number and symbols
EnginFrame Admin PasswordPassword for the EnginFrame "efadmin" user, also created into AWS Directory ServiceMust be at least 8 characters with at least three of these attributes: lowercase, uppercase, number and symbols
Key Pair NameName of an EC2 KeyPair to enable SSH access to the instances, using the default Linux user ("ec2-user" on Amazon Linux)Key pair can be created in the EC2 Console, and must be valid for the region where the cluster is created
Master Instance TypeThe instance type for the cluster's head node, which also hosts the EnginFrame serverm4.large (for more information, see EC2 Instance types)
Compute Instance TypeThe instance type for the cluster's compute nodesc4.2xlarge (for more information, see EC2 Instance types)
Initial Cluster SizeInitial number of EC2 instances to launch as compute nodes within the cluster2
Maximum Cluster SizeMaximum number of EC2 compute nodes (not including the head node) that can be launched in the cluster16
Maintain Initial Sizewhen it is set to "true", the system maintains the initial cluster size even if there is no job running; otherwise the cluster can shrink to only the head node, when no jobs are running or queuedFalse
Job Schedulerthe resource manager and queuing system to use in the clusteropenlava, sge, torque, slurm

 

Deployment procedure 

  1. Click on the "Agree and Continue" link after reviewing the relevant terms, to land on the first page of the CloudFormation template wizard: "Select Template", and select "Next". If you are not already logged-on in the AWS Web Console, you will be asked to authenticate.
  2. In the "Specify Details" page, you must answer all the questions highlighted in the previous paragraph, and then click "Next"
  3. The third page, "Options", is important only if you do not have the full AdministratorAccess managed policy. Here you can choose an alternate role to continue with the deployment.
  4. The final fourth page "Review" contains a mandatory step: you must check the last checkbox about "Capabilities" to authorize explicitly the deployment of all required resources.
  5. After you click the "Create" button, you land on the main page of CloudFormation with the list of all your stacks. You may need to click the "refresh" button to see the progress of your deployment.
  6. When the stack will reach the status "CREATE_COMPLETE", you can access the section "Outputs" where you can find the web URL to access EnginFrame (using the "efadmin" credentials you provided as a parameter).

How to personalize the template for your needs

If you want to modify the deployment procedure, you can access the CloudFormation template as well as the shell scripts used as pre- and post-install parameters in CfnCluster. Access to these files is available only via AWS API calls, or through the command-line interface (AWS CLI). In the following high-level description we will assume you will use the AWS CLI and CfnCluster CLI.
First, you'll need to obtain a local copy of the template and the companion scripts:

  • aws s3 cp s3://nice-enginframe-eu-west-1/latest/enginframe.template .
  • aws s3 cp s3://nice-enginframe-eu-west-1/latest/enginframe.pre.sh .
  • aws s3 cp s3://nice-enginframe-eu-west-1/latest/enginframe.post.sh .

You can use the CloudFormation template above as a starting point to personalize the infrastructure in which you want to deploy your own clusters, for example changing security group defaults or extending it to support alternative solutions for data storage.
If you want to modify the current cluster configuration, or deploy new clusters inside the same VPC, you can download the CfnCluster configuration file inside "Outputs" tab of the CloudFormation stack. Please follow the guidelines described in the CfnCluster documentation when you perform changes to the CfnCluster configuration files or pre- and post-install scripts (see: http://cfncluster.readthedocs.io/en/latest/ for details).