Published on February 28, 2014
What’s new in AWS Philip Fitzsimons, Manager, Solutions Architecture Amazon Web Services UK
What’s new in AWS Amazon Kinesis Managed Service for Real-time Big Data Processing Amazon WorkSpaces Fully Managed Desktops in the Cloud AWS CloudTrail (beta) Get log files of API call made on your AWS account Amazon AppStream Stream resource-intensive applications from the Cloud Amazon RDS for PostgreSQL (beta) Amazon RDS makes it easy to set up, operate, and scale PostgreSQL deployments in the cloud
Amazon Kinesis Managed Service for Real-time Big Data Processing
Introducing Amazon Kinesis • Managed service that can scale without any down time – Create a streaming data intake pipeline and do analysis in near real time – Scale to hundreds of thousands of sensors or producers – Scale to gigabytes to terabytes of throughput per hour • Understand and take action based on data in seconds – Using Amazon Kinesis Client Library on Amazon EC2 instances to process data – Leverage pre-built connectors to store the data in S3, Redshift or DynamoDB • High availability of data streams at low cost – Data ingested in is stored in 3 different physical locations – Pay by the hour with no commitments or up front fees
Why Amazon Kinesis? Internal AWS experiences provided inspiration Big Data Real-time Big Data • CloudWatch metrics: what just went wrong now Weekly / Monthly Bill: What you spent this past billing cycle? • Real-time spending alerts/caps: guaranteeing you can’t overspend • Daily customer-preferences report from your website’s click stream: tells you what deal or ad to try next time • Real-time analysis: tells you what to offer the current customer now • Daily fraud reports: tells you if there was fraud yesterday • Real-time detection: blocks fraudulent use now • Daily business reports: tells me how customers used AWS services yesterday • Fast ETL into Amazon Redshift: how are customers using AWS services now • Hourly server logs: how your systems were misbehaving an hour ago •
Sample Use Cases of Amazon Kinesis Financial Services Leader Digital Advertising Tech. Pioneer Maintain real-time audit trail of every single market/ exchange order Generate real-time metrics, KPIs for online ads performance for advertisers Custom-built solutions operationally complex to manage, & not scalable End-of-day Hadoop based processing pipeline slow, & cumbersome Kinesis enables customer to ingest all market order data reliably, and build real-time auditing applications Kinesis enables customers to move from periodic batch processing to continual, real-time metrics and reports generation Accelerates time to market of elastic, real-time applications – while minimizing operational overhead Generates freshest analytics on advertiser performance to optimize marketing spend, and increases responsive to clients
Clickstream Analytics with Amazon Kinesis Clickstream Archive Aggregate Clickstream Statistics Clickstream Trend Analysis Clickstream Processing App
Simple Metering & Billing with Amazon Kinesis Metering Record Archive Incremental Bill Computation Billing Management Service Billing Auditors
Amazon Kinesis: Key Developer Benefits Easy Administration Managed service for real-time streaming data collection, processing and analysis. Simply create a new stream, set the desired level of capacity, and let the service handle the rest. S3, Redshift, & DynamoDB Integration Reliably collect, process, and transform all of your data in real-time & deliver to AWS data stores of choice, with Connectors for S3, Redshift, and DynamoDB. Real-time Performance High Throughput. Elastic Perform continual processing on streaming big data. Processing latencies fall to a few seconds, compared with the minutes or hours associated with batch processing. Seamlessly scale to match your data throughput rate and volume. You can easily scale up to gigabytes per second. The service will scale up or down based on your operational or business needs. Build Real-time Applications Low Cost Client libraries that enable developers to design and operate real-time streaming data processing applications. Cost-efficient for workloads of any scale. You can get started by provisioning a small stream, and pay low hourly rates only for what you use. 9
Amazon Kinesis Managed Service for Real-Time Processing of Big Data App.1 Data Sources Availability Zone Data Sources Data Sources Availability Zone S3 App.2 AWS Endpoint Data Sources Availability Zone [Aggregate & De-Duplicate] Shard 1 Shard 2 Shard N [Metric Extraction] DynamoDB App.3 [Sliding Window Analysis] Redshift Data Sources App.4 [Machine Learning]
Putting data into Kinesis Managed Service for Ingesting Fast Moving Data • Streams are made of Shards • • Each shard ingests up to 1MB/sec of data and up to 1000 TPS • All data is stored for 24 hours • • A Kinesis stream is composed of multiple Shards You scale Kinesis streams by adding or removing Shards Simple PUT interface to store data in Kinesis • Producers use a PUT call to store data in a stream • A Partition Key is used to distribute the PUTs across Shards • A unique Sequence # is returned to the Producer upon a successful PUT call
Getting data out of Kinesis Client library for fault-tolerant, at least-once, real-time processing • In order to keep up with the stream, your application must: • • Be fault tolerant, to handle failures in hardware or software • • Be distributed, to handle multiple shards Scale up and down as the number of shards increase or decrease Kinesis Client Library (KCL) helps with distributed processing: • • Automatically starts a Kinesis Worker for each shard • Changes number of Kinesis Workers as number of shards changes • • Simplifies reading by abstracting your code from individual shards Uses checkpoints to keep track of a Worker’s location in the stream Use the KCL with Auto Scaling Groups • Auto Scaling policies will restart EC2 instances if they fail • Automatically add EC2 instances when load increases
Amazon Kinesis Resources • Sign up for Limited Preview – http://aws.amazon.com/kinesis/limited-preview/ – Get SDK and Endpoint information after getting into Limited Preview • Getting Started Guide – http://docs.aws.amazon.com/kinesis/latest/dev/getting-started.html • Developer Guide – http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html
Amazon WorkSpaces Fully Managed Desktops in the Cloud
Sample Use Cases (there are many more) • Mobile Device Access • Secure WorkSpaces • Remote Employees • Seasonal Workers • Student WorkSpaces • Developer WorkSpaces
Key Benefits • Fully Managed • Support Multiple Devices • Keep Data Secure and Available • Choose Software & Hardware • Pay as You Go • Corporate Directory Integration
Fully Managed WorkSpaces • Launch the number of WorkSpaces needed • All heavy lifting taken care of by AWS • Users receive email to install clients and connect
Support Multiple Devices • • • • • iPad Kindle Fire HDX (Keyboard & Mouse) Android Tablet Microsoft Windows Mac
Keep Data Secure and Available • No data stored on end-user device • Only Pixels delivered to users (PCoIP) • User volume backed by Amazon S3
Choose Software and Hardware WorkSpaces Bundle Hardware Resources Applications Standard 1 vCPU, 3.75 GiB Memory, 50 GB User Storage Utilities (Adobe Reader, Internet Explorer 9, Firefox, 7-Zip, Adobe Flash, JRE) Standard Plus 1 vCPU, 3.75 GiB Memory, 50 GB User Storage Microsoft Office Professional 2010, Trend Micro Worry-Free Business Security, Utilities (Adobe Reader, Internet Explorer 9, Firefox, 7-Zip, Adobe Flash, JRE) Performance 2 vCPU, 7.5 GiB Memory, 100 GB User Storage Utilities (Adobe Reader, Internet Explorer 9, Firefox, 7-Zip, Adobe Flash, JRE) Performance Plus 2 vCPU, 7.5 GiB Memory, 100 GB User Storage Microsoft Office Professional 2010, Trend Micro Worry-Free Business Security, Utilities (Adobe Reader, Internet Explorer 9, Firefox, 7-Zip, Adobe Flash, JRE) All WorkSpaces Bundles provide the Windows 7 Experience to users (provided by Windows Server 2008 R2 with RDS).
Pay as You Go WorkSpaces Bundle Standard $35 Standard Plus $50 Performance $60 Performance Plus • • • Monthly Price $75 No up-front commitment Delete WorkSpaces at any time Price includes infrastructure (compute, storage, bandwidth) and bundle’s software
Corporate Directory Integration • Users: Get to use existing Enterprise Credentials • IT: WorkSpaces control like regular desktops
Getting Started – What steps do customers take? • • • • • • Integrate with Corporate Active Directory Choose WorkSpaces Bundle Select Users to receive WorkSpaces Launch WorkSpaces Users receive email when provisioned Users connect to WorkSpaces
Availability • Now: Limited Preview • Q1 2014: Public Beta (Initially US-West & US-East) • Q1/Q2 2014: Region Expansion
Resources aws.amazon.com/WorkSpaces • • • • • Detail Page Pricing Limited Preview Sign-Up Documentation will follow at Public Beta Contact firstname.lastname@example.org – Prioritizing customers in Limited Preview, specific briefings
AWS CloudTrail (beta) AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you. The recorded information includes: • • • • • The identity of the API caller The time of the API call The source IP address of the API caller The request parameters The response elements returned by the AWS service
AWS CloudTrail Use Cases • Security Analysis – You can use the AWS API call history produced by CloudTrail as an input into log management and analysis solutions to perform security analysis and to detect user behaviour patterns. • Track Changes to AWS Resources – You can use the AWS API call history produced by CloudTrail to track changes to AWS resources, including creation, modification, and deletion of AWS resources such as Amazon EC2 instances, Amazon VPC security groups and Amazon EBS • Troubleshoot Operational Issues – You can use the AWS API call history produced by CloudTrail to troubleshoot operational issues. For example, you can quickly identify the most recent changes made to resources in your environment. • Compliance Aid – CloudTrail makes it easier to ensure compliance with internal policies and regulatory standards by providing AWS API call history. Integrates with AWS Partner solutions like Alert Logic
Features and Benefits • Increased Visibility – CloudTrail provides increased visibility into your user activity by recording AWS API calls. – You can answer questions such as, what actions did a given user take over a given time period? For a given resource, which user has taken actions on it over a given time period? What is the source IP address of a given activity? Which activities failed due to inadequate permissions?
Features and Benefits • Durable and Inexpensive Log File Storage – CloudTrail uses Amazon S3 for log file storage and delivery, so log files are stored durably and inexpensively. – You can use Amazon S3 lifecycle configuration rules to further reduce storage costs. For example, you can define rules to automatically delete old log files or archive them to Amazon Glacier for additional savings.
Features and Benefits • Easy Administration – CloudTrail is a fully managed service. – You simply turn on CloudTrail for your account using the AWS Management Console, the Command Line Interface, or the CloudTrail SDK and start receiving CloudTrail log files in the Amazon Simple Storage Service (Amazon S3) bucket that you specify.
Features and Benefits • Reliability – CloudTrail continuously transports events from AWS services using a highly available and fault tolerant processing pipeline. – Turning on CloudTrail has no impact on performance of your AWS resources or API call latency.
Features and Benefits • Timely Delivery & Notification – CloudTrail typically delivers events within 15 minutes of the API call and can be configured to publish a notification for each log file delivered. – This feature enables you to automatically take action upon log file delivery. CloudTrail uses the Amazon Simple Notification Service (SNS) for notifications.
Features and Benefits • Log File Aggregation – CloudTrail can be configured to aggregate log files across multiple accounts and regions. – If you use multiple AWS regions, you can choose where log files are delivered for each region. For example, you can have a separate Amazon S3 bucket for each region, or you can aggregate log files from all regions in a single S3 bucket.
Features and Benefits • Choice of Partner Solutions – Multiple partners are available including AlertLogic, Boundary, Loggly, Splunk and Sumologic. – These partners offer integrated solutions to analyze CloudTrail log files. These solutions include features like change tracking, troubleshooting, and security analysis.
Getting Started CloudTrail can be turned on in as few as two clicks from the AWS Management Console. CloudTrail generates log files containing detailed information about API calls made, and periodically saves these files into an Amazon S3 bucket of your choosing. You can also choose to create an SNS topic to receive a notification every time a new log file is delivered. • • • To turn on CloudTrail, just provide a name for an Amazon S3 bucket where you want your log files delivered. If you use multiple AWS regions, you can choose where log files are delivered for each region. For example, you can have a separate Amazon S3 bucket for each region, or you can aggregate log files from all regions in a single S3 bucket. There is no additional charge for CloudTrail, but standard rates for Amazon S3 and Amazon SNS usage apply.
Amazon AppStream Stream resource-intensive applications from the Cloud
Sample Use Cases • • • • Games Media and Entertainment Simulation Software 3D Graphics Development
Key Benefits • Remove Device • Instant On Constraints • Improved Security • Multi Platform Support • Automatic Scaling • Easy Updates
Remove Device Constraints • No longer a need to snap to the device capabilities • AppStream helps reach broadest audience • Users get rich experience across devices
Multi Platform Support • • • • • Kindle Fire Android iOS Windows 7 Mac OS (2014)
Easy Updates • Provide a new application version to AppStream • Immediately upgrade all users • No action on users’ part
Instant On • Users can start using application immediately • No large file downloads • No time consuming installations
Improved Security • Create Entitlement Service to authorize connections • Clients connect to Entitlement Service for access • Your application is secured in AWS Cloud
Automatic Scaling • Set the limits for scaling of your application • The AppStream service takes care of scaling • AppStream uses the G2.2XLarge Instance – Support for more instance types coming in 2014 • Focus on your application, not infrastructure
Getting Started – What steps do customers take? • • • • Integrate AppStream SDK with application Deploy Application to AppStream Create Entitlement Service Create Client Apps
Availability • Now: Limited Preview • Public Beta Date: TBD
Amazon RDS for PostgreSQL Amazon RDS makes it easy to set up, operate, and scale PostgreSQL deployments in the cloud. With Amazon RDS, you can deploy scalable PostgreSQL deployments in minutes with costefficient and resizable hardware capacity.
What does RDS for PostgreSQL enable? • • • • Large-scale web applications Internal and departmental applications Excellent vehicle for ETL into analytics engines Geospatial and mobile applications
Benefits of RDS for PostgreSQL • Easy, managed deployments – Free up time from undifferentiated admin tasks – Database instances with pre-configured parameters, automated patching, monitoring, and notifications • Familiar environment for developers – All your existing PostgreSQL apps work • Fast, predictable performance – Use Provisioned IOPS to tune on the fly • Backup and recovery – Automated backups and database snapshots • High availability – Deploy in multiple Availability Zones Available in all regions
Benefits of PostgreSQL for AWS Customers • The preferred open-source database for many enterprise developers and startups • Support for geospatial queries using the PostGIS extensions • Support for full-text search • Support for advanced data types: JSON and key/value stores (“hstore”)
Getting started with Amazon RDS for PostgreSQL • Review the Getting Started Guide for RDS http://docs.aws.amazon.com/gettingstarted/lates t/awsgsg-intro/gsg-aws-intro.html • Review the RDS Free Tier http://aws.amazon.com/free • Create a PostgreSQL DB instance from the RDS console https://console.aws.amazon.com/rds/
Resources for Amazon RDS for PostgreSQL • User Guide http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/ • Data Import Guide http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/P ostgreSQL.Procedural.Importing.html • Common tasks, including setting up PostGIS geospatial extensions http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/A ppendix.PostgreSQL.CommonDBATasks.html • RDS forum https://forums.aws.amazon.com/forum.jspa?forumID=60
Details for Developers • Launching with PostgreSQL 9.3.1 • Wide selection of available instances – Including the new high-memory db.cr1.8xlarge • Choose Multi-AZ deployment for high availability – Synchronous replication to a secondary in a different AZ • Use Provisioned IOPS for predictable performance – Convert to IOPS with a brief availability impact – Then add IOPS or storage on the fly
What’s new in AWS Amazon Kinesis Managed Service for Real-time Big Data Processing Amazon WorkSpaces Fully Managed Desktops in the Cloud AWS CloudTrail (beta) Get log files of API call made on your AWS account Amazon AppStream Stream resource-intensive applications from the Cloud Amazon RDS for PostgreSQL (beta) Amazon RDS makes it easy to set up, operate, and scale PostgreSQL deployments in the cloud Talk to someone from AWS or our partners at Stand 1070
Amazon Web Services is Hiring. Amazon Web Services (AWS) is a dynamic, growing business unit within Amazon.com. We are currently hiring Software ...
You can now allow AWS CloudFormation to assume service roles which determine what CloudFormation is allowed to do with your stack. A service role is an AWS ...
The following release notes include information for the EMR 5.0.0 release. Changes are relative to the EMR 4.7.2 release. Please see Previous Releases for ...
What’s New at AWS? …looking at just a few new things for Enterprise. Speed of Innovation * As of 30 April 2016 2009 48 280 722 82 ... https://aws ...
Curious about AWS Mobile Services and latest updates? Attend this session for a deep dive on recent updates to AWS Mobile Services aimed at helping
Agenda: Introduction by AWS Usergroup of Delhi NCR Members (10 min) What’s new @ AWS (30 min) “Going Serverless with AWS Lambda” >> by Abhishek ...
View changes: 2016-10-01 06:11 87 new words, 119 deleted words, 3% change ... AWS Date Announcement Sep 30 Amazon Elastic Transcoder ... in time stamps ...
About AWS. Advancing the ... What's Trending. AWS Member Network Read More > Upcoming Events ... Here at the American Welding Society, ...