Published on October 31, 2007
A Storage Engine for Amazon S3 How to store a hundred billion BLOBs without buying a single disk.
Who is this guy?
Traditional Storage Engines use the local disk
Network Storage Engines
Federated ODBC HTTP MemCacheD and...
What is S3?
This can't be free
Why do I want to use it?
“An empty disk costs the same as a full one.”
Can I move my existing database over?
Translating S3 to MySQL AWS Account → CREATE SERVER • S3 Bucket → Table • S3 Item → Row • S3 Item Key → VARCHAR PRIMARY KEY • S3 Item Contents → BLOB or VARCHAR •
CREATE SERVER 'MyAWSAcct' FOREIGN DATA WRAPPER 'AWS' OPTIONS (USER 'aws id string', PASSWORD 'aws secret string');
CREATE TABLE 'bierce' ( 'word' VARCHAR(255) NOT NULL PRIMARY KEY, 'defn' BLOB) CHARSET=utf-8 ENGINE=AWSS3 CONNECTION='awss3 DevilDictionary $server MyAWSAcct';
SELECT defn FROM bierce WHERE word='WIT';
SELECT defn FROM bierce WHERE word='WIT'; WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.
INSERT INTO bierce (word, defn) VALUES ('AUTHOR', 'One noted for confusing bitterness with humor.');
DELETE FROM bierce WHERE word='AUTHOR';
So, what's it good for?
“If you build it, they will come.”
Saving EC2 work • SQL CMS for S3 virtual hosting • Huge list of persistent primary keys • Big slow BLOBs, to join against fast local tables. • “The Image Server Problem” • ... and more •
What do you mean, “No temporal guarantees”?!
No WHERE clause means “MAKE MONEY FAST” for Amazon
The Future (watch me wave my hands)
Code Improvement • S3 & HTTP Metadata • Multiple Data Columns • Information Schema • Security & Authentication • Transfer & Storage Compression • Streaming BLOB • More Storage Engines •
S3 & HTTP Metadata
Multiple Data Columns
Sharing a solution with HTTP and MemCacheD Engines
Security & Authentication
Transfer & Storage Compression
(new slide) Streaming BLOBs like PBXT
More storage engines
More storage engines “A Storage Engine for AWS SQS” • There are more AWS services rolling out • Replication distribution via S3 • Cluster persistent storage via S3 • ... and the EC2 holy grail: •
“A Generic Schema Storage Engine for S3”
Back to the present...
“Where do I get it?”
“Where do I get it?” MySQL 5.1 • GPL • http://fallenpegasus.com/code •
“You never write, you never call...”
Suggestions • Use Cases • Bug Reports • Patches • Feature Requests • Support Requests • Gigs & Proposals • email@example.com
Amazon RDS Managed Relational Database Service for MySQL ... 3D Game Engine, ... Amazon S3 offers a range of storage classes designed for ...
Amazon S3 Storage Engine for MySQL. ... I did some quick googling but I couldn't find the plugin, ... I'm the author/presenter for the mysql s3 storage ...
1. A Storage Engine for Amazon S3How to store a hundred billion BLOBs without buying a single disk. 2. Who is this guy?
Plug-in enables MySQL plus Amazon's S3 ... store their data on Amazon.com Inc.'s Web storage service, S3. ... to serve as the MySQL storage engine on ...
The Cloud Storage Engine for MySQL ... that upload files to Amazon S3? WP2Cloud plugin for ... on Amazon cloud storage. WordPress on S3: ...
A MySQL Storage Engine for AWS S3 project is a plugin storage engine that allows MySQL to access Amazon Web Services Simple Storage Service (AWS S3 ...
OblakSoft is pleased to enable the possibility of running WordPress websites on Amazon S3. With the Cloud Storage Engine for ... so to the MySQL plugin.
The InnoDB Storage Engine. ... 17.1 Using MySQL within an Amazon EC2 Instance ... and upload the AMI to the Amazon Simple Storage Service (S3).
Amazon RDS fully supports the InnoDB storage engine for MySQL DB instances. Amazon RDS features such as ... //s3 .amazonaws.com/rds ... MySQL on Amazon RDS ...