S3 event notifications

When you use your own S3 storage, files can also be added, removed or moved to another storage class directly in the bucket, outside MediaLab. With S3 event notifications, your bucket reports these changes to an Amazon SQS queue, and MediaLab reads that queue to keep your account in sync.

How MediaLab keeps up with your bucket

Each event tells MediaLab which object changed. MediaLab then looks at the object that is in the bucket at that moment and compares it with what it knows. It doesn't rely on the type of event alone, because events can arrive late or out of order, and an overwrite is only reported as an upload. Only the current version of an object counts.

An object counts as different when its size changed, or its checksum changed where both sides know one.

What happens in each case

What happens in your bucketWhat MediaLab does
A new object is uploadedA new file is added, in the folder that matches the object's path. Missing folders are created.
An object is copied to a new keySame as a new upload.
An object is overwritten with different contentThe existing file is moved to the trash, without source. A new file is added for the new content.
An object is overwritten with the same contentNothing changes.
An object is copied onto itself with another storage classThe storage type is updated. See Storage tiering.
An object is deletedFiles that used the object as their source are moved to the trash, without source. Formats stored as that object are no longer available.
An object is deleted and uploaded again before MediaLab processed the deleteThe old file is moved to the trash and the new upload is added as a new file.
A lifecycle rule expires an objectSame as a delete.
A lifecycle rule or Intelligent-Tiering moves an object to another storage classThe storage type is updated, and the file is marked archived (or no longer archived) to match.
MediaLab itself uploads or deletes an objectNothing extra happens: MediaLab already knows about the change.

Buckets with versioning

What happens in your bucketWhat MediaLab does
An object is deleted, which adds a delete markerSame as a delete: the file is moved to the trash, without source.
A lifecycle rule expires the current version, which adds a delete markerSame as a delete.
A new version of an object is uploadedSame as an overwrite: with different content, the existing file goes to the trash and a new file is added.
A delete marker is removed, so the object is backThe object is added as a new file. The file that was moved to the trash earlier stays there, without source.
The current version is permanently deleted, so the previous version is current againHandled as an overwrite: the existing file goes to the trash, and the previous version is added as a new file.
A noncurrent (older) version is permanently deletedNothing changes: the current version is still the same.

MediaLab always uses the current version of an object. A file in the trash doesn't keep a link to an older version of its object.

Good to know

  • Only paths inside a storage container linked to the queue are processed. Changes elsewhere in the bucket are ignored.
  • A file moved to the trash because its object was deleted or replaced can still be found in the trash, but it has no source anymore: its original can't be downloaded or used to make new formats, also not after restoring it. Emptying the trash removes it for good.
  • Locked files are not moved to the trash. They stay in place, without source.
  • If a file's source is also stored in MediaLab's own storage, removing the object from your bucket only removes that cloud copy. The file keeps working.
  • An overwrite with exactly the same size can only be recognized when a checksum is known. That isn't the case for large (multipart) uploads or objects encrypted with your own KMS key, so such an overwrite keeps the existing file.
  • If MediaLab can't reach your bucket while handling an event, nothing changes for that event.

Setting it up in AWS

You need access to the AWS console for the account that owns the bucket.

1. Create the queue

  1. Open Amazon SQS in the same region as your bucket and click Create queue.
  2. Choose type Standard. FIFO queues can't receive S3 event notifications.
  3. Give the queue a name, for example medialab-events.
  4. Under Access policy, choose Advanced and allow your bucket to send messages to the queue:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "s3.amazonaws.com" },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Condition": {
        "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::<bucket-name>" },
        "StringEquals": { "aws:SourceAccount": "<account-id>" }
      }
    }
  ]
}
  1. If you enable encryption with your own KMS key (SSE-KMS), the key policy must also allow s3.amazonaws.com to use the key. Encryption with SQS-managed keys (SSE-SQS) needs nothing extra.
  2. Create the queue.

2. Send the bucket's events to the queue

  1. Open your bucket in Amazon S3 and go to Properties → Event notifications → Create event notification.
  2. Optionally set a prefix to only send events for the path MediaLab uses.
  3. Select these event types:
    • All object create events (s3:ObjectCreated:*)
    • All object removal events (s3:ObjectRemoved:*), which includes delete markers
    • Lifecycle expiration (s3:LifecycleExpiration:*), if a lifecycle rule expires objects
    • Lifecycle transition (s3:LifecycleTransition), if a lifecycle rule moves objects to another storage class
    • Intelligent-Tiering archive (s3:IntelligentTiering), if you use Intelligent-Tiering archive access tiers

    Other event types are ignored.
  4. Under Destination, choose SQS queue and select the queue you created.
  5. Save. S3 sends a test message to the queue to check the setup. MediaLab ignores it.

3. Give MediaLab access to the queue

MediaLab reads the queue with the same credentials it uses for your bucket (the access key, or the role MediaLab assumes). Add this statement to that user's or role's IAM policy:

{
  "Effect": "Allow",
  "Action": [
    "sqs:ReceiveMessage",
    "sqs:DeleteMessage"
  ],
  "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>"
}

MediaLab removes each message from the queue once it has been processed.

Connecting the queue to MediaLab

Send the following to MediaLab's service desk, and say which storage containers the queue is for:

  • the queue name;
  • the region of the queue;
  • the AWS account ID that owns the queue.

Once the queue is connected, changes in the bucket show up in MediaLab shortly after they happen.

Troubleshooting

  • Nothing happens after an upload: check that the object's path is inside a storage container linked to the queue, that the event types above are selected, and that the queue's Monitoring tab shows messages arriving.
  • Messages stay in the queue: MediaLab can't read the queue. Check the IAM permissions from step 3, and that the queue name, region and account ID are correct.
  • Deleted files stay in MediaLab: check that All object removal events is selected, so delete markers in a versioned bucket are reported too, and Lifecycle expiration if a lifecycle rule removes the objects.
  • A file was replaced by a new file: the object at that key was overwritten with different content, or the current version was deleted in a versioned bucket. The previous file is in the trash.