AWS S3 eventual consistency
AWS S3 object consistency model includes:
Read-after-write
Read-after-update
Read-after-delete
Each of them describes how an object reaches its consistent state after an object is created, updated, or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though NetBackup for OpenStack employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reaches its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach the consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest using these regions when you create s3 buckets for NetBackup for OpenStack. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to be restarted.