How do I resolve the "failed to initialize logging driver: failed to create Cloudwatch log stream status code: 400" error when I run an AWS Batch job?
Last updated: 2020-01-29
My AWS Batch job failed, and I received the following error message: " CannotStartContainerError: Error response from daemon: failed to initialize logging driver: failed to create CloudWatch log stream: ThrottlingException: Rate exceeded status code: 400." How can I resolve this error?
Short Description
This error means that you reached your API limits on Amazon CloudWatch and that your CreateLogStream API call is throttled by CloudWatch. You can receive this error when you:
- Submit multiple jobs at the same time programmatically, using the AWS Command Line Interface (AWS CLI) or SDKs
- Use AWS Lambda to submit jobs
Resolution
Consider the following solutions:
- Apply a retry strategy to your jobs using a job definition.
- Use retries and an exponential backoff algorithm with Jitter, and then resubmit your job.
- Avoid throttling by using AWS Batch array jobs to submit jobs with a single SubmitJob API call (up to 10,000 copies of a job).
Did this article help you?
Anything we could improve?
Need more help?