Amazon SageMaker Model Deployment
Easily deploy and manage machine learning (ML) models for inference
Amazon SageMaker makes it easy to deploy ML models including foundation models (FMs) to make predictions (also known as inference) at the best price-performance for any use case. From low latency (a few milliseconds) and high throughput (millions of transactions per second) to long-running inference for use cases such as natural language processing and computer vision, you can use SageMaker for all their inference needs. SageMaker is a fully managed service and integrates with MLOps tools, so you can scale your model deployment, reduce inference costs, manage models more effectively in production, and reduce operational burden.
How it works
Wide range of options for every use case
Broad range of inference options
From low latency (a few milliseconds) and high throughput (hundreds of thousands of requests per second) to long-running inference for use cases such as natural language processing and computer vision, you can use Amazon SageMaker for all your inference needs.
Real-Time Inference
Low latency and ultra-high throughput for use cases with steady traffic patterns.
Serverless Inference
Low latency and high throughput for use cases with intermittent traffic patterns.
Asynchronous Inference
Low latency for use cases with large payloads (up to 1 GB) or long processing times (up to 15 minutes).
Flexible deployment endpoint options
Amazon SageMaker provides scalable and cost-effective ways to deploy large numbers of ML models. With SageMaker’s multi-model endpoints and multi-container endpoints, you can deploy thousands of models on shared infrastructure, improving cost-effectiveness while providing the flexibility to use models as often as you need them. Multi-model endpoints support both CPU and GPU instance types, allowing you to reduce inference cost by up to 90%.
Single-model endpoints
One model on a container hosted on dedicated instances or serverless for low latency and high throughput.
Multi-model endpoints
Multiple models sharing a single container hosted on shared infrastructure for cost-effectiveness. This reduces hosting costs by improving endpoint utilization compared with using single-model endpoints.
Multi-container endpoints
Multiple containers sharing dedicated instances for models that use different frameworks.
Serial inference pipelines
Multiple containers sharing dedicated instances and executing in a sequence. You can use an inference pipeline to combine preprocessing, predictions, and post-processing data science tasks.
Support for most machine learning frameworks and model servers
Amazon SageMaker inference supports built-in algorithms and prebuilt Docker images for some of the most common machine learning frameworks such as TensorFlow, PyTorch, ONNX, and XGBoost. If none of the pre-built Docker images serve your needs, you can build your own container for use with CPU backed multi-model endpoints. SageMaker inference supports most popular model servers such as TensorFlow Serving, TorchServe, NVIDIA Triton, AWS multi-model server. You can also support large models with DJL serving using DeepSpeed and Hugging Face Accelerate model parallelization frameworks. With these options, you can deploy models including foundation models (FMs) quickly for virtually any use case.
Achieve high inference performance at low cost
Deploy models on the most high-performing infrastructure or go serverless
Amazon SageMaker offers more than 70 instance types with varying levels of compute and memory, including Amazon EC2 Inf1 instances based on AWS Inferentia, high-performance ML inference chips designed and built by AWS, and GPU instances such as Amazon EC2 G4dn. Or, choose Amazon SageMaker Serverless Inference to easily scale to thousands of models per endpoint, millions of transactions per second (TPS) throughput, and sub10 millisecond overhead latencies.
Shadow test to validate performance of ML models
Amazon SageMaker helps you evaluate a new model by shadow testing its performance against the currently deployed model using live inference requests. Shadow testing can help you catch potential configuration errors and performance issues before they impact end users. With SageMaker, you don’t need to invest weeks of time building your own shadow testing infrastructure, meaning you can release models to production faster. Just select a production model that you want to test against, and SageMaker automatically deploys the new model in shadow mode and routes a copy of the inference requests received by the production model to the new model in real time. It then creates a live dashboard that shows performance metrics, such as latency and error rate, of the new model and the production model in a side-by-side comparison. Once you have reviewed the test results and validated the model, you can promote it to production.
Automatic inference instance selection and load testing
Amazon SageMaker Inference Recommender helps you choose the best available compute instance and configuration to deploy machine learning models for optimal inference performance and cost. SageMaker Inference Recommender automatically selects the compute instance type, instance count, container parameters, and model optimizations for inference to maximize performance and minimize cost.
Autoscaling for elasticity
You can use scaling policies to automatically scale the underlying compute resources to accommodate fluctuations in inference requests. With autoscaling, you can shut down instances when there is no usage to prevent idle capacity and reduce inference cost.
Reduce operational burden and accelerate time to value
Fully managed model hosting and management
As a fully managed service, Amazon SageMaker takes care of setting up and managing instances, software version compatibilities, and patching versions. It also provides built-in metrics and logs for endpoints that you can use to monitor and receive alerts.
Built-in integration with MLOps features
Amazon SageMaker model deployment features are natively integrated with MLOps capabilities, including SageMaker Pipelines (workflow automation and orchestration), SageMaker Projects (CI/CD for ML), SageMaker Feature Store (feature management), SageMaker Model Registry (model and artifact catalog to track lineage and support automated approval workflows), SageMaker Clarify (bias detection), and SageMaker Model Monitor (model and concept drift detection). As a result, whether you deploy one model or tens of thousands, SageMaker helps off-load the operational overhead of deploying, scaling, and managing ML models while getting them to production faster.
Customer success
AT&T Cybersecurity improved threat detection requiring near-real-time predictions using Amazon SageMaker multi-model endpoints. “Amazon SageMaker multi-model endpoints are not only cost effective, but they also give us a nice little performance boost from simplification of how we store our models.”
Matthew Schneid Chief Architect - AT&T
Forethought Technologies, a provider of generative AI solutions for customer service, reduced costs by up to 80 percent using Amazon SageMaker. “By migrating to Amazon SageMaker multi-model endpoints, we reduced our costs by up to 66% while providing better latency and better response times for customers.”
Jad Chamoun, Director of Core Engineering - Forethought Technologies
Bazaarvoice reduced ML inference costs by 82% using SageMaker Serverless Inference. “By using SageMaker Serverless Inference, we can do ML efficiently at scale, quickly getting out a lot of models at a reasonable cost and with low operational overhead.”
Lou Kratz, Principal Research Engineer – Bazaarvoice
Tapjoy uses Amazon SageMaker to deploy ML models in days instead of months. “We've gone from taking about three to six months to train, build, and deploy a model. Now with SageMaker, we can do that within one week, perhaps even shorter.”
Nick Reffitt, Vice President of Data Science and Engineering - Tapjoy
Amazon SageMaker and Hugging Face have been collaborating to simplify and accelerate adoption of transformer models with Hugging Face DLCs, integration with SageMaker Training Compiler, and SageMaker distributed libraries.
Zendesk hosted thousands of ML models in Amazon SageMaker multi-modal endpoints (MME) for their Suggested Macros feature and achieved 90% cost savings on inference when compared to dedicated endpoints. “We deployed thousands of ML models, customized for our 100K+ customers, using Amazon SageMaker multi-model endpoints (MME). With SageMaker MME, we built a multi-tenant, SaaS friendly inference capability to host multiple models per endpoint, reducing inference cost by 90% compared to dedicated endpoints.”
Chris Hausler, Head of AI/ML – Zendesk
Resources
AWS re:Invent 2022 - Deploy ML models for inference at high performance & low cost, ft AT&T (AIM302)
Follow this step-by-step tutorial to deploy a model for inference using Amazon SageMaker.
In this hands-on lab, learn how to use Amazon SageMaker to build, train, and deploy an ML model.
Get started building with Amazon SageMaker in the AWS Management Console.