roofAI 🏛️: serving — 1

Next, in roofAI 🏛️, researching serving the model.

using https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html

Create a model in SageMaker Inference by pointing to model artifacts stored in Amazon S3 and a container image.

find doc for ⏫ push the roofAI container to ECR

https://github.com/aws/sagemaker-inference-toolkit

https://github.com/aws/amazon-sagemaker-examples/blob/main/advanced_functionality/multi_model_bring_your_own/multi_model_endpoint_bring_your_own.ipynb

Select an inference option. likely “Asynchronous Inference”, digest https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html

Create a SageMaker Inference endpoint configuration … Create a SageMaker Inference endpoint. Invoke your endpoint to receive an inference as a response.

https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html ▶️ https://github.com/aws/amazon-sagemaker-examples/blob/main/serverless-inference/Serverless-Inference-Walkthrough.ipynb ▶️

limiting.

function fun() {
  echo fun,before: x=$x
  local x=$1
  echo fun,after: x=$x

  x=3
  echo fun,later: x=$x
}

x=
echo x=$x

x=2
echo x=$x

echo x=$x

fun 12
echo x=$x

fun 89
echo x=$x

Create, invoke, update, and delete a serverless endpoint

start from https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-prerequisites.html ▶️ item 6 ▶️ Select a prebuilt Docker container image ▶️ https://github.com/aws/deep-learning-containers/blob/master/available_images.md

763104351884.dkr.ecr.us-east-2.amazonaws.com/<repository-name>:<image-tag>

763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:2.1.0-gpu-py310-cu118-ubuntu20.04-ec2

▶️ image url

763104351884.dkr.ecr.us-east-2.amazonaws.com/pytorch-inference:2.1.0-gpu-py310-cu118-ubuntu20.04-ec2

▶️ code

roof inference help

roof inference start

roofai inference start

start the docker,

roof inference help

roof inference pull

@git review

continue from 7,

XGBoost in this notebook: https://github.com/aws/amazon-sagemaker-examples/blob/main/serverless-inference/serverless-model-registry.ipynb is restricted to a specific algo that is supported by the image. Also, the code mixes the image and the classifier.

Can I separate the model from the image? Orthogonality is valuable.

Next: https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-create.html