Last month, we released the OctoML CLI, a free command line interface to package your deep learning models into Docker containers bundled with NVIDIA Triton Inference Server. OctoML CLI makes it easy and fast to deploy your trained models into production on CPUs or GPUs in any cloud environment. For cost or speed sensitive deployments, you can request a OctoML SaaS platform token to accelerate your models with a variety of acceleration engines, such as Apache TVM, ONNX-Runtime, TensorRT, OpenVINO and more.

If you’re just getting started with OctoML CLI, check out our short 8-min developer demo and operations demo to see OctoML CLI in action. Then explore our end to end examples on GitHub to start building your first intelligent applications with OctoML.

The OctoML CLI is a sophisticated and powerful tool to fast-track your machine learning deployments. As we’ve worked with our early customers, we wanted to share our favorite tips to get the most out of your accelerated ML containers.

Here are ten tips for power users to get even more out of the OctoML CLI:

The OctoML CLI names your Docker images [model name]-[target hardware]:latest (this follows the Docker convention of naming images as repository:tag). For example, if your model is named my_model and you are building and deploying on a GCP Skylake instance (using the optional SaaS acceleration feature), then your built image will be named my_model-skylake:latest.

If you’re building and deploying locally without requesting acceleration, then this will look like my_model-local:latest.

Let’s say you changed your underlying model or some other parameters – which will result in a new image being built – but you kept the same model name in your workspace configuration file. The image that will be built will have the same name as your previous version, which can lead to poor Docker hygiene.

To address this, the OctoML CLI will clean up any old images that have conflicting names. If you’d like to keep those old versions around, the CLI will prompt you for a different tag to use on that old image. Set the environmental variable OCTOML_SKIP_TAGGING=1 to automatically trigger this cleanup.

If you notice in the example above, the old image ID is now tagged with the requested newtag, and the new image ID that was most recently built is tagged with latest.

OctoML CLI caches any downloaded models, packages, and base images for your convenience and to speed up your future OctoML CLI runs. However, if there are naming conflicts with new or updated models, then newly-built packages or images may not always get updated in the cache. If this occurs, one of the ways you can fix this is to clear out the cached items:

The docker rmi command un-tags and removes the image from Docker’s cache, while octoml clean clears the CLI cache.

If you need the image ID, first run docker images to list all images.

For automation or neat tricks these environment variables may be useful for you:

Once you have a container running via octoml deploy (or via octoml package followed by docker run), you may want to check the packaged model in that container. The docker exec commands lets you invoke commands in the running container. By default, the packaged model will be located in the container under octoml/models/.

The -i option enables interactive mode to keep STDIN open even if not attached, and -t allocates a pseudo-TTY.

If you need to find your running container id (which is different than the image id), you can run docker container ls:

A common and useful command to explore from within the container is the tritonserver command, which lets you set dozens of configuration options and settings for Triton Inference Server (note: this command should be run after you’ve already used docker exec to run commands within the container). Run this command to learn more about available configurations:

$ /opt/tritonserver/bin/tritonserver --help

octoml init and octoml setup acceleration will launch interactive wizards that help you set up the project configuration YAML file for acceleration, including submitting your access token, hardware(s) of interest, and specifying inputs for dynamic models if applicable. These wizards will automatically generate well-formed project configuration files on your behalf.

Your project configuration is primarily defined in the octoml.yaml file (which you can find in the same folder where you’ve run the octoml init command). Here, you will define the model you’d like to accelerate and package, as well as any hardware targets, inputs, etc. Read on if you’d like to understand how to write your own config from scratch, which you may need, if you plan on fully automating your model deployment for CI/CD pipelines.

First, a bit about the terms we’re using: a dynamic model is a model where some input dimensions (e.g. batch size, input channels) are not specified until runtime. For a static model, those dimensions are known ahead of time or otherwise specified within the model. An example of a dynamic model is YOLOv3 because you have to specify image size and batch size. ResNet50 is an example of a static model since all input dimensions, including batch size, are predefined.

When you request acceleration on a dynamic model, you are required to define your input dimensions. Input dimensions that need to be defined vary for each model. Examples of input dimensions include batch size, image size, number of channels, number of tokens, vocabulary size, embedding length, sequence length, etc.

You do not need to define input dimensions for static models, unless you want to override the default shapes and dimensions. There is one exception, and that is when you’re packaging a TorchScript model. In this case, you will need to manually specify inputs for both static and dynamic models. In doing so, you need to ensure that all your input shape names are suffixed with __#, where the number represents the order in which you defined your inputs in your PyTorch model and is indexed starting from 0. You will also need to specify your output shapes in the same way. Here is an example of what a configuration file looks like for a TorchScript model that is being packaged without acceleration.

You are welcome to use this as a template to write your own configuration. However, we strongly recommend starting with the setup wizards (i.e. octoml init and octoml setup acceleration) because it means you will not have to worry about any of these formatting details for any models, TorchScript or otherwise. After the setup wizard has created your custom YAML file, you can review it to understand how to generate a similarly formatted file programmatically.

Triton Inference Server’s Trtserver process is the inference server executable and by default it listens for HTTP requests on port 8000 and for gRPC requests on port 8001.

cURL can make HTTP requests to port 8000 and jq is a JSON processing utility.

Get names and versions of all loaded models:

Get configuration for a loaded model, including input shapes:

Get statistics of a loaded model:

Get configuration of the triton server:

You can chain jq and shell commands together to build useful tools. The following is a one line command to get queue time (in nanoseconds) per request for the py_model model.

$ curl -s localhost:8000/v2/models/<your_model_name>/stats | jq '.model_stats[].inference_stats.queue' | jq -c '{queue_time_ns_per_request: (.ns / .count), count: .count }'

Note: to pull out these statistics, you must first run inference against the model. This is needed to populate the counts variable; otherwise, you may get a “cannot be divided by zero” error.

You can even define a shell alias for commonly used tasks:

alias tritonindex='curl -s -X POST localhost:8000/v2/repository/index | jq '.'

Then it’s possible to run it directly:

Sometimes, when developing locally with lots of microservices, or if you are running two instances of Triton, you may end up with port conflicts. It can be useful to disable gRPC and Prometheus in these cases.

Note: in the example below we’re moving HTTP to port 8010, to enable two Triton instances on the same host. This can also be done for the gRPC and metrics servers, but is not shown in the following example.

Add the following arguments to the tritonserver command:

tritonserver --http-port 8010 --allow-grpc false --allow-metrics false --allow-gpu-metrics false

Refer to Tip #5: “Interacting with your model and Triton on the deployed container” for details on executing the tritonserver command inside the deployed container.

Triton is designed to work with Kubernetes. Both images created with the OctoML CLI tool and default images from NVIDIA can be deployed to Kubernetes directly. However if you want to ensure that your Triton instance is running with a GPU, there are some non-standard configuration options you need to set.

Set a node selector (on the pod) to force it on to a specific node known to have a graphics card.

Then modify the pod template spec to add a resource request to request a GPU:

There is a complete example manifest in the Transparent AI repository: https://github.com/octoml/TransparentAI/blob/main/deploy/helm/templates/deployment.yaml#L126