Cog is an open-source command-line tool from Replicate that packages a machine learning model in a container: a Docker image that holds the model, its code and all its dependencies, so that it runs the same way on any machine with Docker.
A developer describes the environment in a cog.yaml file (GPU or not, system packages, Python version, Python requirements) and writes the model's setup and run code in Python, with typed inputs and outputs. Cog then builds the Docker image with NVIDIA base images and compatible versions of CUDA, cuDNN, PyTorch and TensorFlow; CUDA is NVIDIA's software layer for running code on its GPUs. From the input and output types it generates an OpenAPI schema and an HTTP inference server, so the container answers prediction requests at a /predictions address. The image runs on the developer's own infrastructure or on Replicate. Cog works on macOS, Linux and Windows 11 with WSL 2.
Cog is free and open source under the Apache License 2.0.