If you want to run artificial intelligence (AI) and machine learning applications such as large language models (LLMs) at scale, you must run them on Kubernetes. However, mastering Kubernetes — everyone’s favorite container orchestrator — isn’t easy. That’s where Kubernetes Al toolchain operator — the latest addition to Microsoft’s Azure Kubernetes Service (AKS) — comes in.
Also: Microsoft Azure introduces Radius open-source development platform
AKS already makes Kubernetes on Azure easier. Instead of working it out by hand, AKS’s built-in code-to-cloud pipelines and guardrails give you a faster way to start developing and deploying cloud-native apps in Azure. With its unified management and governance for on-premises, edge, and multi-cloud Kubernetes clusters, AKS also makes it simpler (there’s no such thing as “simple” when it comes to Kubernetes) to integrate with Azure security, identity, cost management, and migration services.
What Kubernetes AI toolchain operator brings to the table are automated ways to run open-source software AI/ML workloads cost-effectively and with less manual configuration. It also automates LLM model deployment on AKS across available CPU and GPU resources by selecting the optimally sized infrastructure for your LLM or other project.
AI toolchain operator does this by automatically provisioning the necessary GPU nodes and setting up the associated inference server as an endpoint server to your AI models. An inference server, such as Hugging Face’s 7B or NVIDIA Triton Inference Server, applies trained AI models to incoming data to make real-time decisions. Inference is the process of running live data through a trained AI model to make a prediction or solve a task. Using this add-on reduces your onboarding time and enables you to focus on AI model usage and development rather than infrastructure setup.
It also makes it possible to easily split inferencing across multiple lower-GPU-count virtual machines (VMs). This means you can run your LLMs on more Azure regions, thus eliminating wait times for Azure regions with higher GPU-count VMs and lowering overall cost. In other words, you can automatically run your LLMs on lower-power, less-expensive regions. Yes, you may lose processing power, but not all jobs require higher horsepower.
Also: I went hands-on with Microsoft’s new AI features,…