Background and Origin of Service Mesh Network - Istio In enterprise AI architectures, is a choice for and for Kubernetes preferred container-orchestration automating computer application deployment, scaling, and management. Kubernetes with Istio creates a smooth experience of separating traffic flow by compartmentalizing --- The flow of business-related traffic through data plane and The flow of messages and interactions between Istio components to control the mesh behavior. Istio is an first created by Google, IBM, and Lyft, as a collaborative effort. open-source mesh service A is a shared set of names and identities that allows for common policy enforcement and telemetry collection, where Service names and workload principals remain unique. It is an for interacting with each other to reduce the complexity of micro-service management. This is not only limited to only managing connectivity but also privacy, security, and observability of applications in a distributed environment. Service mesh abstraction inter-connected services the is intended to connect application components and thereby the capabilities of the Kubernetes cluster orchestrator, by managing popular micro-services. It is widely known for its ability to simplify enterprise deployment of micro-services by itself transparently onto existing distributed infrastructure. The Devops and developers are hugely benefitted by its flexibilty to add, change, and route them within cloud-native applications. Istio, “service mesh” boost layering In this blog, we describe the importance of Istio in the management of Enterprise-grade API Solutions . Istio’s traffic management and telemetry features help to deploy, serve, and monitor ML models in the cluster. The detailed working components of Istio are out of the scope of this blog, but the links mentioned in the section can be referred. References Motivation - Why Istio The reason behind the origin and development of Service Mesh Networks  can be attributed to the growth and application of different varieties of ML algorithms, (Traditional, Deep Learning - Both Supervised and UnSupervised) for a specific customer/business unit, particularly in production. Further, the growth of has led to the publication of new features, and increased feature collaboration among teams, which in turn facilitated the growth of new micro-services to serve internal purposes as well as for serving customers externally. Feature Stores More specifically Feature Store design by different ML Platform giants (e.g. Microsoft, Gooogle, etc) has revolutionized the phase of micro-service development in Enterprise AI solutions, leading to the growth of robust end-to-end ML pipelines. Source The real challenge with  ML models arises when they are in production with incoming live-traffic. The situation becomes more critical when a host of micro-services need to with and their . deployed managed, monitored, and updated new ML models getting published end-user API interfaces getting released Istio is known for providing a complete solution with insights and operational control over connected services within the “mesh”. Some of the core features of Istio includes: Load balancing on HTTP, gRPC, TCP connections control with Traffic management routing, retry, and failover capabilities. A that includes metrics, tracing, and observability components monitoring infrastructure - Securing communication by encrypting the data flows for communication between services as well as between services and end-users. End to end TLS security Managing in distributed micro-services, when the services can lead to cascading failures. timeouts and communication failures when microservices are triggered by multiple functions or by user actions. Updating simultaneous containers Service meshes help with , allowing organizations to upgrade their services inventory without rewriting applications, adopting microservices or new languages, or moving to the cloud. modernization : Without having Kubernetes applications up and running, service mesh - Istio can be used to measure API usage. API Gateway Architecture The different architectural components of Istio are illustrated in the figure below, where  3 different planes namely the Along with load-balancing, service discovery, and health-checking , Istio provides a policy-driven routing request along with configuration and management of distributed clusters . Data Plane, the Control Plane, and the Management Plane. (in the Data Plane) (Control Plane) (Management Plane) Source Enterprise AI Microservices Any predictive system in an Enterprise AI system is built of a host of micro-services, starting from data pre-processing, modeling, and data post-processing. Such an architecture necessitates individual micro-service deployment and traffic routing. And that's where Istio seats on top of Kubernetes to provide those functionalities. Source Istio Configuration and Deployment Custom resource definitions (CRDs) from downloaded Istio package directory: kubectl apply -f install/Kubernetes/helm/istio/templates/crds.yaml Deployment of Istio operator resources to the cluster from the packaged “all-in-one” manifest: kubectl apply -f install/Kubernetes/istio-demo.yaml Once deployed, we should see the following list of services in the Istio-system namespace: All pods should be in and the above services are available. The below command displays mechanism to enable side-car injection to the default namespace: Running state kubectl label namespace default istio-injection=enabled --overwrite Istio Architecture and Traffic Management Istio provides easy rules and to setup service-level properties like circuit-breakers, timeouts, and retries as well as deployment-level tasks such as A/B testing, canary rollouts, and staged rollouts. traffic routing configurations The most important components of Istio traffic management are and . Pilot Envoy that between all . It applies and them to necessary Envoy side-car proxies. Pilot is the central operator manages service discovery and intelligent traffic routing services conversion rules to high-level routing propagates The is composed of running in the as each service to control all between services, as well as , to provide between . Istio data plane Envoy sidecar proxies same network space network communication Mixer extensible policy evaluation services The is responsible for the used to the and as part of the . The key components of the control plane are Pilot, Citadel, Mixer, and Galley. Istio control plane APIs configure proxies Mixer data plane The below figures illustrates a typical Istio Architecture which depicts the Istio components used to implement the service mesh reference architecture Source Source Envoy is a responsible for controlling all for services in the , which is deployed as a with all within the . Some built-in features of Envoy include: high-performance proxy inbound and outbound traffic mesh side-car container Kubernetes pods mesh Service discovery Load balancing HTTP and gRPC proxies Staged rollouts with %-based traffic split Rich metricsTLS termination, circuit breakers, fault injection, and many more! Istio Egress and Ingress Istio from with to and the of between services. de-couples traffic management infrastructure easy rules configuration manage control flow traffic Source In order for traffic to flow in and out of our “mesh” the following Istio configuration resources, must be setup properly : Load-balancer operating at the edge of the “mesh” handling incoming or outgoing HTTP/TCP connections. Gateway : Configuration to manage traffic routing between Kubernetes services within “mesh”. VirtualService : Policy definitions intended for service after routing. DestinationRule : Additional entry to internal Service Configuration; can be specified for internal or external endpoints. ServiceEntry Configure Egress By default, Istio-enabled applications are not enabled to access URLs outside the cluster. As ML models (traditional, deep learning) are hosted in S3 bucket, what is needed is to setup a to allow for outbound traffic from our ML model Serving Deployment to S3 endpoint, which here in this case is TensorFlow ServiceEntry Create the ServiceEntry rule to allow outbound traffic to S3 endpoint on defined ports: kubectl apply -f algorithm-selector_egress.yaml serviceentry "featurestore-s3" created  (metadata as specified in yaml file) Configure Ingress To allow incoming traffic into our “mesh”, we need to . Our Gateway that will act as a load-balancing proxy by exposing port 31400 to receive traffic: setup an ingress Gateway Use Istio default controller by specifying the label selector istio=ingressgateway so that our ingress gateway Pod will be the one that receives this gateway configuration and ultimately expose the port. Create the Gateway resource we defined above: kubectl apply -f algorithm-selector_gateway.yamlgateway "algorithm-selector-gateway" created Canary Routing With the new model deployment, gradual rolling it out to a subset of the users is easy with Istio. This can be done by updating our VirtualService to route a to v2 subset. Istio allows updating  VirtualService to route 30% of incoming requests to v2 model deployment: small % of traffic kubectl replace -f algo-selector_v2_canary.yaml Replace virtual service and destination rule with "micro-service name" Istio further supports Dashboards to load test and observe the Istio Mesh dashboard for latency metrics across two versions of the algo-selector-serving workloads. Istio further supports Dashboards to load test and observe the Istio Mesh dashboard for latency metrics across two versions of the algo-selector-serving workloads. Source The requests by destination show a similar pattern with traffic split between "microservice-v1" and algo-selector "microservice-v2" deployments. Source V2 Routing Once the canary version satisfies the model behavior and performance thresholds, the deployment can be promoted to be GA to all users. The following VirtualService and DestinationRule is configured to route 100% of traffic to v2 of our model deployment. Source IoT Enterprise AI Solution The following architecture demonstrates Iot based distributed AI solution with a host of micro-services where Istio plays a foremost role due to the following reasons. Rolling out that of and . new micro-services facilitates installation or removal Iot devices applications micro-services due to servicing, rolling out new features or . Updating Iot device recalibration Ensuring and of Iot data Privacy Confidentiality Source Automating Canary Releases Istio traffic routing configuration allows : Perform canary releases by adjusting the relative weighting of traffic between downstream service versions. Such automatic canary deployments are possible using an open-source progressive canary deployment tool by Weaveworks: . programmatically Flagger In Auto Canary Deployment, ( ) automates iterative and along with controls routing features based on custom Flagger Kubernetes operator deployment promotion. Istio App Mesh traffic Prometheus metrics. Source Istio Addons Istio integrate service-level logs with the cloud backend monitoring systems that are usually done for cluster-level . Istio uses persistent storage and querying of Istio metrics.)– the same that are used across different cloud vendor platforms like Azure, AWS, and GCP. plugins logging (e.g., Fluentd, Elasticsearch, Kibana) Prometheus ( metrics collection and alarming utility The objective of Istio Dashboard with a is to provide a mesh service overview along with individual services and workloads overview. Grafana Add-on and Prometheus data source Addition of services to Mesh The below-mentioned code snippets demonstrate the creation of enabled with automatic sidecar injection by adding the istio-injection label to the namespace. These steps allow to identify the namespace that will be used to add services from the application into the mesh: algorithm-selector algorithm-selector Create the algorithm-selector namespace $ kubectl create namespace algorithm-selector Enable the algorithm-selector namespace for Istio automatic sidecar injection $ kubectl label namespace algorithm-selector istio-injection=enabled Validate the namespace is annotated with istio-injection Istio makes it even easier to add services to the mesh by enabling . The most differentiating characteristic feature of Istio is the ability to control which services are added to the mesh to enable . automatic sidecar injection per Kubernetes namespace incremental adoption of services New AI/ML algorithms based on SAAS services, feature-stores, or cloud bases services for caching or database can be rolled out new to sit between existing micro-services. The that Istio allows launching of these services in existing infrastructure gives it an important role in Enterprise AI. flexibility $ kubectl get namespace -L istio-injection Conclusion Service meshes help in the modernization of cloud service management, deployment, and monitoring. In addition, it can provide a to facilitate evolutionary architecture. facade References https://www.oreilly.com/library/view/istio-up-and/9781492043775/ch01.html https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6308717/ https://www.ibm.com/downloads/cas/XWN1WV9Q https://medium.com/better-programming/how-istio-works-behind-the-scenes-on-kubernetes-aeb8003f2cb5#:~:text=Identity%20and%20access%20management%20between,communication%20between%20pods%20is%20encrypted https://towardsdatascience.com/deploy-tensorflow-models-with-istio-on-kubernetes-dd0b2bd3e388 https://istio.io/latest/blog/2018/v1alpha3-routing/ https://www.altoros.com/blog/using-istio-to-unify-microservices-with-a-service-mesh-on-kubernetes/

Discovery

Google

Lyft

Microsoft

Progressive

https://www.linkedin.com/in/sharmistha-chatterjee-7a186310/

2021 - HackerNoon Contributor of the Year - GOOGLE-CLOUD-PLATFORM

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Tips For Using Service Mesh Networks To Leave Your Competition In The Dust

Too Long; Didn't Read

Companies Mentioned

Sharmistha Chatterjee

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Tips For Using Service Mesh Networks To Leave Your Competition In The Dust

Too Long; Didn't Read

Companies Mentioned

Sharmistha Chatterjee

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES