In part one of this blog post, we detailed the VPC structuring in a cloud environment, we broke down the infrastructure into three sections: Environment variables that customize the cloud account Module that refers to Terraform resources Infra that binds the environment variables and the module This strategy allows us…

Terragrunt is a thin wrapper that provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing remote state. In this blog post I will show you how you can use Terragrunt to easily structure and configure your VPC environment on top of AWS. This module…

by Lior Schachter & Lidor Ettinger The challenges involved in operationalizing machine learning models are one of the main reasons why many machine learning projects never make it to production. The process involves automating and orchestrating multiple steps which run on heterogeneous infrastructure — different compute environments, data processing platforms…

Follow this step-by-step guide and learn how to autoscale your Airflow K8s pods based on ExternalMetrics — One of the strengths of Kubernetes lies in its ability to achieve effective resources autoscaling. To make scaling decisions, Kubernetes requires rational decision-based on service metrics. In this post we will learn about the scaling decision, and how we can extract, monitor and autoscale Airflow workers based on Datadog external…

The _1 was an example of the container name that docker-compose created. You can either custom it or it will generate one for you then you will have to use it. Therefore, It's better to predefine it. Check out the compose-file-v3.

Thank you for your feedback. Step 2 is necessary in order to set the integration core in DD. You have to define the url of the prometheus_url based on how you expose your service (ccloudexporter). Do you use an orchestration?

At Natural Intelligence we are running a Kubernetes cluster on AWS which makes it is easy to deploy, manage and scale containerized applications. With cloud native becoming the de-facto standard for building and deploying modern applications, and with the proliferation of microservices style application architectures, agile DevOps processes and continuous…

When I came across Apache Kafka and its concept of a streaming platform I asked myself — how will I monitor it? SaaS adoption continues to accelerate as the world is moving to services, data streaming, events, messaging frameworks and the like — and it’s not getting any smaller. The…

At Natural Intelligence, we have plenty of dockerized services that run on kubernetes and swarm cluster on AWS. Each service may require a different set of credentials. We use AWS System Manager (SSM) to store these credentials and to make them accessible in our services. As part of our DevSecOps…