DISCLAIMER: This project was developed for educational purposes only and is not complete, nor supported. It's publishment is only intended for helping others automate environments for delivering workshops with Keptn & Dynatrace. Even though the exposed endpoints of this cluster have valid SSL certificates generated with Cert-Manager and Let's Encrypt, does not mean the Box is secure.
🚨 ALERT: This install uses keptn 0.8.7 🚨
Keptn-In-A-Box is part of the automation for delivering Autonomous Cloud Workshops with Dynatrace. This is not a tutorial but more an explanation of what the shell file set up for you on a plain Ubuntu image.
A simple Bash script will set-up a fully functional Single Node Kubernetes Cluster with Dynatrace installed and Kubernetes Cluster, Cloud Applications and Events monitoring enabled. This script is used as userdata when spinning up elastic compute images (ec2) in amazon web services, but it can run also manually in a Linux machine or VM with snap installed. The tested distro is Ubuntu Server 18.04 LTS & 20.04 LTS For spinning up instances automatically with AWS completely configured and set up, and also automating the creation and management of Dynatrace environments, take a look at this project- Dynatrace - Rest Tenant Automation
| Name | Version | Description |
|---|---|---|
| kiab | main | keptn 0.8.3 |
| kiab | 0.8.7 | keptn 0.8.7 |
| kiab | 0.8.4 | keptn 0.8.4 |
| kiab | 0.8.3 | keptn 0.8.3 |
| kiab | 0.8.2 | keptn 0.8.2 |
| kiab | 0.8.1 | keptn 0.8.1 (stable) |
| kiab | release-0.8pre | experimental also pushes to main |
- Update the ubuntu repository
- Installation of Docker (for building own Docker images)
- Installation of Microkubernetes
- Allow the Kube-API to run priviledged pods (necessary for deploying the FullStack Agent via Operator)
- Update IPTABLES: allowing traffic for pods internal and external
- Set up of useful BASH Aliases for working with the command line
- Enable autocompletion of Kubectl
- Installation of Dynatrace ActiveGate and configuration of Cluster and Workload monitoring
- Installation of Istio 1.9.1
- Installation of Helm Client
- Enabling own Docker Registry for the Cluster
- Convert the public IP in a (magic) domain (nip.io) for being able to expose all the needed services with subdomains.
- Routing of traffic to Istio-Ingressgateway via a Kubernetes NGINX Ingress using standard HTTP(S) ports 80 and 443. This way we dont need a public IP from the Cloud Provider
- Installation of Keptn (QualityGates also available)
- Expose the Keptn-Bridge Service
- Installation of Dynatrace OneAgent via Keptn
- Deployment of Jenkins preconfigured und managed as code
- Deployment of the Unleash-Server
- Onboard of the Sockshop-Carts Sample project
- Onboard of the keptnOrders project
- Onboard of the easytravel project
- Deployment of a cartsloadgenerator PoD
- Deployment of a Autonomous Cloud teaser home page with links to the pipeline, kubernetes api, keptn-bridge, keptn-api, jenkins
- Creation of valid SSL certificates for the exposed endpoints with Certmanager and HTTPs Let's encrypt.
- Create a user account and copy the standard user (ubuntu on this case) with his own home directory (a replica) and allowing SSH connections with text password. Useful for spinning an army of workshop clusters.
|
The bash file is scripted in a modular fashion allowing you with control flags to enable or disable the modules that you want to install in your box. This allows you to have a very slim cluster running keptn with the bare minimal resources or to have a full blown cluster with pretty much all the desired features and frameworks for your CI/CD pipelines and performance testings. |
|
For a step by step understanding of how Keptn-in-a-Box works and how to use it, take a look at the Keptn in a Box tutorial https://tutorials.keptn.sh/tutorials/keptn-in-a-box/) (This tutorial covers the base KIAB install)
- Ubuntu with internet connection (tested on 18.04 LTS and 20.04 LTS)
- A Dynatrace Tenant
- AWS Account Here you can get a free account
- You will get the most out of it if your DOMAIN is configured and reachable either by Dynatrace SaaS or Dynatrace Managed.
─ doc doc folder.
─ keptn-in-a-box.sh the executable (also where to define your variables)
─ functions.sh The definiton of functions and modules
─ resources
├── cartsloadgenerator Sources of the load container of the carts app
├── catalog Scripts for Onboarding the keptnOrders app
├── easytravel Scripts for Onboarding the easytravel app
├── demo Scripts for Onboarding the Carts app
├── dynatrace Scripts for integrating with Dynatrace
├── homepage Sources of the homepage for displaying the Autonomous Cloud teaser
├── ingress Files and logic for mapping, exposing the endpoints and services. Creation of Certificates.
├── istio istio config files
├── jenkins Deployment and configuration for Jenkins managed as code.
├── misc Miscelaneous (patch kubernetes dashboard)
└── virtualservices YAML files for virtualservices This section will give you an idea the nedded size for your Box. But it all depends on the modules you want to install and what is your goal and usecase. The installer comes with 3 predefined modules: minimal, default and full.
From the testing in AWS minimal installation takes ~ 4 minutes to complete and full ~ 20 minutes.
Below is a table for the sizing reference if you run a local VM or are virtualizing locally.
| Size | vCPUs | Memory (GiB) |
|---|---|---|
| t2.medium | 2 | 4 |
| t2.large | 2 | 8 |
| t2.xlarge | 4 | 16 |
| t2.2xlarge | 8 | 32 |
| c4.4xlarge | 16 | 30 |
The minimum required for running a Single Node Kubernetes cluster with keptn full features is a t2.medium (2 vCPU and 4 Gib of RAM) and 10 Gigabytes of disk space. If you feel frisky go for this size but the experience won't be the best.
Adding more RAM and more CPUs will speed up things. Depending what you want to achieve. Also consider that you'll have available less than 2 Gigs of disk space. This is the available disk after a minimal installation.
df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/root 20G 5.8G 14G 30% /The minimum required for running the default modules is t2.large with 13 Gigs of Disk space. We recommend 20 Gigs and t2.xlarge for the best experience.
The minimum required for running the default modules is t2.2xlarge with 30 Gigs of Disk space. We recommend c4.4xlarge with 60 Gigs of Disk space for the best experience.
-
Ubuntu 20.04 LTS is recommended.
-
Set the storage size to at least 60 GB.
-
Add a tag with key=Project and value=KIAB
-
Add a tag with Key=Name and Value=FirstName-LastName
-
Open Ports
If you define security rules in the Cloud provider or on your datacenter, your instance needs to have the following ports accessible:
- 22/SSH
- 80/HTTP
- 443/HTTPS
- 8080/Custom
- 4200/Custom
Next You will need to create the necessary tokens.
How to Create Tokens
Now you are ready to start the install process.
If you use a authentication key
ssh -i "DevOps.pem" [email protected]curl -O https://raw.githubusercontent.com/jyarb-keptn/keptn-in-a-box/0.8.7/keptn-in-a-box.shYou can also specify a specific release like 'curl -O https://raw.githubusercontent.com/jyarb-keptn/keptn-in-a-box/${KIAB_RELEASE}/keptn-in-a-box.sh' the master branch will be pointing to the actual release.
curl -O https://raw.githubusercontent.com/jyarb-keptn/keptn-in-a-box/main/keptn-in-a-box.shchmod +x keptn-in-a-box.shsudo bash -c './keptn-in-a-box.sh'The script will ask for the following inputs.
-
Use the Dynatrace information for the inputs to the script:
- TENANT="mytenant.live.dynatrace.com"
- PAASTOKEN=myDynatracePaaSToken
- APITOKEN=myDynatraceApiToken
for your tenent just use the folowing: {your-domain}/e/{your-environment-id} for managed or {your-environment-id}.live.dynatrace.com for SaaS. You can also use {your-environment-id}.sprint.dynatracelabs.com or a dev tenant if inclined.
User Email should match your Dynatrace login to the tenant. if the User Email matches your login, then the Dashbaords will show up under your account in the Dynatrace UI.
Dynatrace Tenant ID []:
Dynatrace API Token: []:
Dynatrace PaaS Token: []:
User Email []:answer 'y' if the information is correct, then Press enter
And that was it! Yes that easy!
This command will run the installation in a bash shell as sudo and send the job to the background. You will not see any output since stdout and stderr are piped to a logfile which is located by default in /tmp/install.log
For inspecting the installation on realtime type:
less +F /tmp/install.logBe patient, the deployment process will average ~20 min.
✅ Now you can get the URL for your KIAB homepage.
Just open a Browser with the IP.
Click on "Jenkins"
- First you will need to click "Install Suggested plugins"
You should then see a progress screen.
- Now "Jenkins is ready", just click "start using jenkins".
-
Now we will kick off the Pipeline 01-deploy-order-application to build out the application.
Login to Jenkins with the following credentials
- username = keptn
- password = keptn
After selecting the pipeline click "build"
Now we can follow the next steps to review the Pipelines and understand the different scenerios. Just follow the arrows.
➡️ Application Overview and Problem Scenerios
➡️ Load Test Analysis Performance Issue
➡️ SLOs and Load Test Analysis Error Issue
➡️ Upstream git
➡️ https://github.com/keptn/examples used for the sockshop app
Easytravel is still experimental.
The loadgen pods drive load to the staging and production angular frontends.
The loadgen to the www-staging also works.
You will need to create application detection rules, as these cannot be created by the API.
- All domains containing easytravel-angular.easytravel-production
- All domains that match easytravel-angular.easytravel-staging
- All domains that match easytravel-www.easytravel-staging
TODO: Make problem patterns dynamic.
- Verify Dynatrace config items.
- Verify application deployments in the Keptn bridge.
- if necessary, use Jenkins pipelines to deploy application services.
- Carts for example does not always deploy cleanly.
- Use Jenkins to drive load against the KeptOrders application.
- I run this a few times.
- You may need to set the Anomaly detection, for services, to have a sensitivity of "high" for failure rates.
- Exclude the loadgen processes from monitoring in Dynatrace.
- Check out your KIAB homepage and explore the links.
- Explore the dashboards in Dynatrace.
- Have some fun....
The dynatrace-sli-service is a Keptn service that is responsible for retrieving the values of SLIs from your Dynatrace Tenant via the Dynatrace Metrics v2 API endpoint. For that it handles the Keptn Event sh.keptn.internal.event.get-sli which gets executed as part of a quality gate evaluation! Dynatrace SLI Service
Rest easy, we have created a script to initialize the SLI service and create an OOB Dashboard example.
cd ~/keptn-in-a-box/resources/dynatraceValidate the KEPTN_DOMAIN environment variable has been set.
If it has not been set, then you will need to set the KEPT_DOMAIN environment variable.
Set the KEPTN_DOMAIN
export KEPTN_DOMAIN=<Your DOMAIN>Run this script, first make it executable.
chmod +x setdbenv.sh
./setdbenv.shThis will enable your Dynatrace SLO-based dashboard.
Now we need to trigger a quality gate evaluation.
cd ~/keptn-in-a-box/resources/demo
chmod +x triggerDashbaordSLI.sh
./triggerDashbaordSLI.shNow if you navigate to the carts service under the sockshop project in Keptn, you will see the SLI's from the configured dashboard.
You can also navigate to the dashboard in Dynatrace to check out how the SLI's have been created. Also, any subsequent builds for the sockshop app will use the Dashboard SLI's.
The default installation is installationBundleAll which sets the control flags to true or false for the following modules: Note: full install for complete demo capabilities
update_ubuntu=true
docker_install=true
microk8s_install=true
setup_proaliases=true
enable_k8dashboard=true
istio_install=true
helm_install=true
certmanager_install=false
certmanager_enable=false
keptn_install=true
keptn_examples_clone=true
resources_clone=true
hostalias=false
keptn_catalog_clone=true
git_deploy=true
git_migrate=true
dynatrace_savecredentials=true
dynatrace_configure_monitoring=true
dynatrace_activegate_install=true
dynatrace_configure_workloads=true
keptndeploy_homepage=true
keptndemo_unleash=true
keptndemo_unleash_configure=true
keptndemo_cartsonboard=true
keptndemo_cartsload=true
keptndemo_catalogonboard=true
keptndemo_easytravelonboard=true
keptndashboard_load=false
createMetrics=false
expose_kubernetes_api=true
expose_kubernetes_dashboard=true
patch_kubernetes_dashboard=true
keptn_bridge_disable_login=true
create_workshop_user=false
jmeter_install=false
post_flight=false
patch_config_service=false
enable_registry=true
certmanager_install=true
certmanager_enable=true
create_workshop_user=false
keptn_bridge_disable_login=true
jenkins_deploy=trueDynatrace OneAgent and Dynatrace ActiveGate will be installed and configured if you provided your credentials. Otherwise they won't be installed.
The script will install all the modules shown above and the github repository will be cloned in the home (~) directory of the sudo user that executed the script.
The script was first created to be run as root without an interactive shell since it is passed as userdata on creation of the elastic cloud machine via a python rest automation program.
-
Use the Dynatrace information for the inputs to the script:
- TENANT="mytenant.live.dynatrace.com"
- PAASTOKEN=myDynatracePaaSToken
- APITOKEN=myDynatraceApiToken
for your tenent just use the folowing: {your-domain}/e/{your-environment-id} for managed or {your-environment-id}.live.dynatrace.com for SaaS. You can also use {your-environment-id}.sprint.dynatracelabs.com or a dev tenant if inclined.
Generate an API token and a PaaS token in your Dynatrace environment.
Log in to your Dynatrace tenant and go to Settings > Integration > Dynatrace API. Then, create a new API token with the following permissions
Make sure you have the Access problem and event feed, metrics, and topology switch enabled for the API token.
TODO: API configuration for new API token interface, standby.
In your Dynatrace tenant, go to Settings > Integration > Platform as a Service, and create a new PaaS Token.
For more details on the Integration with Dynatrace read the following links:
- OneAgent Operator
- API Authentication
- PaaS Token
- Kubernetes Cluster Monitoring with Dynatrace
- Kubernetes Events
- Kubernetes Workloads
With the installationModulesDefault or installationModulesFull Dynatrace is automatically installed if the credentials are configured.
➡️ Back
- Log in to AWS
- Click on "Launch instance"
- Select "Ubuntu Server [18|20].04 LTS (HVM) "
- Choose your Instance Type "t2.xlarge" or 2xlarge recommended.
- Select "Next - configure instance details"
- In Configure Instance details - Advanced options (below) copy the keptn-in-a-box.sh file. (as string or drop it, doesn't matter). Remember to edit your file if you want to customize your box.
- Review it and launch your instance.
- Description to be added. Please see the Dynatrace Rest Tenant Automation project for reference. The RTA project can spin as many instances as you want by providing a simple CSV file. It will also create and configure Dynatrace environments for each CSV entry, 😍 perfect for delivering workshops 👨💻. If you are interested in a workshop get in contact with us.
The keptn-in-a-box project is highly customizable (obviously since it's a bashfile) below are some customizations that are the most used. All customizations can and should be done in the ** keptn-in-a-box.sh** file and not in the functions.sh file. This way you keep a nice delegation of tasks and functionality.
Comment out the Default and uncomment the installation type you want. For example for a Demo:
# ==================================================
# ----- Select your installation Bundle ----- #
# ==================================================
# Uncomment for installing only Keptn
# installationBundleKeptnOnly
# - Comment out if selecting another bundle
#installationBundleDemo
# - Comment out if only want to install only the QualityGates functionality
#installationBundleKeptnQualityGates
# - Uncomment for installing Keptn-in-a-Box for Workshops
# installationBundleWorkshop
# - Uncomment below for installing all features
installationBundleAll
# - Uncomment below for installing a PerformanceAsAService Box
#installationBundlePerformanceAsAServiceThis will install single node kubernetes cluster, keptn, k8 dashboard and expose the endpoints. It will also clone the examples, onboard the carts sample app and add a quality gate.
This scenario will create a Box for delivering Performance as a Service in an instant. It will install the QualityGates functionality of Keptn (not installing the other Keptn services and/or components such as Istio). It will install Jenkins preconfigured and managed as Code with 6 example pipelines. You only need to Tag your service in Dynatrace and (if wanting to do a loadtest) provide the URL of your application do a sample Loadtest and validate it. If you want to learn more about Qualitygates and SRE Driven development, take a look at the following tutorial: https://tutorials.keptn.sh/tutorials/keptn-progressive-delivery-dynatrace/
For example triggering the first quality gate and evaluating a service will look something like this:
Info: The first launch of the pipeline wil fail since they need to be initialized and they need parameters to do so. it is ok, on the second run, it will ask you for your information.
- 01-deploy-order-application - Deploy the keptnorders application
- 01.1-easytravel-delivery - Deploy the easytravel application
- 01.2-Sockshop Delivery - Deploy the sockshop application
- 02-qualitygate-evaluation - will do only the evaluation of the given service and timeframe
- 03-simpletest-qualitygate - will do a simple multi-step test directly from Jenkins via HTTP and do the validation via keptn.
- 04-performancetest-qualitygate - will do a Loadtest based on a simple JMeter script.
➡️ Back
For migrating keptn projects to your self-hosted git repository afterwards just execute the following. To make it easier, I created a script you can execute to add to the upstream git repo. Here is the helper script.
cd ~$KEPTN_IN_A_BOX_DIR/resources/gitea
./update-git-keptn-post-flight.sh <DOMAIN>The Domain is required
➡️ Back
By default the Script will get the public ip and give it a magic domain with nip.io like ip=1.2.3.4 will become 1-2-3-4.nip.io. This way with an NGINX Ingress Keptn and all Istio and Kubernetes services are available with subdomains via the standard HTTP ports 443 and 80.
This example could apply if I want to run a box in a VirtualMachine in my home network:
DOMAIN=192.168.0.1.nip.ioBy adding/commenting out the following control flags, you'll install Certmanager. A Cluster Issuer will be added and a valid certificate with let's encrypt for your public endpoints will be created. This way the exposed Endpoints/Services like the Keptn API, Homepage, or Kubernetes API will have a valid Certificate. The Certificate renewal will happen automatically.
certmanager_install=true
certmanager_enable=trueIf you provide your Email, the Cluster issuer will be created with the given email account. If left empty a fake Email account will be generated. for metrics and dashboards, match your Dynatrace account email.
CERTMANAGER_EMAIL="[email protected]"Add the modules before the doInstallation
You might have notice the following variables:
NEWPWD="dynatrace"
NEWUSER="dynatrace"This variables in combination with the control flag create_workshop_user=false will create a workshop user. It will clone the USER home directory and add hi configuration so the NEWUSER can also interact with keptn, docker and kubectl. An SSH Password will be configured and allowed.
This are the actual versions of the different Modules
# **** Installation Versions ****
# ==================================================
# ----- Components Versions ----- #
# ==================================================
KIAB_RELEASE="main"
ISTIO_VERSION=1.9.1
CERTMANAGER_VERSION=0.14.0
KEPTN_VERSION=0.8.2
KEPTN_DT_SERVICE_VERSION=0.12.0
KEPTN_DT_SLI_SERVICE_VERSION=0.9.0
KEPTN_EXAMPLES_REPO="https://github.com/keptn/examples.git"
KEPTN_EXAMPLES_BRANCH="release-0.8.1"
KEPTN_EXAMPLES_DIR="~/examples"
KEPTN_CATALOG_REPO="https://github.com/jyarb-keptn/overview.git"
KEPTN_CATALOG_BRANCH="main"
KEPTN_CATALOG_DIR="~/overview"
TEASER_IMAGE="pcjeffmac/nginxacm:0.8.1"
KEPTN_BRIDGE_IMAGE="keptn/bridge2:0.8.0"
MICROK8S_CHANNEL="1.19/stable"
KEPTN_IN_A_BOX_REPO="https://github.com/jyarb-keptn/keptn-in-a-box.git"
KEPTN_IN_A_BOX_DIR="~/keptn-in-a-box"Feel free to experiment and change the versions. We will try to keep the list up to date. Be careful here as certain deprecations will effect the build
At the begining of the functions.sh file the installation modules are listed. You can enable them in the keptn-in-a-box.sh file before calling the doInstallation function.
- Inspect the installation
less +F /tmp/install.logTo view a verbose output (of every executed command) set the following control flag verbose_mode=true
If you need to rebuild the environment, follow these steps.
cd ~/keptn-in-a-box- The following script will reset the Ubuntu instance by removing Kubernetes, the Dynatrace ActiveGate and any configuration.
./resetenv.sh- In your Dynatrace tenant Delete the kubernetes connection, under "cloud and virtualization"
Now we need to re-initialize the environment.
- Run the following commands, then follow the process from Step 4 above.
cd ~
sudo bash -c './keptn-in-a-box.sh'Run the following script.
cd ~/overview/keptn-onboarding/scripts
./createRequestAttributes.shA common issue is the creation of the Calculated service metrics. If you do not have four calculated service metrics.
run the following script with the inputs.
cd ~/overview/keptn-onboarding/scripts/
./createTestStepCalculatedMetrics.sh CONTEXTLESS keptn_project keptnordersDeepDive into the project (understanding how Microk8s, NGINX Ingress routing, Istio and Keptn work together)
This project was first presented to the Keptn Community in a Keptn Developer Meeting on the 23rd of April 2020.
Join the Keptn Community for more interesting projects and keep up to date with Keptn!
On a high level Kubernetes architecture this is how the setup of the Microkubernetes machine is setted up:
KIAB (Keptn in a Box)
| Componenet | Details |
|---|---|
| Kubernetes |
|
| Keptn |
|
| Upstream Git |
|
| CI/CD |
|
| Order Application |
|
| SockShop Application |
|
| Easytravel Application |
|
If you have any ideas for improvements or want to contribute that's great. Create a pull request or file an issue.