Skip to content

Home

Jump to bottom
ProfiseeAdmin edited this page Oct 4, 2022 · 41 revisions

Table of Contents

Azure

Verify:

  1. Open up cloud shell

    Launch Cloud Shell from the top navigation of the Azure portal.

    CloudShell

  2. Configure kubectl

    az aks get-credentials --resource-group MyResourceGroup --name MyAKSCluster --overwrite-existing

  3. The initial deploy will have to download the container which takes about 10 minutes. Verify it has finished downloading the container:

    kubectl -n profisee describe pod profisee-0 #check status and wait for "Pulling" to finish

  4. Container can be accessed with the following command:

    kubectl -n profisee exec -it profisee-0 powershell

  5. Kubernetes log for the container can be accessed with the following command:

    kubectl logs -n profisee profisee-0 -f --timestamps

  6. System logs can be accessed with the following command after you access shell in the container (step 4 above):

    #Configuration log
Get-Content C:\Profisee\Configuration\LogFiles\SystemLog.log
#Authentication service log
Get-Content C:\Profisee\Services\Auth\LogFiles\SystemLog.log
#WebPortal Log
Get-Content C:\Profisee\WebPortal\LogFiles\SystemLog.log
#Gateway log
Get-Content C:\Profisee\Web\LogFiles\SystemLog.log

  7. Go to Profisee Platform web portal

    • http(s)://app.company.com/profisee

Troubleshooting:

Uninstall profisee and reinstall

helm -n profisee repo add profisee https://profisee.github.io/kubernetes
    helm -n profisee uninstall profiseeplatform
    #Get Settings.yaml from the secret it is stored in. WARNING: This secret gets created only for the **initial** deployment of Profisee. If you have made changes to Profisee since then (new image tag, new license, sql server changes, etc.) these changes would not be reflected by that file. Once you create the file in your cloudshell, please verify its contents against your current deployment.
    kubectl -n profisee get secret profisee-settings -o jsonpath="{.data.Settings\.yaml}" | base64 --decode > Settings.yaml
    helm -n profisee install profiseeplatform profisee/profisee-platform --values Settings.yaml

    #If you get an error during a re-installation that something still exists, run this before the install statement
    helm -n profisee template profiseeplatform profisee/profisee-platform | kubectl delete -f -

Connect to container and look at log

kubectl -n profisee exec -it profisee-0 powershell
Get-Content C:\Profisee\Configuration\LogFiles\SystemLog.log

Check SQL connectivity from container

#Expected outcome is for connection to close and open. Failure would result in a timeout or access denied. 
#If you get an access denied, make sure that your cluster's egress IP in the infrastructure resource group 
#has been added to the SQL server's firewall allow list.

$connectionString = 'Data Source={0};database={1};User ID={2};Password={3}' -f $env:ProfiseeSqlServer,$env:ProfiseeSqlDatabase,$env:ProfiseeSqlUserName,$env:ProfiseeSqlPassword
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection $connectionString
$sqlConnection.Open()
$sqlConnection.Close()

Check connection to Azure storage file share from container

#map drive to X
$pass=$env:ProfiseeAttachmentRepositoryUserPassword|ConvertTo-SecureString -AsPlainText -Force
$azureCredential = New-Object System.Management.Automation.PsCredential($env:ProfiseeAttachmentRepositoryUserName,$pass)
New-PSDrive -Name "X" -PSProvider "FileSystem" -Root $env:ProfiseeAttachmentRepositoryLocation -Credential $azureCredential -Persist;
#remove mapped drive
Remove-PSDrive X

Copying files to/from container

#Run this in cloudshell.
#copy file to container
kubectl -n profisee cp appsettings.json profisee-0:profisee/services/auth/appsettings.json

#copy file from container
kubectl -n profisee cp profisee-0:profisee/services/auth/appsettings.json appsettings.json

#You can also shell into the profisee-0 container and copy files to the c:\<FileShareName> folder, then download from the Azure storage account's files share.

"Edit" a value (logging) in web.config

((Get-Content -path C:\profisee\services\auth\appsettings.json -Raw) -replace 'Warning','Debug') | Set-Content -Path C:\profisee\services\auth\appsettings.json

Upgrade from one version to another via Azure portal

Go to the Kubernetes cluster
Click on Workloads on left under Kubernetes resources
Click Stateful sets
Click on profisee
Click on YAML on left
Replace the value for image: to use the new release
	Note the image names have changed going forward.  They are always profiseeplatform:releasename.version.
	Example
		Old - 'profisee.azurecr.io/profisee2020r1:0'
		New - 'profisee.azurecr.io/profiseeplatform:2020r1.0'
		Old - 'profisee.azurecr.io/profisee2020r1:1'
		New - 'profisee.azurecr.io/profiseeplatform:2021r2.1'
		New - 'profisee.azurecr.io/profiseeplatform:2021r3.0'
		New - 'profisee.azurecr.io/profiseeplatform:2022r1.0'
Click Review + Save
Check confirm
Save

Upgrade from one version to another via uninstall/reinstall

helm -n profisee repo add profisee https://profisee.github.io/kubernetes
helm -n profisee uninstall profiseeplatform

#Get Settings.yaml from the secret it is stored in. WARNING: This secret gets created only for the **initial** deployment of Profisee.
#If you have made changes to Profisee since then (new image tag, new license, sql server changes, etc.) these changes would not be
#reflected by that file. Once you create the file in your cloudshell, please verify its contents against your current deployment.

kubectl -n profisee get secret profisee-settings -o jsonpath="{.data.Settings\.yaml}" | base64 --decode > Settings.yaml

#Note the name is now just called profiseeplatform without the release name in it like 2021r1 and image.tag will vary based on
#the minor release, starting with 0
#Note that, by default, it installs in the profisee namespace which must already exist. To create the namespace, if it is missing,
#connect to cloudshell, authenticate to the cluster and run: kubectl create namepace profisee

helm -n profisee  install profiseeplatform profisee/profisee-platform --values Settings.yaml --set image.repository=profiseeplatform --set image.tag=2022r1.0

Upgrade from one version to another

Create a file named UpdateProfisee.yaml (any name is fine as long as use that file name in the patch statement) that has this content:

spec:
  template:
    spec:
      containers:
      - name: profisee
	image: profisee.azurecr.io/profiseeplatform:2022r1

Upload to cloud shell drive

Launch Cloud Shell from the top navigation of the Azure portal.
Click upload/download, then upload and chose the file you just created

Connect to aks cluster

az aks get-credentials --resource-group MyResourceGroup --name MyAKSCluster --overwrite-existing

Patch it

kubectl -n profisee patch statefulset profisee --patch $(Get-Content UpdateProfisee.yaml -Raw)

Replace license via uninstall/reinstall

#Follow this procedure if you are not using Key Vault. Connect to cloudshell and run:
helm -n profisee repo add profisee https://profisee.github.io/kubernetes
helm -n profisee uninstall profiseeplatform
#Get Settings.yaml from the secret it is stored in. WARNING: This secret gets created only for the **initial** deployment of Profisee.
#If you have made changes to Profisee since then (new image tag, new license, sql server changes, etc.) these changes would not be
#reflected by that file. Once you create the file in your cloudshell, please verify its contents against your current deployment.
kubectl -n profisee get secret profisee-settings -o jsonpath="{.data.Settings\.yaml}" | base64 --decode > Settings.yaml
helm -n profisee install profiseeplatform profisee/profisee-platform --values Settings.yaml --set licenseFileData=PastedBase64LicenseString

Replace license via Azure portal

#Follow this procedure if you are NOT using Key Vault. 
Go to the kubernetes cluster:
Click on Configuration on left under Kubernetes resources
Click Secrets
Click on profisee-license
Click on YAML on left
Replace the value under profisee.plic: >- with the new license string.  Be sure to keep the 4 spaces at the beginning of the line
Click Review + Save
Check confirm
Save
Then you need to delete the profisee-0 pod:
Click on Workloads on left under Kubernetes resources
Click on Pods on the left
Filter by namespace by selecting the profisee namespace
Check the checkbox for profisee-0
Click Delete on Top
Confirm.
You can run this in cloudshell to monitor the progress and verify the license is valid: kubectl logs -n profisee profisee-0 -f
Look in the first few lines for License is valid.

#Follow this procedure if you ARE using Key Vault. 
Go to your Key Vault:
Click on Secrets on the left
Click on the secret that you used for the profisee license, ex: profisee-license
Update it with the one provided to you by Profisee Support. Save.

Now go to kubernetes cluster:
Click on Configuration on left under Kubernetes resources
Click on Secrets on the left
Type in the search profisee-license, select and delete it.
Click on Workloads, then Pods
Filter by namespace by selecting the profisee namespace
Find the profisee-keyvault-xxxxxxxxx-xxxx pod (random letters/numbers), select it and delete it. Once the new profisee-keyvault-xxxxxxxxx-xxxx
pod starts you can check back under Configuration, then Secrets and you will see a new profisee-license secret that should be just a few seconds
old.
Click on Workloads on left under Kubernetes resources
Click on Pods on the left
Filter by namespace by selecting the profisee namespace
Check the checkbox for profisee-0
Click Delete on Top
Confirm.
You can run this in cloudshell to monitor the progress and verify the license is valid: kubectl logs -n profisee profisee-0 -f
Look in the first few lines for License is valid.

Scale to more than one container

#this will add another pod (container) to have two servers that are completely load balanced
kubectl -n profisee scale sts profisee --replicas=2

Uninstall, reinstall or upgrade nginx.

    #WARNING: this process will generate a new Public IP address for you ingress and you'll need to update your DNS record with that IP.
    #If you do NOT want to uninstall and reinstall nginx please read further down.
			
    #Uninstall nginx. Please give it a minute after the uninstall so you get a clean slate.
    helm -n profisee uninstall nginx
    #Install nginx
    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    #Get profisee nginx settings
    curl -fsSL -o nginxSettings.yaml https://raw.githubusercontent.com/profisee/kubernetes/master/Azure-ARM/nginxSettings.yaml;
    #If you are NOT using Let's Encrypt for certificates (you've supplied your own) run this command:
    helm install -n profisee nginx ingress-nginx/ingress-nginx --set controller.service.appProtocol=false --values nginxSettings.yaml
    #If you ARE using Let's Encrypt for certificates but your FQDN is profisee.mydomain.com run this command:
    helm install -n profisee nginx ingress-nginx/ingress-nginx --set controller.service.appProtocol=false --values nginxSettings.yaml
    #If you ARE using Let's Encrypt for certificates AND you are using an FQDN like profisee.eastus2.cloudapp.azure.com run these commands:
    $DNSHOSTNAME="myUniqueHostnameInTheRegion" #ex. profisee
    helm install -n profisee nginx ingress-nginx/ingress-nginx --values nginxSettings.yaml --set controller.service.appProtocol=false --set controller.service.annotations."service\.kubernetes\.io/azure-dns-label-name"=$DNSHOSTNAME

    #If you'd like to simply upgrade nginx run these commands:
    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    helm repo update
    helm upgrade -n profisee nginx ingress-nginx/ingress-nginx --reuse-values --set controller.service.appProtocol=false

Add cluster IP address to SQL

#If you experience odd SQL connectivity issues from the container(s), try adding the AKS egress IP to the SQL firewall
#From in the container run this to get the egress IP of the cluster
Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
#Another method is to go to Kubernetes, then click on Properties, click on the link there that will take you to the infrastructure resource
#group, there will be two public IP addresses in the resource group. Ingress public IP is always named kubernetes-(followed be about 30
#alphanumeric characters), whereas egress will be named as a block of five sets of randomly generated letters and numbers in an
#8-4-4-4-12 format, ex. xxxxxxxxx-xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. For SQL you need the egress one.

#Now go to SQL server firewall and add the egress IP to it.

Add FileShare volume to container

#This can happen if using an old deployment of 2020r1 before the fileshare was added (before 9/14/2020)
#set these variables
STORAGEACCOUNTNAME="MyStorageAccountName"
FILEREPOPASSWORD="MyStorageAccountAccessKey"
STORAGEACCOUNTFILESHARENAME="files"

#Run this for azure cloud shell
curl -fsSL -o StatefullSet_AddAzureFileVolume.yaml "https://raw.githubusercontent.com/profiseedev/kubernetes/master/Azure-ARM/StatefullSet_AddAzureFileVolume.yaml";
STORAGEACCOUNTNAME="$(echo -n "$STORAGEACCOUNTNAME" | base64)"
FILEREPOPASSWORD="$(echo -n "$FILEREPOPASSWORD" | base64 | tr -d '\n')" #The last tr is needed because base64 inserts line breaks after every 76th character
sed -i -e 's/$STORAGEACCOUNTNAME/'"$STORAGEACCOUNTNAME"'/g' StatefullSet_AddAzureFileVolume.yaml
sed -i -e 's/$STORAGEACCOUNTKEY/'"$FILEREPOPASSWORD"'/g' StatefullSet_AddAzureFileVolume.yaml
sed -i -e 's/$STORAGEACCOUNTFILESHARENAME/'"$STORAGEACCOUNTFILESHARENAME"'/g' StatefullSet_AddAzureFileVolume.yaml
kubectl -n profisee apply -f StatefullSet_AddAzureFileVolume.yaml

If your SQL password has changed, then you need to update it in kubernetes

#If you are NOT using Key Vault, follow this procedure.
Secrets are base64 encoded, so you first need to get the base64 version of the password.
	$OrigString="MyPassword"
	$B64String =[Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($OrigString)) 
	write-host $B64String 
Go to the kubernetes cluster
Click on Configuration on left under Kubernetes resources
Click Secrets on the left
Click on profisee-sql-password
Click on YAML on left
Find ProfiseeSqlPassword and replace it with the base64 version of the password.
Click Review + Save
Check confirm
Save.
Now go to Workloads, Pods, click the checkbox next to profisee-0 and delete the pod.

#If you ARE using Key Vault, follow this same procedure in this wiki for replacing a Profisee license file when using Key Vault.
#You'll be deleting the profisee-sql-password secret from Configuration, Secrets and then restarting the
#profisee-keyvault-xxxx-xxxxxxxx pod, followed by the profisee-0 pod.

View container config manager logs "live" in kubernetes

Starting with 2020R2, the configuration manager log (startup) is now streamed to kubernetes
you can view it by running this, the -f is "follow" and will stream it as it goes (tail)
kubectl -n profisee logs profisee-0 -f

Certificate issues with Let's Encrypt

#Inspect the certificate
kubectl -n profisee get certificate
#if Ready is false, then get details
kubectl -n profisee describe certificate
#if error, look at the request
kubectl -n profisee describe certificaterequest

troubleshooting certificates

troubleshooting acme

    #if the certificate got issued properly but it's still in a false state, delete the certificate and a new one will be issued
    kubectl -n profisee delete certificate profisee-tls-ingress

    If the certificate has expired, use the above command to delete the certificate and a new one will be issued.

Enable Key Vault key rotation

If you enable Key Vault integration when deploying the Profisee platform with the ARM template and you want to enable key rotation, this is how:

Uninstall CSI driver

helm uninstall -n profisee csi-secrets-store-provider-azure

reinstall the driver with the rotate flag

helm install -n profisee csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --set enableSecretRotation=true

Post and Pre Init Scripts

The latest version of the deployment support post and pre init scripts in the Settings.yaml. The setting names are preInitScriptData and postInitScriptData. The scripts must be base64 encoded PowerShell script.

The Pre init script fires before the container startup logic runs. If you need something changed before the container startup logic runs, then put script in here. eg change log levels.

The post init script fires after the container startup logic has runs. At this point the pod is fully configured and is running. If you need something changed after the container startup logic runs, then put script in here. eg change OIDC information.

Deployment Managed Identity (DMI)

Permissions needed for the Deployment Managed Identity for a Profisee PaaS ARM template install.

  • Contributor to the resource group
    • This can be directly assigned to the resource group, or at the subscription level. Note: If you would like for the Managed Identity to create the resource group for you, this would have to be at the Subscription level.
  • If updating DNS?
    • DNS Zone Contributor access to the DNS zone that will get updates by the Managed Identity.
  • If updating Azure Active Directory to create Application registration
    • Application Administrator – Note this is a change from before
      • We used to only have Application Developer role but it does not have the ability to assign the needed permissions to the Application registration, that is why we used to have to manually add them before in some cases. If you would like the DMI to create the Azure AD application registration and assign the required permissions (User.Read) then you would need to grant the DMI Application Administrator role in Azure AD.
  • If using Key Vault – Microsoft documentation is here
    • Managed Identity Contributor at subscription level
      • This is required because the DMI needs to create a Key Vault Specific Managed Identity in the AKS infrastructure resource group. As the infrastructure resource group cannot be pre-created which we need this permission so that we can Key Vault Specific Managed Identity.
    • User Access Administrator as subscription level
      • This is required because the DMI needs to assign roles or permissions to the Key Vault Specific Managed Identity that AKS runs under. This MI will be used by AKS to talk to Key Vault. If policy-based Key Vault we assign Get permissions only. If RBAC-based Key Vault we assign Key Vault Secrets User role.

Add editor to container

Install Chocolatey

    Invoke-WebRequest https://chocolatey.org/install.ps1 -UseBasicParsing | Invoke-Expression

Install VIM

    choco install vim
    example usage: vim c:\profisee\web\web.config
    example commands when in vim: press i to get into insert mode to edit the file.  press :wq to save and close the file

Troubleshooting with Lens

Install Lens (Kubernetes IDE)

Main website https://k8slens.dev

Install the latest https://github.com/lensapp/lens/releases/latest

Enable metrics

Lens uses prometheus https://prometheus.io/ for metrics

To enable it right-click on node name in left bar and click settings

Scroll down to metrics and click install

Add AKS cluster to Lens

Go to Azure portal, open cloud shell

Run this to "configure" kubectl
az aks get-credentials --resource-group MyResourceGroup --name MyAKSCluster --overwrite-existing

Get contents of kube.config
run kubectl config view --minify --raw
copy all the out put of that command (select with mouse, right click copy)

Go to Lens
Click big plus (+) to add a cluster
Click paste as text
Go to select context dropdown and choose the cluster
Click outside the dropdown area
Click "Add Cluster(s)"
Wait for it to connect and now Lens is connected to that AKS cluster.

Connect to pod (container)

In Lens, choose workloads, then pods
Click on pod - profisee-(x)
Click on the "Pod Shell" left icon in top blue nav bar.  This will "connect" you to the container
Now in the terminal window (bottom), you are "connected" to the pod (container)

Replace license with Lens

#If you are NOT using Key Vault, please follow the procedure below. 
In Lens, choose Configuration, then Secrets
Click on profisee-license
Paste your new license string supplied by Profisee Support in textbox under profisee.plic
Click save.
Your license has been updated.
You have to destroy the profisee-0 pod and have it recreate itself for it to take affect. 

#If you ARE using Key Vault, please follow the procedure in this Wiki to update the license in the Key Vault first. Then do this in Lens:
In Lens, choose Configuration, then Secrets
Delete the profisee-license secret.
Go to Workloads, Pods, find and delete the profisee-keyvault-xxxx-xxxxxxxx pod. Wait till it recreates then delete the profisee-0 pod.
Clone this wiki locally