Skip to content

Chef Cookbook - Base Linux Cookbook (Amazon Linux 1 & 2)

License

Notifications You must be signed in to change notification settings

bonusbits/bonusbits_base

Repository files navigation

Bonus Bits Base Cookbook

Project Release Circle CI Join the chat at https://gitter.im/bonusbits/bonusbits_base GitHub issues

Purpose

Chef Cookbook that will setup the basics for Amazon Linux or Ubuntu systems. Be sure to set appropriate overrides for what you do and don't want to be setup in your environment files. This is a great starting point for all your nodes. It's also a great example for writing wrapper cookbooks.

The secondary purpose of this cookbook is to give various examples that can be replicated in other cookbooks. I did my best to include all the major coding and testing scenarios.

Supported Platforms

Primary Focus is on Amazon Linux then Ubuntu. RHEL and CentOS not tested but should work.

  • Amazon Linux v1 & v2
    • Amazon Linux v2 is el7 and has systemd. The issue is systemd requires kernel access and is not fully functioning in a Docker container as of yet and can cause issues with standard configurations we are used to doing with say upstart (initd). I have it working with this cookbook, but when you start adding your app side of things it may be very problematic if deploying as a Docker image/container.
  • Ubuntu 16/18 Not Fully Tested

Successfully Tested Versions

Software Version
Chef Development Kit 4.5.0
Chef Client 15.4.45
Cookstyle 5.9.3
CircleCI v2.1
Docker Desktop 2.1.6.0 (edge)
Docker 19.03.5
Foodcritic 16.1.1
Inspec 4.18.0
Kubernetes 1.15.5
Rubocop 0.72.0
Test Kitchen 2.3.3
MacOS 10.15.1 (Catalina)

Attributes

Section Attribute Default Type Description
Audit bonusbits_base:audit:configure false Boolean only when deployed with dockerfile
AWS bonusbits_base:aws:inside false Boolean discovery toggles this
AWS bonusbits_base:aws:install_tools false Boolean Install aws cli on other none Amazon Linux OSs
AWS bonusbits_base:aws:region N/A String Used Ohai EC2 Plugin to Discover, but optional overriding this way
Bash Profile bonusbits_base:bash_profile:configure true Boolean Add a few minor profile customizations
Certs bonusbits_base:certs:configure false Boolean Enable deployment of Internal/Other CA Certificates
Certs bonusbits_base:certs:data_bag bonusbits_base String Data Bag Name (Folder) that contain Certificate Data Bag Item
Certs bonusbits_base:certs:data_bag_item internal_ca String Data Bag Item (JSON) that contain CA Certificate PEM & HASH
CloudWatch Logs bonusbits_base:cloudwatch_logs:configure true Boolean Enable AWS CloudWatch Logs Agent Installation and configuration
CloudWatch Logs bonusbits_base:cloudwatch_logs:deploy_logs_conf true Boolean Deploy log streams configuration. Disable if have custom log streams in your wrapper
CloudWatch Logs bonusbits_base:cloudwatch_logs:logs_group_name kitchen-bonusbits-base String Name of CloudWatch Logs Log Group. Override in CloutFormation with created group name
Default run_state:detected_environment dev String Determine if actual environment is dev, qa, stg or prd based on Chef Environment name
Default bonusbits_base:deployment_type N/A String Determine if deployment type is docker, lxc, lxd, kvm, vbox, ec2 or other using Ohai and Discovery Library
Default bonusbits_base:deployment_location N/A String Determine if deployment local is circleci, aws or local using Env var and Discovery Library
Default bonusbits_base:deployment_method N/A String Determine if deployment method is kitchen, cloudformation, dockerfile, dockerimage or unknown
                                                   | Boolean | Install Package list from EPEL Repo   |

| Gem Source | bonusbits_base:gem_source:configure | false | Boolean | Enable System and Chef Ruby Gem Source replacement with say an internal repo | | Gem Source | bonusbits_base:gem_source:source_url | https://artifactory.localdomain.com/artifactory/api/rubygems/ | String | Source Ruby Gem Repo URL to replace rubygems.org. This is a bogus example URL | | Gem Source | bonusbits_base:gem_source:source_url | https://artifactory.localdomain.com/artifactory/api/rubygems/ | String | Source Ruby Gem Repo URL to replace rubygems.org. This is a bogus example URL | | Kitchen Shutdown | bonusbits_base:kitchen_shutdown:configure | *true | Boolean | Enable creation of crontab to shutdown and terminate an AWS EC2 instance deployed by kitchen. *If an EC2 instance and deployed with Kitchen then the default is 'true'. | | Kitchen Shutdown | bonusbits_base:kitchen_shutdown:minute | 0 | String | Crontab minute to run | | Kitchen Shutdown | bonusbits_base:kitchen_shutdown:hour | 6 | String | Crontab hour to run. Default is 7AM UTC = 11PM PST/2AM EST. Cron runs on whatever the timezone is set on system. Default example based on UTC | | Node Info | bonusbits_base:node_info:configure | true | Boolean | Deploy a Bash/Batch Shell Script that can be used to quickly display information about the local system. Script deployed to standard system path locations: Linux /usr/local/bin/nodeinfo Windows C:/Windows/System32/nodeinfo.cmd | | Packages | bonusbits_base:packages:install | true | Boolean | Install a set of basic packages | | Packages | bonusbits_base:packages:update | true | Boolean | Update all but kernel packages on system | | Packages | bonusbits_base:packages:amazon_linux1:packages | List Here | Array | List of Amazon Linux 1 Packages | | Packages | bonusbits_base:packages:amazon_linux2:packages | List Here | Array | List of Amazon Linux 2 Packages | | Packages | bonusbits_base:packages:debian:packages | List Here | Array | List of Debian Linux Packages | | Packages | bonusbits_base:packages:redhat:packages | List Here | Array | List of Redhat Linux Packages | | Proxy | bonusbits_base:proxy:configure | false | Boolean | Configure proxy settings on system | | Proxy | bonusbits_base:proxy:use_ssl | false | Boolean | Use SSL will set https for the Proxy URL attribute. Otherwise http used | | Proxy | bonusbits_base:proxy:host | 10.0.2.2 | String | Proxy Host Address | | Proxy | bonusbits_base:proxy:port | 8888 | String | Proxy Host Port | | Proxy | run_state:proxy:user | nil | String | Proxy Authentication User Name. Run state used for security so not wrote to node | | Proxy | run_state:proxy:password | nil | String | Proxy Authentication Password. Run state used for security so not wrote to node | | Proxy | bonusbits_base:proxy:url | http://10.0.2.2:8888 | String | Proxy Full URL. http/https and user:password added dynamically if overrides converged | | Security | bonusbits_base:security:selinux:configure | true | Boolean | Configure SeLinux | | Security | bonusbits_base:security:selinux:action | disabled | String | Set SeLinux Enforcement Action. Options: enforcing, permissive and disabled | | Sudoers | bonusbits_base:sudoers:configure | true | Boolean | Add /usr/local/bin to secure path for sudoers. This is for Amazon Linux in AWS. |

Features

All operations have attributes to disable/enable them. Here's what is out of the box.

Defaults (Item )

  • Create Docker or AWS EC2 Instance
  • Install basic packages.
    • I put a few basic packages I like, but you can override the array.
  • Install CloudWatch Logs Agent if in AWS
  • Create Basic CloudWatch Logs Config if in AWS
    • Disable if going to deploy your own from your wrapper
  • Create nodeinfo shell script to output system info
  • Disable SeLinux
  • Add /usr/local/bin to sudo secure path in sudoers for Amazon Linux
  • Add a custom Bonus Bits Message of the Day (motd)
  • Set chefdk Ruby and gems in BASH profile

Optionals

  • Proxy Setup
  • Gem Source Change
  • Deploy Internal CA Certificate from Data Bag Item

Chef Config Files

There are Chef data bag items, environments, node and roles files under the test folder. These are called by the Test Kitchen configurations instead of entering all the attribute overrides inside the YAML. It cleans up the kitchen YAML, plus makes it easier to quickly understand how you'd setup these Chef configurations in your environment. Basically copy/paste, instead of converting YAML to JSON... I've done it for you!

test
├── data_bags
│         ├── encrypted_data_bag_secret
│         └── bonusbits_base
│             ├── credentials.json
│             └── enterprise_ca.json
├── environments
│         └── bonusbits_base.json
└── node
    └── client.rb
└── roles
    └── base.json

The data bag items are just examples. They are not currently used in the cookbook. I added it mainly for when I copy/paste to write a new wrapper it's already staged. I usually end up adding the customers CA cert chain as part of my base cookbook. Here's some information on how to accomplish that task.

CloudFormation

Prerequisites

Launcher

Click this button to open AWS CloudFormation web console with the Template URL automatically entered.

CloudFormation Tasks

Public S3 Link:
https://s3.amazonaws.com/bonusbits-public/cloudformation-templates/cookbooks/bonusbits_base.yml

The CloudFormation Template the following:

  1. Create Autoscale Group for service check (hardware failure) DR not Scaling (No ELB/ALB)
  2. Adds the EC2 Instance to selected Security Groups
  3. Create IAM Instance Profile Role
  4. Create Cloudwatch Logs Group
  5. UserData
    1. Installs some basic packages needed for bootstrapping
      1. cfn-init
      2. aws-cfn-bootstrap
      3. cloud-init
      4. git
    2. Run cfn-init
    3. Signal completed after Chef run (cfn-signal)
  6. Cloud Init (cfn-init)
    1. Configure CFN Hup and Auto Reloader Hook Conf
    2. Setup and Execute Chef Zero
      1. Install ChefDK from internet
        • Using ChefDK because already has berkshelf and InSpec etc.
      2. Create Chef Configuration Files
      3. Download bonusbits_base cookbook from Github
      4. Download depend cookbooks with berks
      5. Triggers Chef Zero run
    3. Warm EBS Volume

Test Kitchen

The default Kitchen configuration .kitchen.yml is setup with AWS EC2 and Dokken Docker Drivers.

If using Kitchen simple specify the test suite with the driver you'd like to use. Both driver gems are included with ChefDK. The only prerequisites are having Docker installed and/or AWS Environment Variables setup. Unless you are using a public subnet in AWS then you'll want a direct connect or VPN solution in place.

Command Examples

The kitchen commands need to be ran from the root directory of the cookbook.

Task Driver Command
List All Test Suites ALL kitchen list
List All EC2 Test Suites EC2 kitchen list ec2
List All Docker Test Suites Docker kitchen list docker
Test all Test Suites (destroy, create, converge, setup, verify and destroy) ALL kitchen test
Test all EC2 Test Suites (destroy, create, converge, setup, verify and destroy) EC2 kitchen test ec2
Run Chef on a Single Test Suite EC2 kitchen converge ec2-base-amazon
Run Integration Tests with InSpec on a Single Test Suite Docker kitchen verify docker-base-amazon
Login to a Single Test Suite That is Already Created EC2 kitchen login ec2-base-amazon
Login to a Single Test Suite That is Already Created Docker kitchen login docker-base-amazon

EC2 Requirements

  1. Direct Connect, VPN or public subnet

    • A direct connect or VPN solution must be in place from you to the AWS VPC where you plan to stand up EC2 instances.
  2. NAT or IGW

    • The subnet that you plan to stand up the EC2 instances may require internet access to pull from Github etc.
  3. AWS Credentials Profile Configured

    • Be sure to setup your AWS CLI profile even if you only have one. It's a more secure method to pass credentials to Test Kitchen.
    aws configure
    vim ~/.aws/config
    vim ~/.aws/credentials
  4. Environment Variables

    • Set these Environment Variables or use .kitchen.local.yml config to overwrite the values with static personal information.
    AWS Environment Variables
    -----------------------------------------
    AWS_SSH_KEY_ID = {aws ssh key name}
    AWS_SSH_KEY_PATH = {/path/to/sshkey.pem}
    AWS_PROFILE = {aws config profile name}
    AWS_REGION = {us-west-2 or us-east-1}
    AWS_VPC_ID = {VPC ID to deploy}
    AWS_SUBNET = {subnet ID to deploy}
    AWS_PUBLIC_IP = {true or false}
    AWS_SECURITY_GROUP_1 = {security group ID}
    AWS_IAM_INSTANCE_PROFILE_1 = {EC2 Instance IAM Profile Role Name}
    

    Setup to support up to 5 security groups. More IAM Profiles staged for multiple instance role support.

Docker Requirements

Dockerfile

There is a Dockerfile in the root of the cookbook that can be used to build a local Amazon Linux Docker Image using Chef Client / Chef Zero. It is setup to run the InSpec Tests at the end of the Chef Run for this type of deployment.

Usage

Build Local Image

docker build -f Dockerfile . -t bonusbits_base

Create Container

docker run --name amazonlinux_base -it bonusbits_base

Start Container (Optional)

docker start amazonlinux_base

Login Container (Optional)

docker exec -it amazonlinux_base /bin/bash --login

Cleanup

Stop Container

docker stop amazonlinux_base

Remove Container

docker rm amazonlinux_base

Remove Image

docker rmi bonusbits_base

Docker Image

There is an Amazon Linux Docker image built from this cookbook on Docker Hub if interested it can be found HERE

NodeInfo Script

You can run the nodeinfo script locally or use Test Kitchen to run it. You can have it run on one, multiple or all of the test suite VMs you have running. Below are some examples:

Example Output


---------------------------------------------------------------
Node Information
---------------------------------------------------------------
## NETWORK ##
IP Address:                 (10.80.0.221)
Hostname:                   (ip-10-80-0-221)
FQDN:                       (ip-10-80-0-221.us-west-2.compute.internal)
## AWS ##
Instance ID:                (i-0c32017a62a32ad3b)
Region:                     (us-west-2)
Availability Zone:          (us-west-2a)
AMI ID:                     (ami-d61a92b6)
## PLATFORM ##
Platform:                   (redhat)
Platform Version:           (6.9)
Platform Family:            (rhel)
## HARDWARE ##
CPU Count:                  (1)
Memory:                     (994MB)
## CHEF ##
Detected Environment:       (dev)
Chef Environment:           (bonusbits_base_epel_repo)
Chef Roles:                 ([base])
Chef Recipes:               ([bonusbits_base, bonusbits_base::default])
---------------------------------------------------------------

Cookbook Dependencies

Testing

  • Style/Linting
    • Foodcritic and Rubocop
    • Ran from Rakefile tasks by CircleCI
  • Integration

Resources

Setup DevOps Workstation

Test Kitchen

Chef