In high-performance IT delivery (such as DevOps) the implementation of the complete IT-system is a responsibility of the team, this includes the implementation and configuration of the infrastructure. To make use of all benefits described in the previous section, it is recommended to define the infrastructure as code and apply the same quality engineering principles as done on application code.
Infrastructure verification is key when you make use of Infrastructure as Code (IaC), without automated tests IaC must be considered broken. The purpose of verifying the infrastructure is similar as it is for application source code.
Verification (and also validation) of the Infrastructure takes place regarding all fundamental DevOps activities. The following sections briefly describe this per fundamental DevOps activity.
Examples of these kind of verifications are:
- The operation system in
- Status of databases (up or down)
- Accessibility of servers (firewalls, port numbers)
- Size of internal and external memory
Plan
Does the infrastructure architecture adhere to key quality characteristics? For example: security and maintainability, but also characteristics like fault-tolerance, fail-over, scalability and resilience should meet the desired requirements. The planning phase is the phase were the architectural design should be verified against policies and industry regulations. During the planning phase preventive quality measures are used to prevent insufficient quality.
Code
Transforming the architectural design into code is an agile process. The architecture is divided into small, reusable, flexible, easy codable and testable modules. The modules and also IaC should be verified on different aspects.
- Syntactical correctness
- Code formatting (linting)
- Code testing (deep linting)
- Accordance with company or team policies and industry regulations
With linting you make sure that all code uses the same formatting and styling regardless which team member writes the code. On top of that, deep linting can be applied which is called ‘code testing’. Code testing is a technique to verify the IaC on the correctness of the types that are used. E.g. AWS uses instance-type for their computing machine. The IaC should not use unavailable, incorrect, incomplete or outdated types.
In accordance with the stages in the build pipeline dynamic testing is also part of testing the IaC, State testing is the test variety which is performed on the IaC. The purpose of state testing is verifying the IaC in a runtime environment, so the infrastructure is actually deployed on the target platform. State testing the IaC is not related to the state transition testing technique.
State testing the infrastructure is done in 3 steps:
- Deploy real infrastructure which needs to be tested
- Verify basic attributes of the infrastructure:
- Assigned compute power (CPU’s/cores)
- Assigned internal memory (RAM)
- Assigned diskspace
- Version of OS (Eg.: RHEL8, MS Window Server 2019)
- Patch level of OS
- Type of database (Postgres, MySQL)
- Undeploy the infrastructure under test
- Verify that deployed IaC is completely removed
By state testing IaC the code is verified (and validated) in a real environment which eliminates the need for mocks, stubs or other artificial components. This testing type can be slow for large infrastructures and potentially also be very costly.
It’s advised to perform the state test in an isolated ‘sandbox’ environment to prevent interference with existing infrastructure.
Detective quality measures are predominantly used during the coding activity to detect a possible insufficient level of quality.
Integrate
The most extended test for IaC is the integration test. During this test the IaC is tested including the deployment and configuration middleware components. Middleware components in the infrastructure can occur in many forms, for example: application servers, web application, firewall, database engines and web servers.
Attributes which should be assessed during this test are (non-exhaustive):
- Is server listening on correct port?
- Are security certificates correctly installed?
- Are the firewall rules implemented? (Thus, can server A connect to server B, or the opposite, is the connection from server A to server B denied.)
- Are databases correctly configured? Listening to correct port? Is naming of schema correct?
- Are proxy rules configured?
Deploy
Deployment of the IaC takes place at several stages. During the team test stage and the business test stage the IaC is verified in combination with the application under test.
Verification during the deploy activity has a focus on stability, security, performance, resource consumption. Relevant test varieties are smoke testing, performance testing and security testing. All tests are performed under close watch of monitoring-tooling to detect resource consumption.
The last deployment in the iteration is to the production environment. During the deployment activity detective quality measures are used.
Operate and Monitor
During the last activities of the DevOps cycle the complete IT-system including the IaC is live and used by the customer. The IT-system is monitored on several aspects and feedback is collected from the IT-system. This feedback is used to continuously improve the IT-system. Corrective and detective quality measures are used to detect and improve lack in quality.
----------
Sources
https://itnext.io/principles-patterns-and-practices-for-effective-infrastructure-as-code-e5f7bbe13df1
https://blog.gruntwork.io/5-lessons-learned-from-writing-over-300-000-lines-of-infrastructure-code-36ba7fadeac1
https://medium.com/faun/terraform-acceptance-testing-the-basics-5450d35a4421
https://www.infoq.com/presentations/iac-terraform-testing/
https://www.thoughtworks.com/radar/techniques/infrastructure-as-code
https://www.terraform.io/
Overall Building Block
Related Building Blocks