The need for DevOps engineers has been growing in recent years, although professionals are not easy to find. The main task of DevOps is to maximize the predictability, efficiency, and security of software development. The methodology (development + operations) originated in 2009 to establish interaction between programmers and system administrators to increase the release frequency. In essence, DevOps engineers work at the intersection of these two professions and are engaged in automation. They are involved in the development, testing, and release phases.
The main responsibilities include deploying the release of products, integration and deepening of development processes into delivery, standardization of development processes, setting up the infrastructure in accordance with software requirements, and process automation. Of course, the greatest burden falls on the DevOps engineers during the preparation of the infrastructure for the application, as well as setting up the CI\CD process, which then works in automatic mode. DevOps uses various tools for configuration management, virtualization and automation of operational processes, and the use of cloud technologies. To keep up with the rapid development of technology, DevOps engineers must constantly learn, be focused, and diligent.
Today, we will reveal some of the important trends that DevOps should definitely consider in 2022.
Terraform is a tool by Hashicorp that helps declaratively manage infrastructure. Thanks to this technology, you do not have to manually create instances, networks, etc. in the cloud provider console, but simply write a configuration that will set out how you see the future infrastructure. This configuration is created in text format. If you want to change the infrastructure, then edit the configuration and run terraform apply. Terraform will direct API requests to your cloud provider according to the configuration specified in the file.
Suppose we have an infrastructure of 1000 computers in the cloud. Computers are connected by certain network equipment. These 1000 computers can be deployed with a single terraform apply command by previewing the terraform plan. It will take about 10 minutes from start to finish, and the infrastructure will go up.
Or let's say we want to create a web server that would give us a home page upon www.arizon.page request. The process of creating the infrastructure (deploy the instance, install the web server, configure the web server) can take up around 3-5 minutes, and we will get a ready-made infrastructure with all the functionality and engines.
One or more people can work with Terraform. As a rule, development is conducted by a group of people. One is working on one feature, the other is working on another, and so on. When it's time to deploy, we use the terraform apply command.
Thanks to the lock mechanism at a time when one of the teammates is deploying their part of the infrastructure, no one else can do it. The lock mechanism works with a tf.state file that is stored remotely (aws s3 bucket or something similar). If you do not use the remote tf.state file and do everything locally, then your teammates will not know about the changes you have made in the cloud and can simply delete/change them. When everyone is working with the same tf.state file, the whole team will have the relevant state of the infrastructure. When one of the teams deploys, everyone else is shown an indicator that you cannot deploy because another person is working with it. When the deployment process is complete, Terraform automatically configures unlock, and others can already work.
Technology is constantly evolving, supplemented by features and modules, becoming more interesting and complex in terms of architecture. Terraform in the DevOps engineer toolkit is a must-have.
Unfortunately, not everyone covers the infrastructure with code and thus puts themselves in danger. Therefore, in a system failure situation, DevOps is at the center of the fire. Everything goes into "downtime" — networking is stopped, instances on which web servers are run, and, as a result, money is lost. At the end of the day, you realize how important it is to cover the infrastructure with code.
With the help of two terraform plan\terraform apply commands, the infrastructure will be restored to its original operating state within minutes while debugging can take hours. The first command (terraform plan) will show the changes that have been made in the infrastructure, while the second (terraform apply) will make the changes that were issued by the first team. This is the whole power of Terraform (Infrastructure as code). There are other technologies that cover the infrastructure with code, such as Cloud Formation in AWS, Pullumi in Python, and others. The basic rule is that the infrastructure must be covered by code. In case of manual changes, Terraform will show the state you need to return the code to. But manual changes should be avoided and everything should be done directly in Terraform, simply because that is exactly what it is designed for.
DevOps Trend #2 - Cloud Technologies
Another trendy technology in DevOps toolbox is Cloud Technologies, which provides fast network access to a system of computer resources, including cloud storage, and database to the extent and for the time required for your needs.
Some companies do not trust their confidential information to AWS, Google Cloud, or Azure. They store everything locally and maintain their own servers. But if due to unpleasant circumstances, hardware is taken away with the environment which is run and brings money, they find themselves in the "downtime" status — you should always keep that in mind. Some companies may not be allowed to use cloud technologies, such as state-owned companies, but that's a whole other story.
Cloud technology is also profitable in terms of saving money for renting a place where you would have to store computers, money for electricity and staff compensation, and more. When using cloud resources, we only pay for what we use. If you need an instance of a certain capacity for a long period of time (half a year, a year, two, etc.), you can also save money. The longer the period of time — the greater the discount.
AWS and various cloud storage are very efficient in terms of flexibility. They can automatically add power at peak times.
Think of a situation during Black Friday when there is a peak load on the server.15-20 servers can work at night, and in the morning when the traffic decreases, the number of servers is reduced to 4.
Therefore, thanks to cloud storage you don’t have to overpay for extra capacity when peak loads are over. The work of the optimal number of servers that serve usual traffic is actually charged.
DevOps Trend #3- Docker
Docker is one of the most well-known tools for working with containers. This technology allows you to raise the working application in minutes. And we no longer need to create a virtual machine, install an operating system on it, and install the necessary components on the operating system to run the application. At the moment, all we need is a Dockerfile (instructions for running the application inside the container), in which we specify which image (docker image) we need to use, as well as which additional components are needed for the application, and so on.
Docker uses only those services that are needed to run the application and no more, unlike the operating system, which requires many additional services just to get it up and running.
With the Docker container, both the developer and the tester can quickly test the code locally. Because Docker is the same for everyone, we can be assured that the application will work the same on the developer’s, tester’s, and client’s sides.
It often happens that everything works on the part of the developer, but when the application gets to the testers — it does not work quite correctly, because the environments are different (disparate versions of installed packages, etc.). To save valuable time and avoid flipping tickets between developers and testers — Docker is indispensable.
DevOps Trend #4 - Kubernetes
Kubernetes is a Docker container administrator, or a complete container orchestration system. This is a Google development designed as an open-source solution for automatically deploying, scaling, and managing containerized applications. Recently, most applications have been developed as container-level microservices.
Container technologies help in the daily updating of applications for support of uninterrupted work of service 24/7. Kubernetes is a solution that allows applications to be updated and run anytime, anywhere by orchestrating containers.
Imagine that a website is hosted on a certain container. Kubernetes makes sure the container runs. If the Docker container collapses for some reason, Kubernetes will create the same new working container according to a certain template, but in that case, there will be a specific "downtime" status. Therefore, it is recommended to run at least two containers that perform identical functions at the same time. In this case, if one container collapses, the other will be available while the other is deployed — this is how we avoid the "downtime" status. In this way, Kubernetes can monitor hundreds of services that run at the same time. Kubernetes resembles an octopus — it has one center and many tentacles aka services.
In summary, there are several advantages of Kubernetes: 1) easy scaling of container applications; 2) easy migration — it is very easy to transfer container applications from local machines to the cloud storage for further deployment; 3) self-government — containers are reloaded, changed or destroyed automatically; 4) Secure Deployment — Kubernetes automatically updates applications by analyzing their status.
DevOps Trend #5 - GoLang and Python
There is no standard set of technologies that a DevOps engineer must have, but he must know one of the programming languages. This task can be solved in several ways — C ++, Python, Java. Knowledge of at least one programming language is a condition for the task. In 2022, a programming language such as GoLang is gaining in popularity among DevOps engineers.
GoLang is a programming language developed by Google and becoming a popular technology. In 2019, it was included in the list of fastest-growing languages. According to 2022 data, Go ranks 14th in the world ranking of popular languages and 10th among Ukrainian programmers as per the . Thanks to this language, you can quickly and easily create a product. Usually, programs based on this technology run faster.
Go has many libraries that work with a particular application. For example, you can call a library that will work with Amazon. With the help of the program written on GoLang, you can disable all services on your Amazon account that do not correspond to the current state (of course, if you have enough rights to do so). As we said before, this can also be done with another programming language, such as Python.
Having mastered one of the technologies, you can go on and study another. It is important to understand the logic of technology and learn the syntax gradually.
DevOps Trend #6 - Jenkins\GithubActions
Jenkins is a very flexible system written in Java and allows you to continuously integrate and deploy any level of complexity. You can use it to test the code and detect possible errors.
The pipeline is configured in a declarative or scripting style in Groovy, and the configuration file (Jenkinsfile) is in the version control system along with the source code.
Jenkins tunes in to the challenges developers and testers need. To do this, open the Jenkins UI — there is a list of jobs. We select the desired task, set the parameters, and click run. After that, Jenkins goes to Github (the repository where the developer ran his code), downloads the code, and starts building. If Jenkins detects a syntax error in the process, the developer can look at the Jenkins Console or get a message in the messenger that something went wrong, fix the error, run new code, click build and check the result of Jenkins. There are many different plugins for Jenkins that help with this or that feature. One of them is the "GitHub" plugin, which allows Jenkins Job to be launched after the code has entered a GitHub branch.
To sum up, there are several important features of Jenkins:
- Continuous Integration and Continuous Delivery
Jenkins can be used as a simple CI server or turned into a continuous delivery center for any project.
Jenkins is a standalone Java-based program, ready to run immediately with packages for Linux, macOS, and other Unix-like operating systems. Jenkins can also be launched in a Docker container.
With hundreds of plugins, Jenkins integrates with virtually every tool in the chain of continuous integration and delivery tools.
Jenkins can easily distribute work across multiple machines, helping to create, test, and deploy faster across multiple platforms.
Jenkins is free software, but the downside is the lack of technical support. An analog of Jenkins is GithubActions, which is a product of GitHub and was developed only a year or two ago. With GitHub actions, you can start builds right in the git. The advantage and convenience are that you do not need equipment and maintenance to run builds. You can configure the trigger build (git tag, create a pull request, push to a specific branch, and so on). Instructions for GitHub are written in .yml format.
Modern DevOps, just like a system administrator, needs to know the command line, because most systems are Linux.
It is impossible to remember all the commands, but it is vital to know a certain algorithm of actions. Suppose you need to change the firewall rule in the file. When you open the document, a canvas of different rules will appear in the IP table. If there are up to 500 of them, it is inconvenient to view manually, you need to create a pipeline from the teams that will make the necessary changes.
Even if you are running Windows, where a crash may also occur and a black screen with text will appear in front of your eyes. To fix the situation, you need to view the logs, and this can be done using certain commands or a text editor in Terminal.
DevOps engineers should navigate Linux systems like a duck to water.
The convenience of Linux-based systems is that they do not require a graphical shell that takes up resources. To work in a Linux-based system, a command-line is enough, where all manipulations in the system are performed.
Sooner or later we have to find out why this or that service does not work. To do this, you need to build a clear chain of the debugging process. If the service does not start — you need to review the logs, and if you see errors in the logs that require certain actions — perform them, and so on. To get the tasks done, you need a certain body of knowledge and experience that is gained in practice. If you have a problem today, you can spend 2 hours solving it which is fine, but you can handle it in 2 minutes tomorrow. This approach develops skills. And not only with Linux systems, but this is also the process of any practice.
Automation is one of the DevOps engineer’s roles. If you need to do something several times, then the process requires automation.
Bash scripting is a command-line script written for the bash shell, which is a powerful way to automate frequently performed actions.
Command-line scripts are sets of the same commands that can be entered from the keyboard, compiled into files, and combined into a common goal. Team results can be used as input for other teams.
For example, if you have to create a folder with a file and the current date several times in different places, you can write a script in a text editor that will do it automatically. First, the actions are prescribed in a certain order, which would be performed in the Terminal, after which the file is saved with the extension sh. To run the script, you need to run the command sh script.sh
Scripting is very convenient, but still, remember that it is a machine that can create, edit and delete. Therefore, if you use scripts for removal, it is better to spend time on testing and be sure that only the information you planned to delete will be deleted, and not any other. This way you avoid unwanted loss of important information, which will lead to unpleasant consequences.
As you know, if you have the information — you have the money. If information is lost, so is money. Based on this, backups are a very important factor in access to and integrity of information. Because people work with data, the human factor works. Sometimes misunderstandings between people or insufficiently tested code can easily erase information. In these cases, it is crucial to have backups. Yes, you may not have the latest information, but it is better to have information until yesterday and not have it for the last day than not to have data at all.
Monitoring is a guarantee that your system is working properly and all operations are being performed correctly. If suddenly something goes wrong — the monitoring will be the first to know and notify you in a way convenient for you. Therefore, you can be sure that everything is fine until there are no appropriate notifications. Thanks to monitoring, the downtime status can be reduced many times over, as the notification arrives seconds after it happens.
DevOps Engineer Soft Skills & Hard Skills Requirements
Because the DevOps Engineers work at the intersection of developers, testers, and the operations team, they should develop both hard skills and soft skills. Here are some important tips to bear in mind.
- Develop flexibility in working with a team
Effective communication, patience, flexibility, responsibility, resilience, and the ability to find approaches to different specialists are among the most important soft skills. Flexibility is one of the most essential features. Programmers do not always respond to a request, and often have to wait or be reminded several times. It is important not to panic, but to adjust to the team.
- Attentiveness at the stage of production
DevOps engineers should learn a lot, and during training, no one is safe from mistakes. It is inadmissible to make a mistake at the stage of production. If it is not possible to test the product locally, you can create your own AWS account for a fee, where you can do whatever you want. You can experiment and test on dev, but not on production.
- In the early stages, it is important to practice a lot
Not only theory but also practice is important to a young DevOps engineer. Before deploying the code, you should check several times and make sure that after pressing enter everything will go according to plan. Fatal mistakes can be made without testing, which later might have a negative impact on the career of a DevOps specialist.
- Ask before you delete something
DevOps engineers have a wide spectrum of authority. They have rights that developers do not have. For example, deleting the entire database. Before deleting something, it is necessary to clarify and ask colleagues to avoid making a fatal delete. In this work, it is important to have an analytical mind and develop resilience. Never give up even if things don't go as planned.
- It is important to have a good mentor and learn a lot
It is best to have the opportunity to learn from a more experienced DevOps engineer. At one time I worked with a good mentor who set tasks for me, helped me learn certain technologies, checked the completed tasks, and gave advice on growth. In addition to the mentor, you can also take training courses. It is important to constantly master new technologies, learn a lot and follow trends. In the past, I was given a fishing rod, rather than a fish. Faced with a certain task, I first googled the documentation, looked for answers, and if I still could not do something on my own. And only after that did I turn to a mentor who could always guide me.
- Possible development paths for a DevOps Engineer
Most DevOps engineers are system administrators who have studied programming tools in the past or developers who have delved deeper into operational processes. In any case, it is necessary to have basic technical education and understand the issues related to system administration and automation of various tasks. In career development, just like any engineer, DevOps passes the levels — junior, middle, senior, and lead.
Regardless of the chosen development vector, every DevOps engineer in 2022 must master the important rule of Life-Long Learning. The more technologies you know, the more demanded specialist you become in the market.
In this article Ihor Kanivets, Advanced DevOps at , tells about the role of DevOps engineers, their responsibilities, growth opportunities, a set of important soft & hard skills, and most importantly, DevOps trends in 2022.
