DevOps Adventure - Chapter 1

Reading time: 8 minutes



Checkout Hero Image Source

Hi again! Lately I was really into one big thing, which I was dreaming of making a reality. As the title states, I am building a DevOps lab! This idea have been around for the last year, after I got certified as DevOps & Software Engineering Professional issued by IBM. This sounded to me rather pompously, because even though I was playing around with many concepts through that course (including Kubernetes), I still wasn't feeling myself confident enough (spoiler: this feeling got worse after I started working on DevOps lab) to say, that I am indeed a professional in this topic. I wanted to try to create things from scratch. So I had decided to build something that would help me gain this confidence, knowledge and experience. Let's start!

DevOps Essential Components

First let's go through the list of essentials, that are required to build a proper DevOps lab. I've split these into two categories.

  • General components:

    • Linux (e.g. Debian, Ubuntu)
    • Cloud Environment (Hetzner.com)
    • Networking and Security (e.g. iptables, nftables, CrowdSec)
    • Monitoring and Logging Tools (e.g. Prometheus + Grafana)
    • Backup and Recovery (e.g. rsync)
    • Collaboration (e.g. YouTrack, Trello, GitLab)

  • DevOps specific components:

    • Development Environment (e.g. VS Code IDE, Coder.com)
    • Source Code Management (e.g. GitLab, OneDev, Gitea)
    • Continuous Integration/Continuous Deployment (e.g. Dedicated GitLab, OneDev, Gitea)
    • Containerization and Orchestration (e.g. Docker, Podman, Kubernetes or K8S, Docker Swarm)
    • Infrastructure as Code (e.g. Terraform, Chef)
    • Configuration Management Tools (e.g. Ansible, Chef)
    • Testing Frameworks (e.g. Python PyTest)
    • Artifact Repositories (e.g. JFrog Artifactory, GitLab)
    • Code Security & Quality (e.g. SonarQube, Snyk)
    • Notification Tools (e.g. mail server with MailCow)

General components include everything any infra should have, that's just a must and I won't go too much into details here, it is just not my cup of tea yet, but of course I will have to set it up myself. As per DevOps specific components then here it really becomes interesting, as the amount of tools you may use is really huge. Of course I was looking into the things, which I could use for free, that was the main pre-condition. Essentially, everything except Cloud provider should be free. Let's dive into the concept itself.

Conceptual Idea

So the main idea was to use a dedicated machine for this project and build a virtual environment with several VMs, where each of these would serve a specific purpose. When there will be a need, I would utilize additional resources by quick provisioning of such with IaC. Then definitely this environment should be based on Linux. The other thing is to use containers obviously, because these are definitely easier to maintain than the traditional environment with multiple different apps and I chose Docker as the most popular tool. As per everything else then it would just need to comply with the list of essential components I have already mentioned. Here is the diagram I created.

It is worth mentioning that while creating the diagram I was mostly struggling with answering the question on how to manage general access across my whole DevOps lab. I mentioned LDAP tool (OpenLDAP) suggested by someone on the net without knowing anything about how it operates. I mean I understand, there is some tool, which manages access to certain resources based on some group policies (like everyone knows what Active Directory is). I didn't know at the time, that this would be a pretty complex thing to setup and configure. So the moral here is that this was the concept, but in reality I did things differently without compromising security. Also there were a lot of questions regarding reverse proxy, whether it should be one or few of these? How should I route traffic to higher and lower environments? Should it be a separate route for each? etc. I still haven't yet found the answer, so I continue my research.

The Beginning

First I had to choose the fundamental thing, which was the hardware, where I would make my dream come true. Again I chose Hetzner.com as I can use server located in Europe, it has a great price and I never had any issues with it. So I went for a dedicated server (so called bare-metal server) with a code name AX41-NVMe. Not even close to an enterprise level machine, but good enough for small personal projects like mine. And no, they don't pay for mentioning them, I really just like them.

Next would be to understand, how to manage VMs. From my conceptual diagram you may state, that I haven't thought about it and that's true. I just remembered the time I played around installing Kali Linux on Windows with VMware Workstation and Oracle VirtualBox, and from the IBM course I learned about KVM, so I started looking for the tool, which would help me to manage my VMs in a user-friendly manner with GUI (dodging a tomato). My choice would land on Proxmox. I need to say that I am not even close to be a professional in this tool, but main concepts are pretty easy to understand and you have plenty of documentation on it. Managing VMs, backups, VLANs, Firewall etc. all of these features are really self-explanatory except for the hardware resource allocation (but the default options tend to be universal ones). Most probably in the future I will dig more into all the features of Proxmox, but it is not a priority right now.

Then I had to figure out, which VMs (and how many of those) I would like to set up. I haven't mentioned anything about it in the conceptual diagram, so I started thinking and here is what I came up with:

  • VM1-CON – would handle some network related stuff and be available from the outside;
  • VM2-GEN – required to act as a primary hub or a manager of everything else within a cluster;
  • VM3-LOW – lower environment machine, where the development and testing is handled;
  • VM4-HIGH – higher environment or production for my applications;
  • VM5-MISC – this is made just for experimenting, it also could be unallocated space for some smaller resources like LXCs (linux containers).

All of these machines are able to access external networks (essentially the internet) and they all are a part of the common VLAN via a virtual switch. There should also be a bridge between this VLAN and a network interface, which would lead us to our network interface controller (NIC) and to the global net from there. This part is also managed by Proxmox.

Basic Security

Definitely need to think about secure connection to my environment. I need to define some specific firewall rules on my dedicated server (actually you should do it before everything else!). Hetzner uses stateless firewall where you can define specific rules on the switch level to filter out packets based on their headers. You would want to allow main ports to be open for incoming connections (like 443, 22 and 80 if you really need it) and also ephemeral ports range (check these out) to allow temporary communication between peers, as well as your pre-defined VPN tunnel port. With general restrictive policy you would also want to allow incoming connections from dns and ntp ports (53, 123) to properly handle DNS queries and time sync. Additionally for ssh connections on port 22 I can set specific allowed IPs, just one additional security measure.

Next on my Proxmox node I have to disable ssh logins with passwords (you can just google it, nothing complex there). In order to do it I need to configure ssh keys first (won't dig into details here as well). Then I have to setup VPN and as my connection point I would use VM1-CON machine. I had hard times trying to make WireGuard work in a docker (mainly the issues were on how to properly configure traffic forwarding), so I just didn't bother and set it up the traditional way. I also had hard times trying to use VPN and access external networks. With Wireshark I have learned that my packets were being dropped randomly. For now I have solved this issue by lowering MTU and allowing incoming ICMP on my machine. I didn't find any clear explanation on why it worked, as I am not a network engineer myself, but I just assume that although using UDP for VPN is faster it causes issues when TCP connections are encapsulated, because packets are not guaranteed to be delivered and there is no error-checking mechanism in UDP. So when UDP packets are being dropped or fragmented (for any reason) it causes issues for these TCP connections, which were encapsulated, which means that external host may send ICMP packets to inform the sender about it in order to react accordingly. Since my machine was not reacting to ICMP, no action could be taken. No idea, whether this logic makes sense, but it definitely shows me, that I need to learn more things about networking.

Next Steps

So for the next steps I would need to start deploying my DevOps components. And for this I would prefer to write a separate article. In the next chapter of this Journey I will show you, how I have configured my VM1-CON machine. I will write about self-hosted DNS, reverse proxy, personal dashboards, identity management and docker compose. Learned quite a few things configuring that part. So see you soon!

Be curious, proactive and take care.

Best regards,

Ed

Previous Post