DevOps & CI/CD

The value of a release pipeline is not measured by how fast you can ship. It is measured by how many minutes it takes to get back when a bad version reaches production. A pipeline you cannot reverse is not automation; it is a mechanism that spreads mistakes faster. That is why the first step we write into every pipeline is not deployment but rollback: which command, whose permission, and how you return to the previous version without touching database migrations. A team without a written answer to that question accumulates risk as it increases release frequency. The sections below cover the parts of a pipeline, environment and secret management, the container side, observability, and how the pipeline is operated after handover.

The point of CI/CD is safe rollback, not speed

CI/CD, short for continuous integration and continuous delivery, describes the system that tests code changes automatically and ships them to production; but this automation does not lower the error rate, it only changes what an error costs, because the same mistake happens in a system deployed by hand and the real difference shows up in what can be done afterwards. Three questions decide whether a pipeline is worth anything: can the application version return to the previous image with one command, is the database change written so it can be reversed, and on release channels that cannot be rolled back, such as mobile, can the feature be switched off behind a flag. The three answers are not the same. An application version comes back in minutes. A migration that drops a column does not, which is why the schema is widened first and narrowed in a second, later step. A mobile build published to a store is not rolled back at all, so it ships behind a switch. Lextum AI is the product-side version of the same principle: every document change on the platform is recorded so it can be traced and reversed, and who changed what and when stays visible. What we expect from infrastructure is no different.

What a release pipeline is made of

The six headings below cover every decision made while a release pipeline is built. The first three are the pipeline itself: the steps that run from build to deploy, how environments are kept apart, and the package format the application travels in. The next two are what surrounds the pipeline: the measurement layer that shows what happened after a release, and who is allowed to reach what. The sixth is where the other five are tested, meaning what actually happens when things go wrong.

Pipeline anatomy: build, test, image, deploy

A release pipeline has four steps, and every step holds the right to stop the rest. The build step proves the code stands up in a shared environment, not on one developer's laptop. If the test step is red, nothing moves forward; a test that can be skipped is a test that does not exist. The image step turns the output into a single versioned package, so what was tested and what goes live are the same artifact. The deploy step carries that package to a target environment. This separation is the backbone: once package and environment are split, the same image runs through test, staging and production in order, with no rebuild per environment.

Environment management and configuration

The difference between environments belongs in configuration, never in code. The application reads which database to connect to, which service address to call and which feature is on from environment variables. A branch inside the code that checks the environment name is a path that will run for the first time in production. We set up at least two environments: a staging environment that matches production in schema and migrations, and production itself. Staging differs from production only in data and scale. Configuration is versioned like code, because the cause of an incident is more often an unrecorded setting change than a code change.

Containers and orchestration

Containers exist to retire the sentence "it worked on my machine". Application, runtime and dependencies merge into one image, and that image opens the same way on a developer laptop and on a production server. Orchestration is a separate decision and is not required on every project. For a single-service application with predictable traffic, Kubernetes adds more operational weight than it returns. It earns its place when several services scale independently, deploys must happen without downtime, and automatic recovery is expected. If the real problem is load rather than the pipeline, the work moves to Backend Scaling, where architecture, data access and caching are the subject.

Observability: logs, metrics, alerts

"Looks fine" is not a measurement after a release. We build three layers: structured logs, error and latency metrics, and alerts that actually reach a person once a threshold is crossed. If an alert reaches no one, the dashboard is decoration. Marking each release in the same data belongs to this layer too. When the error rate climbs, the first question is which version it started after, and if the answer is not visible on the chart, the search stretches into hours. We keep alert thresholds few and set them where they demand action, because an alert that fires constantly gets muted.

Secret management and access

A key that has entered the repository stays in the history even after the line is deleted, so the only correct move is to revoke it and issue a new one. Database passwords, payment provider keys and cloud credentials live in a secret store the pipeline reads, not in the code repository, and they are separated per environment: a staging key must not reach production data. Permission to deploy to production is assigned by name and leaves a record. The goal is not to reduce the number of people who can deploy, but to make it readable later who deployed and when.

Rollback and incident response

Rollback is not a plan written during an incident. It is an ordinary step of the pipeline and it is exercised outside incidents too. The previous image is kept ready, the deploy command takes a version parameter, and going back is a single action. The database side is treated separately: migrations are written with both a forward and a backward direction, and steps that destroy data are deferred to their own release. During an incident the order is fixed: roll back first, look for the cause second. The reverse order is the expensive one, because diagnosis on a broken system is slow and made under pressure. A short written record afterwards is the only thing that stops the same failure returning for the same reason.

How release frequency relates to fragility

Releasing often does not make a system fragile. Releasing big does. What sets the risk is not the number of deploys but the size of the change that travels at once. When a release that accumulated for weeks breaks, nobody knows which change broke it, and rolling back also removes dozens of things that worked. With small and frequent releases, what gets reverted is obvious. The precondition is that deploying stops being an event; if a team has to gather for every release, frequency was never available. This matters most in systems that cannot afford downtime. Anneekspres is a marketplace: buyers place orders while sellers update stock, so a planned maintenance window is not a luxury that exists. The project was built with a RabbitMQ message queue, PostgreSQL for data management and Jenkins pipelines, and high availability and fast deployment come out of that same structure.

The benefits of managing infrastructure as code

A server built by hand becomes an undocumented system the day the person who built it leaves. Writing infrastructure as code moves the answer to "what is on this server" out of human memory and into the repository. With tools such as Terraform, server, network, security group and database definitions get versioned; a change is reviewed as text first and applied second. The gain shows up in three places. First, repeatability: standing up a second environment drops from a week of work to a single command. Second, auditability: when a setting changed and why stays in the record. Third, the disaster case: "if the server is wiped, how many hours until it is back" gets a measured answer instead of a guess. The only way to confirm all three is to build the infrastructure from nothing at least once.

Handover: how you operate the pipeline

A release pipeline is yours only if it runs in your accounts. The repository, cloud account, domain and secret store are opened in your name; we take access, not ownership. Four things are delivered in writing at handover: what the pipeline does step by step, how a new developer goes from a clean machine to a release, what the rollback command is, and who holds deploy permission for which environment. The on-call side is added to that: which alert reaches whom, and what is done in the first twenty minutes. Documents alone are not enough, so before handover we run at least one real release and one real rollback together with your team. We wrote about the engineering approach of working close to the customer's own systems in our Forward Deployed Engineer article; handover follows the same logic.

Our tool selection criteria

Tooling is chosen by where the project already lives, not by habit. If the code sits on GitHub and nobody wants to operate a separate runner machine, GitHub Actions is the shortest route. For pipelines that run on your own server, take a long time, or contain custom steps, Jenkins remains a reasonable choice: in six of the nine references attached to this service, CI/CD was built with Jenkins and the server side runs on Ubuntu. In the remaining three the technology list is different, because we do not copy the same pipeline everywhere. Docker removes the gap between environments and becomes the default once containerisation is decided. Kubernetes and Azure depend on justification: several services, independent scaling, or a corporate cloud policy. Where the system is hosted is your decision rather than ours; the managed SaaS infrastructure of Lextum AI was built on AWS, while other projects went elsewhere.

How this has worked in practice

The work below is live or completed. The detail of each one sits on its own page in the Success Stories section.

• Lextum AI: Microservice architecture with asynchronous processing over RabbitMQ; a Next.js interface, .NET services and a managed SaaS infrastructure on AWS, with CI/CD automation on Jenkins. The structure is ready for high-traffic enterprise use.
• Anneekspres: Marketplace infrastructure built with a RabbitMQ message queue, PostgreSQL for data management and Jenkins pipelines, delivering high availability and fast deployment.
• Biletico: Redis caching on a Next.js and Node.js stack, with CI/CD automation on Jenkins pipelines; every update is tested before it goes live.
• Welldone: A Flutter mobile application, a React management panel and .NET services, on scalable Ubuntu-based server infrastructure with CI/CD automation on Jenkins. The whole operation from order to shipment runs on one platform.
• World Summer Schools: Cloud-based scalable infrastructure on a Next.js, .NET and PostgreSQL stack, with CI/CD automation on Jenkins. The infrastructure is ready for growing program and user volume.
• BiTalih: A content production platform running on Next.js and MongoDB, with release infrastructure on Jenkins and Ubuntu.