Cutting release cycles 40% with containerised delivery
Dockerised CI/CD onto Kubernetes for cloud-native Python and Node services, plus a data-model redesign that doubled pipeline throughput for AI processing.
- Role
- Senior Software Engineer · BROCENT
- Stack
- PythonNode.jsDockerKubernetesSQLNoSQLCI/CD
- 40% faster release cycles
- 2× data throughput after the pipeline and schema redesign
- Testing strategy and engineering standards adopted team-wide
The problem
Releases were slow for the ordinary reasons. Environments differed enough that "works locally" carried no information. Deployment was a checklist someone followed by hand, so it happened rarely, so each one carried more change and more risk — the familiar loop where deploying is scary, which makes you deploy less, which makes it scarier.
Underneath that, the data pipelines feeding AI processing were the throughput ceiling. Adding capacity meant adding machines, because the models the queries ran against were not shaped for the queries being run.
Delivery
Containerising the services removed the environment question, and putting them on Kubernetes removed the by-hand deployment. Neither is novel; what mattered was the order.
The pipeline stages were kept deliberately boring and identical across services: build once, test the artefact, promote the same artefact through environments. A build that reaches production is byte-for-byte the one that passed tests, which eliminates a class of "it passed CI though" incidents entirely.
Throughput
The 2× came almost entirely from modelling, not from hardware.
The original schema stored AI processing records in the shape they arrived in. Every downstream query then reassembled them, so the expensive work happened on every read instead of once on write. Splitting the write path from the read path — normalised for ingest, denormalised projections for the queries that actually ran — is unglamorous and doubled throughput on the same infrastructure.
Batching was the other half. Per-record round trips to the database dominated the profile; batching writes and reads to the pipeline's natural chunk size cut that overhead sharply.
The part that outlasted me
Alongside the infrastructure work I mentored engineers on the team and wrote down the standards we had been carrying informally — what gets tested, what blocks a merge, what an on-call handover contains. The delivery numbers were the visible outcome; the standards are the reason they held after the project ended.
Outcome
Release cycles 40% faster, data throughput doubled, and a deployment process routine enough that shipping stopped being an event.