It was late, and I had just finished tuning a custom CUDA kernel benchmark dashboard for a fresh LLM fine-tuning run. The code was lean, the metrics were crisp, and all that remained was getting the web view into the hands of the team before morning standup.
Naturally, the classic software engineering brain rot kicked in.
“I should set up a sub-domain. Configure a Cloudflare CNAME. Spin up Traefik. Wire up Let’s Encrypt for automatic TLS cert generation. Write a systemd unit file. Maybe add basic auth while I’m at it.”
Two hours later, I was three tabs deep into Traefik v3 migration docs, debugging an ACME HTTP-01 challenge failure because of a botched DNS record propagation delay.
I paused. What am I doing?
The Setup
Here’s the context. I was running a fine-tuning job on a rented GPU box, a bare-metal server with a static IPv4 address. The training script spat out JSON metrics every epoch, and I had a small Python Flask app that served them as a live chart. Nothing fancy.
Three people needed to see it: me, my teammate, and our advisor. All of us were on the same internal Slack channel. None of us cared about a pretty URL. None of us would remember bench.internal.sujal.xyz any faster than 159.65.x.x:8080.
But my brain didn’t think that way at 2 AM. My brain thought: “A real engineer would do this properly.”
So I did what any self-respecting systems dev would do. I opened Cloudflare, created a CNAME record, pointed it at the server, and then realized I needed HTTPS for the browser to stop complaining. That meant a reverse proxy. That meant Traefik.
The Rabbit Hole
Traefik is a brilliant piece of software. I mean that genuinely. For production workloads with dynamic service discovery across Docker Swarm or Kubernetes, it is an absolute gift.
For a three-person internal dashboard on a throwaway GPU box? It’s a bazooka aimed at a mosquito.
But I was already committed. I pulled the Traefik Docker image, wrote a docker-compose.yml with the ACME Let’s Encrypt configuration, set up the HTTP-01 challenge, and waited.
level=error msg="Unable to obtain ACME certificate for the dashboard domain"The CNAME hadn’t propagated yet. Cloudflare’s proxy was intercepting the challenge. I toggled the proxy to DNS-only. Waited. Still failing. Turns out the subdomain was pointing to the wrong A record because I fat-fingered the IP address in the dashboard.
Fixed it and waited for the DNS change to settle. The cert finally issued.
Then the Flask app wouldn’t start because Traefik was already binding port 80. I had to configure the internal routing, add labels to the Docker container, and map the service.
It was well past the time I intended to stop.
I had spent hours setting up infrastructure for a dashboard with a short lifespan.
The Moment of Clarity
I stared at the terminal. The Traefik logs were scrolling happily. The TLS cert was valid. The CNAME resolved. Everything was technically correct.
And I thought about my C++ days. When I was doing low-level systems work, including building compilers, matching engines, and custom allocators, the mantra was always the same: measure, don’t assume. Profile before you optimize. Don’t add a layer of abstraction until the complexity demands it.
Somewhere between systems programming and DevOps, I had forgotten that lesson.
I killed the Traefik container. Stopped the Docker Compose stack. Ran one command:
nohup python3 bench_server.py --host 0.0.0.0 --port 8080 &Verified it:
curl http://159.65.x.x:8080/metricsJSON came back. Charts loaded in the browser. I pasted the raw IP URL into Slack with a one-liner: “Dashboard is live, bookmark this.”
The working solution was up almost immediately.
Why We Over-Engineer
I’ve thought a lot about why this happens. Not just to me, but to every developer I’ve worked with on infra.
We confuse professionalism with ceremony. Setting up Traefik, writing Terraform modules, and configuring Nginx with proper SSL termination can feel like real work. Pasting a raw IP in Slack feels sloppy. We’re trained to believe that production-grade tooling is always the right answer. But not every problem is a production problem.
We optimize for a future that may never arrive. “What if this dashboard needs to be public later?” What if it does? Then I’ll set up the CNAME at that point. The cost of deferring is near-zero. The cost of premature infrastructure is hours of my life late at night.
We treat internal tools like external products. Internal tooling has a different SLA. It has a different audience. It has a different lifespan. A benchmark dashboard that exists for three days does not need the same infrastructure as a customer-facing API.
The Decision Framework
After that night, I started asking myself three questions before touching any infra:
1. Who is the audience? If it is a small internal group and the service exposes only low-risk, read-only data, a direct IP may be enough. That is a scoped choice, not a default. The URL still needs to stay within a trusted access boundary, and the absence of TLS is not acceptable once credentials, sensitive data, or untrusted users enter the picture.
2. What is the lifespan? If the thing has a short lifespan, that may justify deferring a domain name or reverse proxy. It does not waive TLS, authentication, or network controls when the data or audience requires them.
3. What’s the blast radius of getting it wrong? An internal telemetry dashboard serving read-only JSON to a few trusted engineers had a limited blast radius in this incident: no passwords, no sensitive PII, and no public audience. Even there, a bare HTTP port would only be reasonable while that boundary held. Accidental exposure, port scans, or a change in the data would change the decision.
When the audience is trusted, the data is low-risk, and the service is genuinely short-lived, I consider a direct IP and document the boundary. If any of those conditions change, I add the security controls before sharing the URL.
When Correctness Matters
I want to be clear: I’m not advocating for sloppy infrastructure everywhere.
When I built my email system replacement from scratch, HTTPS was non-negotiable. That was a public-facing system handling user data. When I designed the matching engine for BetaTrader, thread safety wasn’t optional, because the entire architecture depended on partitioned single-writer guarantees.
Production systems deserve production infrastructure. User-facing services need TLS, proper DNS, health checks, monitoring, and all the ceremony.
But internal dev tooling is not production. Conflating the two is how you end up spending your best engineering hours on plumbing that nobody will remember in a week.
The Broader Lesson
This isn’t really about Traefik or CNAME records. It’s about a mindset.
In systems programming, we talk about premature optimization as the root of all evil. In DevOps, the equivalent is premature infrastructure. Building the deployment pipeline before you’ve validated the thing being deployed. Writing the Helm chart before the Docker image is stable. Configuring the monitoring stack before you know what metrics matter.
The best infrastructure decision in that incident was not a clever Kubernetes config or a well-tuned Terraform module. It was typing nohup ./server & and going to bed.
Good enough was right for that bounded incident. It would not be the rule for an exposed service.
The Checklist I Keep Now
For anyone who’s been bitten by the same bug, here’s my quick gut-check:
- Is this internal and limited to the intended users? A direct IP may be enough for a low-risk read-only tool, but record the boundary.
- Will it be short-lived? Defer a domain or reverse proxy if that reduces work, but keep the controls the data and audience require.
- Does it handle sensitive data or credentials? Use TLS and authentication, and check the network exposure.
- Is it user-facing or reachable by untrusted users? Use the proper DNS, HTTPS, proxy, and monitoring setup.
The goal isn’t to avoid doing things right. It’s to avoid doing things right at the wrong time.
Use the smallest safe setup. Skip the rabbit hole. Sleep before standup.