DEV Community

Sergey Shinder profile picture

Sergey Shinder

I am Sergey Shinder

Joined Joined on 
We looked up our new hostname four minutes too early and our resolver remembered that it did not exist

We looked up our new hostname four minutes too early and our resolver remembered that it did not exist

1
Comments
2 min read
We added one nullable column and every pod failed its order lookups until each connection had seen the error

We added one nullable column and every pod failed its order lookups until each connection had seen the error

1
Comments 1
2 min read
Our carrier switched to American date order and we only noticed on the thirteenth

Our carrier switched to American date order and we only noticed on the thirteenth

1
Comments
2 min read
Our new paging route for payments sat below a broader one and Alertmanager never reached it

Our new paging route for payments sat below a broader one and Alertmanager never reached it

1
Comments
2 min read
We autoscaled our Java pricing service on memory and it grew to twenty pods and stayed there

We autoscaled our Java pricing service on memory and it grew to twenty pods and stayed there

1
Comments
2 min read
Ansible changed our feed service's unit file and the restart kept the old one

Ansible changed our feed service's unit file and the restart kept the old one

1
Comments
2 min read
Our Node 20 image crashed on start because Docker copied the build agent's node_modules over its own

Our Node 20 image crashed on start because Docker copied the build agent's node_modules over its own

1
Comments
2 min read
Our reports hung for fifteen minutes every morning on connections the firewall had already forgotten

Our reports hung for fifteen minutes every morning on connections the firewall had already forgotten

Comments
2 min read
We deleted our old reminder CronJob from Git and Argo CD kept running it for six weeks

We deleted our old reminder CronJob from Git and Argo CD kept running it for six weeks

Comments 1
2 min read
One customer's price list grew past four megabytes and our gRPC client refused to read it

One customer's price list grew past four megabytes and our gRPC client refused to read it

Comments
2 min read
Our order graph went flat at peak because the kernel was dropping StatsD packets

Our order graph went flat at peak because the kernel was dropping StatsD packets

Comments
2 min read
Our snapshot init container asked for six gigabytes and every pod kept the reservation

Our snapshot init container asked for six gigabytes and every pod kept the reservation

Comments
2 min read
Ansible put our export job in /etc/cron.d and Debian's cron never read the file

Ansible put our export job in /etc/cron.d and Debian's cron never read the file

Comments
2 min read
Our images carried an empty commit label for five months because the ARG sat above FROM

Our images carried an empty commit label for five months because the ARG sat above FROM

Comments
2 min read
One finance report ran our database host out of memory and Postgres dropped every connection

One finance report ran our database host out of memory and Postgres dropped every connection

Comments
2 min read
We removed replicas from our manifest for the autoscaler and checkout dropped to one pod

We removed replicas from our manifest for the autoscaler and checkout dropped to one pod

Comments 1
2 min read
RabbitMQ took our booking job back after thirty minutes and gave it to the other consumer

RabbitMQ took our booking job back after thirty minutes and gave it to the other consumer

Comments
2 min read
Our latency alert summed away the bucket label and never returned a value

Our latency alert summed away the bucket label and never returned a value

Comments
2 min read
Our new Compose project took a carrier's address range and the host stopped reaching them

Our new Compose project took a carrier's address range and the host stopped reaching them

Comments
2 min read
Ansible updated our servers five at a time and kept going after most of each batch failed

Ansible updated our servers five at a time and kept going after most of each batch failed

Comments
2 min read
We moved our portal build to Vite and stopped type checking for four months

We moved our portal build to Vite and stopped type checking for four months

Comments
2 min read
We gave our notification executor fifty threads and it sent everything on one

We gave our notification executor fifty threads and it sent everything on one

1
Comments
2 min read
We moved our Redis chart to Argo CD and every new render gave it a new password

We moved our Redis chart to Argo CD and every new render gave it a new password

1
Comments 1
2 min read
Our carrier rotated our API secret and the longer one broke our header in two

Our carrier rotated our API secret and the longer one broke our header in two

1
Comments
2 min read
Our request graph showed thousands of calls a second because gunicorn had four counters

Our request graph showed thousands of calls a second because gunicorn had four counters

1
Comments
2 min read
We scaled our pricing service to twelve pods and the first four kept every request

We scaled our pricing service to twelve pods and the first four kept every request

1
Comments
2 min read
We moved our reporting servers to Leeds and Ansible gave them the Leeds memory setting

We moved our reporting servers to Leeds and Ansible gave them the Leeds memory setting

2
Comments
2 min read
Our two person rule for production deploys let people approve their own runs

Our two person rule for production deploys let people approve their own runs

1
Comments
2 min read
One slow report kept marking our web servers down for ten seconds at a time

One slow report kept marking our web servers down for ten seconds at a time

1
Comments
2 min read
We fixed our label client in 0.0.15 and npm update kept every service on 0.0.14

We fixed our label client in 0.0.15 and npm update kept every service on 0.0.14

1
Comments
2 min read
Our largest partner's access token outgrew a header buffer and every call came back 400

Our largest partner's access token outgrew a header buffer and every call came back 400

1
Comments
2 min read
Our uptime check stayed green for forty minutes because CloudFront was answering it

Our uptime check stayed green for forty minutes because CloudFront was answering it

1
Comments 1
2 min read
We renamed a container port and our Service quietly dropped every new pod

We renamed a container port and our Service quietly dropped every new pod

1
Comments
2 min read
Ansible wrote yesterday's server addresses into our load balancer from its fact cache

Ansible wrote yesterday's server addresses into our load balancer from its fact cache

1
Comments
2 min read
Our nightly patch build reused one cached apt-get upgrade for seven months

Our nightly patch build reused one cached apt-get upgrade for seven months

1
Comments
2 min read
We gave our servers IPv6 addresses and every call to two of our carriers waited three seconds first

We gave our servers IPv6 addresses and every call to two of our carriers waited three seconds first

1
Comments
2 min read
Our index build timed out, the retry found it by name, and the release went out without it

Our index build timed out, the retry found it by name, and the release went out without it

1
Comments
2 min read
Our despatch job stopped at the first empty answer from SQS and left parcels for the next day

Our despatch job stopped at the first empty answer from SQS and left parcels for the next day

Comments
2 min read
Our restart alert resolved itself after ten minutes and the pod kept crashing until Monday

Our restart alert resolved itself after ten minutes and the pod kept crashing until Monday

Comments
2 min read
We downloaded our config in a postStart hook and one pod in four started without it

We downloaded our config in a postStart hook and one pod in four started without it

Comments
2 min read
We lowered our login rate limit during an attack and Terraform kept the old one

We lowered our login rate limit during an attack and Terraform kept the old one

Comments 1
2 min read
An advisory published that afternoon blocked our hotfix and every rollback we could build

An advisory published that afternoon blocked our hotfix and every rollback we could build

1
Comments 1
2 min read
We replaced our deploy keys with OIDC and every repository in the organisation could deploy our website

We replaced our deploy keys with OIDC and every repository in the organisation could deploy our website

1
Comments
2 min read
Our refund consumer was stopped for nine days and came back at the end of the topic

Our refund consumer was stopped for nine days and came back at the end of the topic

Comments 1
2 min read
We switched to squash merges and semantic-release found nothing to release for five weeks

We switched to squash merges and semantic-release found nothing to release for five weeks

Comments
2 min read
RabbitMQ confirmed every DHL Express label request and put none of them in a queue

RabbitMQ confirmed every DHL Express label request and put none of them in a queue

Comments
2 min read
Our CPU dashboard counted every pod twice and we added twelve nodes to match

Our CPU dashboard counted every pod twice and we added twelve nodes to match

Comments
2 min read
We required non root containers and our nightly export waited twelve days to start

We required non root containers and our nightly export waited twelve days to start

Comments
2 min read
Ansible looked for our old partner certificates on the CI runner and left them on every gateway

Ansible looked for our old partner certificates on the CI runner and left them on every gateway

Comments
2 min read
We added a fourth cache server and three quarters of our keys moved

We added a fourth cache server and three quarters of our keys moved

Comments
2 min read
We moved a Go release tag to fix a bug and half our users got the old code

We moved a Go release tag to fix a bug and half our users got the old code

2
Comments 2
2 min read
One curly apostrophe made every delivery text three messages long

One curly apostrophe made every delivery text three messages long

Comments
2 min read
Our warehouse link was full and the graph said it was using three hundred megabits

Our warehouse link was full and the graph said it was using three hundred megabits

Comments
2 min read
Our workers wedged after a broker failover and Kubernetes never ran the health check in their image

Our workers wedged after a broker failover and Kubernetes never ran the health check in their image

Comments
2 min read
We started shipping to Norway and Ansible wrote it into our config as false

We started shipping to Norway and Ansible wrote it into our config as false

Comments
2 min read
Three services shared one deploy queue and the middle one never deployed

Three services shared one deploy queue and the middle one never deployed

Comments
2 min read
Our address lookups took one second or three and the service said it had answered in forty milliseconds

Our address lookups took one second or three and the service said it had answered in forty milliseconds

Comments
2 min read
Our deploy script restarted twelve servers in three minutes and systemctl said yes to every one

Our deploy script restarted twelve servers in three minutes and systemctl said yes to every one

Comments
2 min read
Our supplier moved its price file to a CDN and every accented product name came back garbled

Our supplier moved its price file to a CDN and every accented product name came back garbled

Comments
2 min read
We upgraded Prometheus and our latency alert stopped matching the bucket it was written for

We upgraded Prometheus and our latency alert stopped matching the bucket it was written for

Comments
2 min read
loading...