Sergey Shinder - DEV Community

Sergey Shinder

Sep 29

Every cache entry we warmed at deploy expired in the same minute

#sergeyshinder #sre #reliability #caching

2

1

2 min read

Sergey Shinder

Sep 29

Our deploy tool decided that 2.9.0 was newer than 2.10.0

#sergeyshinder #releaseengineering #versioning #automation

1

2 min read

Sergey Shinder

Sep 29

Every order paid in yen looked a hundred times underpaid

#sergeyshinder #api #payments #integration

1

2 min read

Sergey Shinder

Sep 29

Which service lost its logs depended on who wrote first after midnight

#sergeyshinder #observability #logging #elasticsearch

1

2 min read

Sergey Shinder

Sep 29

Every pod on the node was ready and none of them could open a connection

#sergeyshinder #kubernetes #linux #networking

1

2 min read

Sergey Shinder

Sep 29

Our disaster recovery copies were in the same region as the originals

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 29

A hung test held the only runner allowed to deploy our hotfix

#sergeyshinder #cicd #githubactions #testing

1

2 min read

Sergey Shinder

Sep 28

The partner recovered in four minutes and our circuit breaker kept them out for three hours

#sergeyshinder #sre #reliability #resilience

2 min read

Sergey Shinder

Sep 28

We switched traffic to green and blue kept doing the work

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 28

We moved our webhook and the partner's POSTs arrived as empty GETs

#sergeyshinder #api #webhooks #integration

1

2 min read

Sergey Shinder

Sep 28

Four in ten of our requests were our own health checks

#sergeyshinder #observability #monitoring #sre

2

2 min read

Sergey Shinder

Sep 28

Our service became PID 1 and never collected its dead children

#sergeyshinder #docker #kubernetes #linux

1

2 min read

Sergey Shinder

Sep 28

The plan said update in place and our database failed over at half past ten

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 28

Our migration test passed because the last job left its database behind

#sergeyshinder #cicd #docker #testing

3

2 min read

Sergey Shinder

Sep 26

Our database was fast for six hours and then it ran out of credit

#sergeyshinder #sre #reliability #database

2 min read

Sergey Shinder

Sep 26

The app release was the one deploy we could not take back

#sergeyshinder #releaseengineering #mobile #automation

2 min read

Sergey Shinder

Sep 26

The carrier kept the first thirty five characters of every address

#sergeyshinder #api #integration #logistics

1

2 min read

Sergey Shinder

Sep 26

Our p99 could never go above two and a half seconds

#sergeyshinder #observability #monitoring #prometheus

2 min read

Sergey Shinder

Sep 26

Our cluster upgrade waited two days for one pod that could not leave

#sergeyshinder #kubernetes #containers #devops

2 min read

Sergey Shinder

Sep 26

A cleanup pull request scheduled the key our backups needed

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 26

A pull request from a stranger ran with our publishing token

#sergeyshinder #cicd #security #githubactions

2 min read

Sergey Shinder

Sep 25

One message we could not parse stopped four hours of dispatches

#sergeyshinder #sre #reliability #kafka

2 min read

Sergey Shinder

Sep 25

We pressed the button again and the release came out half published

#sergeyshinder #releaseengineering #automation #devops

2 min read

Sergey Shinder

Sep 25

Our integration was a select statement against somebody else's tables

#sergeyshinder #api #integration #database

2 min read

Sergey Shinder

Sep 25

Our service got forty percent more expensive and nothing could say where

#sergeyshinder #observability #performance #profiling

2 min read

Sergey Shinder

Sep 25

The volume we mounted hid the files our image had shipped

#sergeyshinder #kubernetes #containers #helm

2 min read

Sergey Shinder

Sep 25

Our code was correct and the account said no

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 25

Twelve test containers and one of them ran for twenty six minutes

#sergeyshinder #cicd #testing #devops

2 min read

Sergey Shinder

Sep 24

The incident ended and the spare never came back

#sergeyshinder #sre #reliability #incident

2 min read

Sergey Shinder

Sep 24

The deploy took four minutes and the release took six days

#sergeyshinder #releaseengineering #database #automation

2 min read

Sergey Shinder

Sep 24

A third of our calls came from an address our carrier had never been given

#sergeyshinder #api #integration #networking

2 min read

Sergey Shinder

Sep 24

We spent thirteen weeks about to buy a bigger database

#sergeyshinder #observability #tracing #performance

2 min read

Sergey Shinder

Sep 24

The node was ready in seventy seconds and our pod was not

#sergeyshinder #kubernetes #docker #autoscaling

2 min read

Sergey Shinder

Sep 24

We forked a module for one line and owned all of it

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 24

Thirty seven tests failed the day our runner stopped running them in order

#sergeyshinder #cicd #testing #devops

2 min read

Sergey Shinder

Sep 23

The order was committed and nothing else ever heard about it

#sergeyshinder #sre #reliability #messaging

2 min read

Sergey Shinder

Sep 23

The fix we backported to our oldest branch did nothing

#sergeyshinder #releaseengineering #versioning #automation

2 min read

Sergey Shinder

Sep 23

Twenty workers refreshed one token and invalidated each other

#sergeyshinder #api #oauth #integration

1

2

2 min read

Sergey Shinder

Sep 23

The requests our customers gave up on were in none of our numbers

#sergeyshinder #observability #monitoring #latency

2 min read

Sergey Shinder

Sep 23

Two in three of our internal calls left the zone they started in

#sergeyshinder #kubernetes #networking #cloud

2 min read

Sergey Shinder

Sep 23

Somebody's laptop upgraded our state file

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 23

Our integration tests were talking to our partner's live system

#sergeyshinder #cicd #testing #integration

2 min read

Sergey Shinder

Sep 22

Our deploys were also our restart policy

#sergeyshinder #sre #reliability #kubernetes

2 min read

Sergey Shinder

Sep 22

We shipped the version that could not receive the next one

#sergeyshinder #releaseengineering #automation #deployment

2 min read

Sergey Shinder

Sep 22

We were the unreliable half of the integration

#sergeyshinder #api #integration #backend

1

2 min read

Sergey Shinder

Sep 22

Every question we asked our logs took six minutes to answer

#sergeyshinder #observability #logging #elasticsearch

2 min read

Sergey Shinder

Sep 22

Our cluster was full at nineteen percent CPU

#sergeyshinder #kubernetes #containers #capacity

2 min read

Sergey Shinder

Sep 22

One workspace planned clean and eleven were never asked

#sergeyshinder #terraform #iac #testing

2 min read

Sergey Shinder

Sep 21

The cheapest check in our pipeline ran last

#sergeyshinder #cicd #kubernetes #devops

1

2 min read

Sergey Shinder

Sep 20

Our rate limit punished everyone except the customer causing the problem

#sergeyshinder #sre #reliability #ratelimiting

2 min read

Sergey Shinder

Sep 20

Nine teams were taking turns on one staging environment

#sergeyshinder #releaseengineering #devops #automation

1

2 min read

Sergey Shinder

Sep 20

We never told the partner which version of their API we wanted

#sergeyshinder #api #integration #versioning

2 min read

Sergey Shinder

Sep 20

We spent the first twenty three minutes of an outage reading staging's graphs

#sergeyshinder #observability #grafana #monitoring

2 min read

Sergey Shinder

Sep 20

Our cluster had kept every job it ever finished

#sergeyshinder #kubernetes #containers #devops

1

2 min read

Sergey Shinder

Sep 20

An empty list in our module meant every bucket

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 20

Every merge in the company depended on the dev cluster staying up

#sergeyshinder #cicd #testing #devops

1

2 min read

Sergey Shinder

Sep 19

Nobody had timed the part that comes after the restore

#sergeyshinder #sre #reliability #backups

2 min read

Sergey Shinder

Sep 19

Five percent of our traffic was not five percent of our callers

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 19

Our totals were wrong by pennies and the pennies were in the parser

#sergeyshinder #api #integration #json

2 min read

Sergey Shinder

Sep 19

One new label multiplied our metrics by every order we take

#sergeyshinder #observability #prometheus #monitoring

2 min read

Writing Debut