#9 Mistakes have been made

In this episode, Toby and Krisztian drop the technical polish and get honest about the biggest mistakes of their careers. From wiping a month of startup data with a single wrong command, to nearly electrocuting himself pulling a chassis from a live rack, to a rounding error in financial software that ended up in front of the CFO — the stories are equal parts hilarious and painful. They also cover bad search-replaces on live Cassandra clusters, taking on management too early, a wrong-direction DD command, and accidentally generating a massive AWS bill. A candid episode about how experience is often just accumulated failure.

Chapters

0:00 Introduction

0:51 The search-replace that corrupted a Cassandra cluster

2:20 Migrating a print shop to Linux in the 90s

5:45 Data centre migration disaster: wrong rack, 3am

8:27 Wiping a month of startup code with DD in the wrong direction

10:24 Tape backups and old-school data loss

13:16 Descending into a coal mine without the tools

18:07 The accidental text that went to the boss

20:33 The kill switch that locked out a paying customer

25:07 Pushed into management too early

28:35 Not surrounding yourself with business people soon enough

38:26 The AWS bill that dwarfed the customer contract value

41:08 The rounding error that ended up in front of the CFO

45:54 The ClickHouse lesson: check managed services first

48:25 Nearly electrocuted pulling a live power supply

51:30 Airport runway lighting and the buffer overflow

1:02:30 Mission command, autonomy and lessons from other industries

1:04:45 Summary: own up fast, learn, and keep doing things

Technologies Mentioned

Cassandra - https://cassandra.apache.org

AWS - https://aws.amazon.com

ClickHouse - https://clickhouse.com

Kubernetes - https://kubernetes.io

#9 Mistakes have been made

Show Notes