While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat …

Read More

It's a beautiful day, and it started with a simple code review: 1# tools/foo/main.go 2- fmt.Println("found it") 3+ log.Println("found it") The author explained the advantages of using a logging library over plain printf. The rationale was straightforward, so I approved the change without hesitation. …

Read More

We started migrating from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023) a month ago. While testing workloads on AL2023 in the pre-production environment, I noticed slightly higher disk usage compared to the same workload on AL2. In this post, I'll share my investigation. AL2023 Has Less Free Disk Space with ext4, …

Read More

Our Go programs recently triggered an alarm due to excessive panics. Panic is a Go runtime mechanism that halts execution. It got me thinking about different ways a Go program can die. I don't expect many - not like A Million Ways to Die in the West. In this post, we'll go through the various ways Go programs die. …

Read More

As Amazon Linux 2 (AL2) approaches its End of Life on June 30, 2025, we have started migrating our container platform from AL2 to Bottlerocket. The migration encountered a few speed bumps. In this post, we'll examine one of them: missing container disk I/O stats. Why are container I/O dashboards blank? Since …

Read More

I've been building a service for a month, and the day finally arrived when I had the artifact - an EC2 AMI. The AMI passed my "rigorous" manual tests so I launched 100 EC2 instances. Surprise! Around 28 instances failed to launch. What's going on? All failed instances were stuck in the …

Read More

Given an array of numbers A, find out whether it contains a 1-3-2 pattern. A 1-3-2 pattern is a subsequence of three numbers, A[i], A[j] and A[k] such that i < j < k and A[i] < A[k] < A[j]. For clarity, let's call the 1-3-2 pattern the Bronze-Gold-Silver pattern. If A[j] is Gold, then we should consider the …

Read More

Tony Hoare invented QuickSort in 1961. At the time of its publication, the best comparison-based sorting algorithm was merge sort. Merge sort divides an unordered array into two equally sized subarrays, sorts each subarray, and then merge the two subarrays to produce a sorted array. Merge sort is simple to understand. …

Read More

I often find myself confused by "upstream" and "downstream", in the context of software development. They bother me so much that I avoid using them in my own writing and I have to pause whenever I see them. In this post, I'll show a simple rule that helps you remember the difference: downstream adds …

Read More

Modern CPUs operate significantly faster than memories. A 4.5 GHz x64 CPU operates 30 times faster than a 6000 MHz DDR5 memory of CAS Latency 36. Adding latencies incurred by the bus and memory coherency protocols, memory could be 100 times slower than registers. To mitigate the speed gap, CPU uses layers of caches, …

Read More