Posts tagged: cloud

The Secret of the AWS Outage | The Standup - YouTube

2025-12-06T21:29:48Z

What a great campfire story Casey stumbled into. Whether any of this is true few will ever know, but its very reasonable that a race condition and a stalled job to apply configuration caused by someone who left the company 10 years ago caused an outage. I find it hilarious that they call this guy he answers, yup I still know the password, but how do I know you’re legit, I’m not just handing out the password. Casey did a stand up job telling this story.

Another Big Cloud Outage Nov 2025

2025-11-18T18:39:00Z

Today I woke up to finding out that cloudflare hade a widespread outage. My Reader uses tailwind cdn for styles and it was down. Otherwise it was not so impactful to me and felt kike they were quick to have it up.

I’m not really researching here, just jotting thoughts down from a parking lot waiting for pickup.

It feels like we are seeing a lot of these lately. They feel much more frequent. It feels like a whole industry was sold on 9’s and reliability of big cloud that we just aren’t getting.

There’s a huge push to go back to self hosting, racking and stacking. I think this is great. I love it. I’m a big proponent for ownership and self hosting. It’s not the right move for everything and everyone, and is certainly not something to make a knee jerk reaction about in the moment of frustration.

There’s a lot of things that are just impossible to do yourself, cdn caching, edge compute, ddos protection.

These companies are not magic they are vulnerable to changes just like you and I. It really feels like more and more of these are due to misconfigurations, and small bugs introduced. As we see big tech downsize and lean more on ai that likes to do big code changes I dont see it getting better soon. Theres a lot of things we can all armchair quarterback about here, better testing, review, canary deployment, staged rollouts, rigorous review. All great things. I can hear Uncle Bob talking about rigor, giving a shit, and following principles.

We are at odds of reliability and speed. This critical infrastructure runs so many important things in our lives it feels like it deserves a professional engineer signoff on changes. Documentation of changes and testing done between changes. This would all but hault forward progress, taking us back to the level of physical components and manufacturing.

How do you decide what needs this rigor and how to regulate it when companies are incentized by number go up.

The answer right now is that we can’t and if you are managing critical infrastructure you need to take these outages into account in your disaster recovery plan and understand what you are willing to allow go down on failure.

My Bed Doesn't Work Because of AWS Outage? TheStandup - YouTube

2025-10-24T00:48:31Z

Cloud is cooked bois. Seriously too much dumb shit relies on the cloud. Too much critical shit relies on single AZ’s. If normies are literally loosing sleep over an AWS outage (queue the Uncle Roger Voice), You’ve Fucked up. It’s wild to even think about a bed relying on the cloud let alone fully stop working when UE-1 goes down. I want to live in a world of opt in FEATURES, things that bring value to a product because it makes it better. Somehow a bed smells suspiciously like a cash grab for a subscription because its cloud connected. And yet for some reason it takes 16GeeeBee’s per month. I don’t own one of these, and I don’t want to. I don’t want a subscription for everything, I want my shit to just work. The future we are headed towards a world that is ever more reliant on a few key clouds. Which is fine. It’s fantastic that small companies can start and scale without owning an infrastructure team. It’s great that they have the ability to give us many nines of reliability. Some things just don’t need the cloud.

Ben Dicken on X: "I am once again begging you to put your data...

2025-09-05T23:45:00Z

Ben Dicken (@BenjDicken) on X

I am once again begging you to put your database servers and application servers in the same region.

X (formerly Twitter) · x.com

This is a really great animation of latency across different aws regions based on us-east-1.

Recovering from Disaster with Seth Eliot

2025-05-04T17:48:54Z

Recovering from Disaster with Seth Eliot

Disaster recovery is more than automation and infrastructure. There's a lot that goes into your services and some of those things can't be defined as code or automa…

Fork Around And Find Out · fafo.fm

This episode really got me thinking about the difference between HA and DR and my approach to each one. They talk about it from the perspective of a cach cow kind of app rather than a homelab or internal tooling, but think of HA as 9’s how many 9s are we willing to pay for, tink of DR as dollars how many dollars will we loose during the period of recovery. So much more in the episode, a lot of talk around cloud vendors and what they give you vs a purpose build platform with HA and DR in mind.

Deploy to Fly using a Depot builder

2024-12-25T02:20:08Z

Deploy to Fly using a Depot builder

Using Fly.io's new Depot builder, we'll walk you through how to deploy a TypeScript service globally with speed.

Depot · depot.dev

Here the integration to depot appears to be opt in using the --depot flag on fly deploy. This must have changed over time though because today it was giving me issues and I had to opt out using fly deploy --depot='false'. Looks like a great service and I just learned about them on their bad day.

Changelog on X: "🗣️@dhh on Kubernetes' migration pitch: “Oh, w...

2024-11-25T02:30:44Z

@changelog) on X — 🗣️@dhh on Kubernetes' migration pitch:

“Oh, we’re on Amazon? Flip the switch, George! Let’s move over to GCP.”

Vs reality:

"Flip the switch, George! We’ll start having the meetings about the meetings about how we can move out four years from now."" loading=“lazy”>

Changelog (@changelog) on X

🗣️@dhh on Kubernetes' migration pitch:

“Oh, we’re on Amazon? Flip the switch, George! Let’s move over to GCP.”

Vs reality:

"Flip the switch, George! We’ll start having the meetin…

X (formerly Twitter) · x.com

switching cloud providers, theres no easy way. K8s was supposed to get us there, haha, the deep integrations with each vendor just keep locking us in

Dax Raad - SST, Build modern full-stack applications on AWS wi...

2024-05-12T19:24:28Z

such a sick episode with dax.

SST’s free tier will be free as long as aws allows a free tier, their free tier literally costs them nothing.

They talked about keeping SST small, the limitations that brings, but also the number of problems that just go away when you only have 3 people building. Lots of process disappears, everyone can trust everyone, no one needs to wait for approval, everyone is their own PM and just builds cool shit. They don’t have to worry about big costs and making payroll because they are profitable so much higher than their costs.

If they can get through phase one of just being the go platform for a very specific audience of users, and gain marketshare, the ideas of offerings on top of this are endless.

How an empty S3 bucket can make your AWS bill explode | by Mac...

2024-05-01T02:31:56Z

How an empty S3 bucket can make your AWS bill explode

Imagine you create an empty, private AWS S3 bucket in a region of your preference. What will your AWS bill be the next morning?

Medium · medium.com

Imagine waking up to a $1,300 for running an example project! That sounds like peanuts for a cloud bill but for an individual trying to learn that hits my monthly budget real hard.

That’s what happened to Marciej, make sure you check out the full article and give them a 👏 on Medium if you have an account.

The more I see things come out about aws, the more it makes me sick, and confirm my feelings that I cannot possibly use them for a side project without some real $$ planning to come out of it.

Yes, S3 charges for unauthorized requests (4xx) as well[1]. That’s expected behavior.

They offer no DDOS protection against 4xx or 5xx requests against your bucket. Absolutely bonkers that you have ZERO control over this.

This response just feels absolutely gross.

I notified the AWS security team. I suggested that they restrict the unfortunate S3 bucket name to protect their customers from unexpected charges, and to protect the impacted companies from data leaks. But they were unwilling to address misconfigurations of third-party products.

It sounds like this guy followed some default instructions for an example site, HOW MANY OTHERS have done this or will do this? And AWS has no response other than to take thier money.

After contacting them he was able to get it cancelled, but this is no guarantee. We’ve seen other cloud vendors stick users like this with a few thousand dollar bills after cutting their bill to 5% of the original.

AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.

2024 is wild on the cloud hosting front, own your shit or be careful.

Udating Cloudflare Pages using the Wrangler cli

2024-04-09T17:03:51Z

Before deploying to cloudflare pages with wrangler you need a cloudflare api token. You can get one at dash.cloudflare.com/profile/api-tokens.

Install Wrangler #

Next install wrangler using npm.

npm i -g wrangler

Create a Project #

Before you deploy to cloudflare pages you need to create a project. You might already have one, or you might want to create one in the webui, but you have the option to create it at the command line with wrangler.

npx wrangler pages deploy markout --project-name reader-waylonwalker-com --branch markout

Deploy #

Now you can deploy your static application using wrangler to cloudflare pages.

In this example I have my application built into the markout directory, and since the production branch is named markout I need to pass that in here as well.

wrangler pages deploy markout --project-name reader-waylonwalker-com --branch markout

Waylon Walker 🐍 on X: "Which is more complicated" / X

2023-10-30T13:25:53Z

waylon walker (@_WaylonWalker) on X

Which is more complicated

X (formerly Twitter) · twitter.com

Wow, shocked at these results. All this time I’ve been told and believed that k8s is incredibly hard, and you need a $1M problem before you think about it because it will take a $1M team to maintain it. So far my experience has been good, and I definitely do not have a $1M problem in my homelab.

Delete a Postgres Cluster · Fly Docs

2023-10-17T18:42:30Z

Delete a Postgres Cluster

Documentation and guides from the team at Fly.io.

Fly · fly.io

Deleting a fly postgres db cluster was not straightforward to me as the app name is not inferred from the toml like it is for the main app.

fly apps destroy <pg-app-name>
fly pg db list -a <pg-app-name>

s3-tree · PyPI

2023-07-30T14:37:06Z

s3-tree

list s3 objects in tree-like format.

PyPI · pypi.org

Super useful way to show a tree view of an s3 bucket’s structure!

pip install s3-tree

s3-tree bucketname

running aws cli commands with localstack

2022-10-17T14:28:33Z

Upon first running an aws cli command using localstack you might end up with the following error.

Unable to locate credentials. You can configure credentials by running "aws configure".

Easy way #

The easy easiest way is to leverage a package called awscli-local.

pipx install awscli-local

Leveraging the awscli #

If you want to use the cli pro

pipx install awscli

aws config --profile localstack
# put what you want for the keys, but enter a valid region like us-east-1

alias aws='aws --endpoint-url http://localhost:4566 --profile localstack'

localstack status permission

2022-10-17T08:55:26Z

I ran into an issue where I was unable to ask localstack for its status. I would run the command and it would tell me that it didn’t have permission to read files from my own home directory. Let’s fix it

The issue #

I would run this to ask for the status.

localstack status

And get this error

PermissionError: [Errno 13] Permission denied: '/home/waylon/.cache/localstack/image_metadata'

What happened #

It dawned on me that the first time I ran localstack was straight docker, not the python cli. When docker runs it typically runs as root unless the Dockerfile sets up a user and group for it.

How to fix it #

If you have sudo access to the machine you are on you can recursively change ownership to your user and group. I chose to just give myself ownership of my whole ~/.cache directory you could choose a deeper directory if you want. I feel pretty safe giving myself ownership to my own cache directory on my own machine.

whoami
# waylon

chown -R waylon:waylon ~/.cache

Now it’s working #

Running localstack status now gives me a nice status message rather than an error.

❯ localstack status
┌─────────────────┬───────────────────────────────────────────────────────┐
│ Runtime version │ 1.2.1.dev                                             │
│ Docker image    │ tag: latest, id: dbbfe0ce0008, 📆 2022-10-15T00:51:03 │
│ Runtime status  │ ✖ stopped                                             │
└─────────────────┴───────────────────────────────────────────────────────┘