Incidents reported on status page for Buttondown https://status.buttondown.com/ en https://status.buttondown.com/ Wed, 03 Dec 2025 21:21:25 +0000 https://status.buttondown.com/#8914a32549ed641b247c41611398556912564c93d526146cfd48b868fadecf25 p95 response time recovered https://status.buttondown.com/ Wed, 03 Dec 2025 21:05:48 +0000 https://status.buttondown.com/#8914a32549ed641b247c41611398556912564c93d526146cfd48b868fadecf25 p95 response time went down https://status.buttondown.com/ Tue, 02 Dec 2025 21:20:48 +0000 https://status.buttondown.com/#9fea323b0d26c63f3914c181394dfa69410bf7d35734c05e71fb84505ce03d5b p95 response time recovered https://status.buttondown.com/ Tue, 02 Dec 2025 21:05:39 +0000 https://status.buttondown.com/#9fea323b0d26c63f3914c181394dfa69410bf7d35734c05e71fb84505ce03d5b p95 response time went down https://status.buttondown.com/ Mon, 01 Dec 2025 01:31:17 +0000 https://status.buttondown.com/#1bdd7c8ba73bc4d55d16691f29d04f1badc43bf9b0474e4148a2a9c6b8666729 p95 response time recovered https://status.buttondown.com/ Mon, 01 Dec 2025 01:15:39 +0000 https://status.buttondown.com/#1bdd7c8ba73bc4d55d16691f29d04f1badc43bf9b0474e4148a2a9c6b8666729 p95 response time went down https://status.buttondown.com/ Fri, 28 Nov 2025 21:06:16 +0000 https://status.buttondown.com/#f280537641ddde3990d8cc4fdebee9dceeb3bc883a3974f7c839ed2a9bd33874 p95 response time recovered https://status.buttondown.com/ Fri, 28 Nov 2025 20:25:56 +0000 https://status.buttondown.com/#f280537641ddde3990d8cc4fdebee9dceeb3bc883a3974f7c839ed2a9bd33874 p95 response time went down https://status.buttondown.com/ Tue, 25 Nov 2025 20:55:42 +0000 https://status.buttondown.com/#404950ca30c8de2685126b55d629e0624a27d9410157c6e99f57040092691ff6 p95 response time recovered https://status.buttondown.com/ Tue, 25 Nov 2025 20:25:37 +0000 https://status.buttondown.com/#404950ca30c8de2685126b55d629e0624a27d9410157c6e99f57040092691ff6 p95 response time went down https://status.buttondown.com/ Tue, 25 Nov 2025 05:00:44 +0000 https://status.buttondown.com/#3db9d9bb6f01d2b7ec77c68d95dba7b57ba4c4d8ed95617697b7588a29172630 External event backlog recovered https://status.buttondown.com/ Tue, 25 Nov 2025 04:01:17 +0000 https://status.buttondown.com/#3db9d9bb6f01d2b7ec77c68d95dba7b57ba4c4d8ed95617697b7588a29172630 External event backlog went down https://status.buttondown.com/ Mon, 24 Nov 2025 16:25:22 +0000 https://status.buttondown.com/#9d2598e876e9bfcf5f4d5220263835c12268d23731151a95224e3d6c361b9c39 p95 response time recovered https://status.buttondown.com/ Mon, 24 Nov 2025 16:05:22 +0000 https://status.buttondown.com/#9d2598e876e9bfcf5f4d5220263835c12268d23731151a95224e3d6c361b9c39 p95 response time went down https://status.buttondown.com/ Mon, 24 Nov 2025 14:35:55 +0000 https://status.buttondown.com/#f161909df2260aa57730c854f6f6142251231b0ae7368ad17898f916d2b629d9 p95 response time recovered https://status.buttondown.com/ Mon, 24 Nov 2025 14:15:33 +0000 https://status.buttondown.com/#f161909df2260aa57730c854f6f6142251231b0ae7368ad17898f916d2b629d9 p95 response time went down https://status.buttondown.com/ Sat, 22 Nov 2025 04:05:34 +0000 https://status.buttondown.com/#a190d00696faa1151e0c58286e466f872f57cd25adf0e5edf9c80e006011f136 p95 response time recovered https://status.buttondown.com/ Sat, 22 Nov 2025 02:35:30 +0000 https://status.buttondown.com/#a190d00696faa1151e0c58286e466f872f57cd25adf0e5edf9c80e006011f136 p95 response time went down https://status.buttondown.com/ Fri, 21 Nov 2025 21:35:35 +0000 https://status.buttondown.com/#5a9a07a875a8fbcf60d60dc95ae628b6613249b9bf7330e3dbc604886227952b p95 response time recovered https://status.buttondown.com/ Fri, 21 Nov 2025 20:35:28 +0000 https://status.buttondown.com/#5a9a07a875a8fbcf60d60dc95ae628b6613249b9bf7330e3dbc604886227952b p95 response time went down https://status.buttondown.com/ Fri, 21 Nov 2025 17:35:48 +0000 https://status.buttondown.com/#88e81c1f7fe592490cb9a81489dafcf23dd9e07bf1aa61894a8855a1adeededa p95 response time recovered https://status.buttondown.com/ Fri, 21 Nov 2025 17:11:35 +0000 https://status.buttondown.com/#88e81c1f7fe592490cb9a81489dafcf23dd9e07bf1aa61894a8855a1adeededa p95 response time went down https://status.buttondown.com/ Fri, 21 Nov 2025 15:25:59 +0000 https://status.buttondown.com/#0d5919113d27f67130339983a4055f93e1af6b0df18aecd79d24cec63c6d2702 p95 response time recovered https://status.buttondown.com/ Fri, 21 Nov 2025 15:15:54 +0000 https://status.buttondown.com/#0d5919113d27f67130339983a4055f93e1af6b0df18aecd79d24cec63c6d2702 p95 response time went down https://status.buttondown.com/ Fri, 21 Nov 2025 15:05:42 +0000 https://status.buttondown.com/#ac134a232a3eb23066f9f2b6fe36efe010194c59b6b3076e407a9cf511eccce1 p95 response time recovered https://status.buttondown.com/ Fri, 21 Nov 2025 14:35:29 +0000 https://status.buttondown.com/#ac134a232a3eb23066f9f2b6fe36efe010194c59b6b3076e407a9cf511eccce1 p95 response time went down https://status.buttondown.com/ Fri, 21 Nov 2025 13:51:06 +0000 https://status.buttondown.com/#56a29de287f1a0cab98e67000fe6e387305e2bc30a86cec89261edd21ebe8db5 p95 response time recovered https://status.buttondown.com/ Fri, 21 Nov 2025 13:40:32 +0000 https://status.buttondown.com/#56a29de287f1a0cab98e67000fe6e387305e2bc30a86cec89261edd21ebe8db5 p95 response time went down https://status.buttondown.com/ Wed, 19 Nov 2025 16:20:44 +0000 https://status.buttondown.com/#c6dd22892f70cdaef63d37eda547f42321efc883c74a8969a6f2453c83b14861 p95 response time recovered https://status.buttondown.com/ Wed, 19 Nov 2025 15:21:19 +0000 https://status.buttondown.com/#c6dd22892f70cdaef63d37eda547f42321efc883c74a8969a6f2453c83b14861 p95 response time went down https://status.buttondown.com/ Tue, 18 Nov 2025 23:35:44 +0000 https://status.buttondown.com/#3989db65a13e2ec5682496d0b33e11699b30bcb13469a3dcb5f51405459871d3 p95 response time recovered https://status.buttondown.com/ Tue, 18 Nov 2025 23:05:44 +0000 https://status.buttondown.com/#3989db65a13e2ec5682496d0b33e11699b30bcb13469a3dcb5f51405459871d3 p95 response time went down https://status.buttondown.com/ Tue, 18 Nov 2025 22:21:14 +0000 https://status.buttondown.com/#09d457bd2a10e41d4d872009132b59c8e5d55f45e54e45df7c9b9e8379b8122c p95 response time recovered https://status.buttondown.com/ Tue, 18 Nov 2025 21:36:29 +0000 https://status.buttondown.com/#09d457bd2a10e41d4d872009132b59c8e5d55f45e54e45df7c9b9e8379b8122c p95 response time went down https://status.buttondown.com/ Tue, 18 Nov 2025 20:57:39 +0000 https://status.buttondown.com/#28ae8997adc267dc2b9043014e64a7c620b5f1dd0d582e8e3b4918c4dfbbe6b4 p95 response time recovered https://status.buttondown.com/ Tue, 18 Nov 2025 20:35:59 +0000 https://status.buttondown.com/#28ae8997adc267dc2b9043014e64a7c620b5f1dd0d582e8e3b4918c4dfbbe6b4 p95 response time went down https://status.buttondown.com/ Tue, 18 Nov 2025 18:51:28 +0000 https://status.buttondown.com/#f9e68eff0e79f20670742d2f32ec38d641eb831479d8087dfb70c2eb7f380bf8 p95 response time recovered https://status.buttondown.com/ Tue, 18 Nov 2025 18:35:52 +0000 https://status.buttondown.com/#f9e68eff0e79f20670742d2f32ec38d641eb831479d8087dfb70c2eb7f380bf8 p95 response time went down https://status.buttondown.com/incident/769197 Tue, 18 Nov 2025 16:13:00 -0000 https://status.buttondown.com/incident/769197#f91483718fe0cde960516c3ff0a0a75cc8b113def4751b358e15384152a8b256 Cloudflare has identified the root cause of the incident and is currently recovering. https://status.buttondown.com/incident/769197 Tue, 18 Nov 2025 14:29:00 -0000 https://status.buttondown.com/incident/769197#73c9a768086fdb91960233f9e56f454a63dc055b51b49ed5e28c3bae80c3680f Our CDN is currently impacted by the wide-scale Cloudflare outage (https://www.cloudflarestatus.com/). Additionally, some of our vendors are impacted, too, such as our primary customer service support center. We will update this ticket if their specific failures bleed over into anything that may impact you significantly. In the meantime, you may experience issues with assets loading or delayed support responses. https://status.buttondown.com/ Sat, 15 Nov 2025 21:05:35 +0000 https://status.buttondown.com/#fe61c62d680dcfdc0e8b3be2111739d96d5cdd3c442239d9310b916fee8eb733 p95 response time recovered https://status.buttondown.com/ Sat, 15 Nov 2025 20:35:36 +0000 https://status.buttondown.com/#fe61c62d680dcfdc0e8b3be2111739d96d5cdd3c442239d9310b916fee8eb733 p95 response time went down https://status.buttondown.com/ Thu, 13 Nov 2025 22:01:31 +0000 https://status.buttondown.com/#b116f26fa22805df6e8d6773603369e3df9cdbeedae60db2eca907a7dbfd847c External event backlog recovered https://status.buttondown.com/ Thu, 13 Nov 2025 21:00:45 +0000 https://status.buttondown.com/#b116f26fa22805df6e8d6773603369e3df9cdbeedae60db2eca907a7dbfd847c External event backlog went down https://status.buttondown.com/ Tue, 11 Nov 2025 20:01:41 +0000 https://status.buttondown.com/#5419cac4986a6b5b7301598708bbd91e7365b9dc254fabd60ee0b4a0e64cdb38 External event backlog recovered https://status.buttondown.com/ Tue, 11 Nov 2025 19:01:29 +0000 https://status.buttondown.com/#5419cac4986a6b5b7301598708bbd91e7365b9dc254fabd60ee0b4a0e64cdb38 External event backlog went down https://status.buttondown.com/ Mon, 10 Nov 2025 00:25:45 +0000 https://status.buttondown.com/#5aaeaa73a0b478f4f6bf2afbc0b4b7ff77e843308fa67967adf41f04837aa28f p95 response time recovered https://status.buttondown.com/ Mon, 10 Nov 2025 00:10:52 +0000 https://status.buttondown.com/#5aaeaa73a0b478f4f6bf2afbc0b4b7ff77e843308fa67967adf41f04837aa28f p95 response time went down https://status.buttondown.com/ Sun, 09 Nov 2025 04:06:07 +0000 https://status.buttondown.com/#394e6bc6ddf3f6c3d24a0424927b1911c241347b51d67de183f5d845294b1c29 p95 response time recovered https://status.buttondown.com/ Sun, 09 Nov 2025 03:36:09 +0000 https://status.buttondown.com/#394e6bc6ddf3f6c3d24a0424927b1911c241347b51d67de183f5d845294b1c29 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 12:05:26 +0000 https://status.buttondown.com/#83c67d4036e5ca26460c8252c664c52c0d589e1b986f960f75b92b6d9a33bc83 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 11:35:22 +0000 https://status.buttondown.com/#83c67d4036e5ca26460c8252c664c52c0d589e1b986f960f75b92b6d9a33bc83 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 09:05:28 +0000 https://status.buttondown.com/#033850475d68b5beab8c7761b68cd29b204ea46dd67175b8870f46f3773165f2 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 08:35:29 +0000 https://status.buttondown.com/#033850475d68b5beab8c7761b68cd29b204ea46dd67175b8870f46f3773165f2 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 08:05:24 +0000 https://status.buttondown.com/#0df1efbac5544cabf2aa037816e07da921e85ea8e0d43bf91d9c8f8b87facbde p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 07:05:29 +0000 https://status.buttondown.com/#0df1efbac5544cabf2aa037816e07da921e85ea8e0d43bf91d9c8f8b87facbde p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 06:36:01 +0000 https://status.buttondown.com/#bc45ba46f1e0550cc43acc46d5369b5c4c86f80ff4dd04299895b83420bd6fdb p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 06:15:59 +0000 https://status.buttondown.com/#bc45ba46f1e0550cc43acc46d5369b5c4c86f80ff4dd04299895b83420bd6fdb p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 04:35:54 +0000 https://status.buttondown.com/#7567a238967652200da53945c760e2f0c55fdd85ba90302c83e7a4a6c1bcfd37 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 04:20:58 +0000 https://status.buttondown.com/#7567a238967652200da53945c760e2f0c55fdd85ba90302c83e7a4a6c1bcfd37 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 03:51:04 +0000 https://status.buttondown.com/#8edf0860df56c9c558e967d91b13c55a42024f690bf5262ae7213d33058a3ea4 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 03:40:55 +0000 https://status.buttondown.com/#8edf0860df56c9c558e967d91b13c55a42024f690bf5262ae7213d33058a3ea4 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 03:26:02 +0000 https://status.buttondown.com/#3553ec0d10276455c4504ccd5904551117f198ec606b94b587c1830c7f83036f p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 02:51:02 +0000 https://status.buttondown.com/#3553ec0d10276455c4504ccd5904551117f198ec606b94b587c1830c7f83036f p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 01:55:56 +0000 https://status.buttondown.com/#9960cebe7db75eacedacffc1c8b793fcacafcb82875df846ced30bc4883fe560 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 01:41:03 +0000 https://status.buttondown.com/#9960cebe7db75eacedacffc1c8b793fcacafcb82875df846ced30bc4883fe560 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 01:20:56 +0000 https://status.buttondown.com/#f05a2ef3b2f0dfe2518fa29fc7783d8623bcf25ccf91020564097aa97a890081 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 01:01:30 +0000 https://status.buttondown.com/#f05a2ef3b2f0dfe2518fa29fc7783d8623bcf25ccf91020564097aa97a890081 p95 response time went down https://status.buttondown.com/ Sat, 08 Nov 2025 00:40:58 +0000 https://status.buttondown.com/#e36e276abe3b604dda251871efc10ce07dca8f6b1633253cfa4734a2e0c3dd30 p95 response time recovered https://status.buttondown.com/ Sat, 08 Nov 2025 00:25:59 +0000 https://status.buttondown.com/#e36e276abe3b604dda251871efc10ce07dca8f6b1633253cfa4734a2e0c3dd30 p95 response time went down https://status.buttondown.com/ Fri, 07 Nov 2025 22:06:01 +0000 https://status.buttondown.com/#cd0332f10dcd77d4a04592be48f332072c7fafb00f8c8bb11375569fb7cb8ba6 p95 response time recovered https://status.buttondown.com/ Fri, 07 Nov 2025 21:26:01 +0000 https://status.buttondown.com/#cd0332f10dcd77d4a04592be48f332072c7fafb00f8c8bb11375569fb7cb8ba6 p95 response time went down https://status.buttondown.com/ Thu, 06 Nov 2025 21:30:59 +0000 https://status.buttondown.com/#37c2c370fb3d3ba729a21c300b21abe2a83b01207e98049ae68391c28addc54f p95 response time recovered https://status.buttondown.com/ Thu, 06 Nov 2025 21:05:59 +0000 https://status.buttondown.com/#37c2c370fb3d3ba729a21c300b21abe2a83b01207e98049ae68391c28addc54f p95 response time went down https://status.buttondown.com/ Thu, 06 Nov 2025 14:31:20 +0000 https://status.buttondown.com/#2b08d794e4391b541f8c5d7605b24c42bfe0161920317c5c7dddf40766ed7cf4 p95 response time recovered https://status.buttondown.com/ Thu, 06 Nov 2025 14:12:17 +0000 https://status.buttondown.com/#2b08d794e4391b541f8c5d7605b24c42bfe0161920317c5c7dddf40766ed7cf4 p95 response time went down https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 03:10:00 -0000 https://status.buttondown.com/incident/758751#02360f03a310bf66f7543057f6c696300f4f20d0b2766d05fa06560fff4b4697 We're seeing that all services are back online and stable. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 03:10:00 -0000 https://status.buttondown.com/incident/758751#02360f03a310bf66f7543057f6c696300f4f20d0b2766d05fa06560fff4b4697 We're seeing that all services are back online and stable. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 03:10:00 -0000 https://status.buttondown.com/incident/758751#02360f03a310bf66f7543057f6c696300f4f20d0b2766d05fa06560fff4b4697 We're seeing that all services are back online and stable. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 03:10:00 -0000 https://status.buttondown.com/incident/758751#02360f03a310bf66f7543057f6c696300f4f20d0b2766d05fa06560fff4b4697 We're seeing that all services are back online and stable. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 03:10:00 -0000 https://status.buttondown.com/incident/758751#02360f03a310bf66f7543057f6c696300f4f20d0b2766d05fa06560fff4b4697 We're seeing that all services are back online and stable. https://status.buttondown.com/ Thu, 06 Nov 2025 01:40:50 +0000 https://status.buttondown.com/#2aaf042eeb944755d28fc038f8aafdf545023791a94b0e909bdb63ca86fa564d p95 response time recovered https://status.buttondown.com/ Thu, 06 Nov 2025 01:26:18 +0000 https://status.buttondown.com/#2aaf042eeb944755d28fc038f8aafdf545023791a94b0e909bdb63ca86fa564d p95 response time went down https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 00:27:00 -0000 https://status.buttondown.com/incident/758751#c2b67388a8bf40402b62831e3f4ab1415c4397b14f625cdc7d0819d04daa14e9 We're seeing that the author-facing app, login and archives have stabilized. But we're still monitoring some intermittent errors on the API. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 00:27:00 -0000 https://status.buttondown.com/incident/758751#c2b67388a8bf40402b62831e3f4ab1415c4397b14f625cdc7d0819d04daa14e9 We're seeing that the author-facing app, login and archives have stabilized. But we're still monitoring some intermittent errors on the API. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 00:27:00 -0000 https://status.buttondown.com/incident/758751#c2b67388a8bf40402b62831e3f4ab1415c4397b14f625cdc7d0819d04daa14e9 We're seeing that the author-facing app, login and archives have stabilized. But we're still monitoring some intermittent errors on the API. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 00:27:00 -0000 https://status.buttondown.com/incident/758751#c2b67388a8bf40402b62831e3f4ab1415c4397b14f625cdc7d0819d04daa14e9 We're seeing that the author-facing app, login and archives have stabilized. But we're still monitoring some intermittent errors on the API. https://status.buttondown.com/incident/758751 Thu, 06 Nov 2025 00:27:00 -0000 https://status.buttondown.com/incident/758751#c2b67388a8bf40402b62831e3f4ab1415c4397b14f625cdc7d0819d04daa14e9 We're seeing that the author-facing app, login and archives have stabilized. But we're still monitoring some intermittent errors on the API. https://status.buttondown.com/ Wed, 05 Nov 2025 23:46:05 +0000 https://status.buttondown.com/#d55ab4589932f51a3f6a69fdd671aef85e56ad27cf3adc76ed44d8b569397c94 p95 response time recovered https://status.buttondown.com/ Wed, 05 Nov 2025 23:16:20 +0000 https://status.buttondown.com/#d55ab4589932f51a3f6a69fdd671aef85e56ad27cf3adc76ed44d8b569397c94 p95 response time went down https://status.buttondown.com/ Wed, 05 Nov 2025 22:56:15 +0000 https://status.buttondown.com/#56383ec2f0e83d24448b6e280b4985bb372eec168ded55e772749e1f63712eca p95 response time recovered https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:50:00 -0000 https://status.buttondown.com/incident/758751#e8843e073c5be29bd19d063b10e395e64c2000ae9c4592f6ca1b72d1d17439c2 The author-facing app, the API and hosted archives are back online. That said, we're still seeing intermittent errors and slowness. We're continuing to monitor the situation until we're confident that this is stable. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:50:00 -0000 https://status.buttondown.com/incident/758751#e8843e073c5be29bd19d063b10e395e64c2000ae9c4592f6ca1b72d1d17439c2 The author-facing app, the API and hosted archives are back online. That said, we're still seeing intermittent errors and slowness. We're continuing to monitor the situation until we're confident that this is stable. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:50:00 -0000 https://status.buttondown.com/incident/758751#e8843e073c5be29bd19d063b10e395e64c2000ae9c4592f6ca1b72d1d17439c2 The author-facing app, the API and hosted archives are back online. That said, we're still seeing intermittent errors and slowness. We're continuing to monitor the situation until we're confident that this is stable. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:50:00 -0000 https://status.buttondown.com/incident/758751#e8843e073c5be29bd19d063b10e395e64c2000ae9c4592f6ca1b72d1d17439c2 The author-facing app, the API and hosted archives are back online. That said, we're still seeing intermittent errors and slowness. We're continuing to monitor the situation until we're confident that this is stable. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:50:00 -0000 https://status.buttondown.com/incident/758751#e8843e073c5be29bd19d063b10e395e64c2000ae9c4592f6ca1b72d1d17439c2 The author-facing app, the API and hosted archives are back online. That said, we're still seeing intermittent errors and slowness. We're continuing to monitor the situation until we're confident that this is stable. https://status.buttondown.com/ Wed, 05 Nov 2025 22:23:28 +0000 https://status.buttondown.com/#a0f1bd7c90d7b79547df5b29b48db15b1237d0a67a6e399761a94e6c3ff0ff5b buttondown.com/applied-cartography recovered https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:16:00 -0000 https://status.buttondown.com/incident/758751#9e5350feaeaaf60ad1ec5be2a034589312ed1365994c641564eadae47d1c8831 We've received additional reports of downtime with the author-facing app, login and the archives. The team is investigating and will have an update soon. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:16:00 -0000 https://status.buttondown.com/incident/758751#9e5350feaeaaf60ad1ec5be2a034589312ed1365994c641564eadae47d1c8831 We've received additional reports of downtime with the author-facing app, login and the archives. The team is investigating and will have an update soon. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:16:00 -0000 https://status.buttondown.com/incident/758751#9e5350feaeaaf60ad1ec5be2a034589312ed1365994c641564eadae47d1c8831 We've received additional reports of downtime with the author-facing app, login and the archives. The team is investigating and will have an update soon. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:16:00 -0000 https://status.buttondown.com/incident/758751#9e5350feaeaaf60ad1ec5be2a034589312ed1365994c641564eadae47d1c8831 We've received additional reports of downtime with the author-facing app, login and the archives. The team is investigating and will have an update soon. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 22:16:00 -0000 https://status.buttondown.com/incident/758751#9e5350feaeaaf60ad1ec5be2a034589312ed1365994c641564eadae47d1c8831 We've received additional reports of downtime with the author-facing app, login and the archives. The team is investigating and will have an update soon. https://status.buttondown.com/ Wed, 05 Nov 2025 22:11:22 +0000 https://status.buttondown.com/#56383ec2f0e83d24448b6e280b4985bb372eec168ded55e772749e1f63712eca p95 response time went down https://status.buttondown.com/ Wed, 05 Nov 2025 22:09:12 +0000 https://status.buttondown.com/#a0f1bd7c90d7b79547df5b29b48db15b1237d0a67a6e399761a94e6c3ff0ff5b buttondown.com/applied-cartography went down https://status.buttondown.com/ Wed, 05 Nov 2025 21:35:53 +0000 https://status.buttondown.com/#46c1d8144b38cfe5fe12cef81e8e982dc8916dfd53edb3b1b680b4594d693e71 p95 response time recovered https://status.buttondown.com/ Wed, 05 Nov 2025 21:23:16 +0000 https://status.buttondown.com/#5e51f85a2fde7dde02a11cc1e678f9cac6af302365f3b3babd156f527c8ef838 buttondown.com/applied-cartography recovered https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:22:00 -0000 https://status.buttondown.com/incident/758751#5ef1a590fb6dd442ccdc13b1413b83f7b1ccfd191bbc10b2d3351fc3e10c1d2f We've identified the issue, and are seeing all services come back online. There may be some temporary disruptions until we're fully recovered. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:22:00 -0000 https://status.buttondown.com/incident/758751#5ef1a590fb6dd442ccdc13b1413b83f7b1ccfd191bbc10b2d3351fc3e10c1d2f We've identified the issue, and are seeing all services come back online. There may be some temporary disruptions until we're fully recovered. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:22:00 -0000 https://status.buttondown.com/incident/758751#5ef1a590fb6dd442ccdc13b1413b83f7b1ccfd191bbc10b2d3351fc3e10c1d2f We've identified the issue, and are seeing all services come back online. There may be some temporary disruptions until we're fully recovered. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:22:00 -0000 https://status.buttondown.com/incident/758751#5ef1a590fb6dd442ccdc13b1413b83f7b1ccfd191bbc10b2d3351fc3e10c1d2f We've identified the issue, and are seeing all services come back online. There may be some temporary disruptions until we're fully recovered. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:22:00 -0000 https://status.buttondown.com/incident/758751#5ef1a590fb6dd442ccdc13b1413b83f7b1ccfd191bbc10b2d3351fc3e10c1d2f We've identified the issue, and are seeing all services come back online. There may be some temporary disruptions until we're fully recovered. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:00:00 -0000 https://status.buttondown.com/incident/758751#c0fa52ea4f68c9e4ba2a20c21311120bdfa0c9aab955127fe06bd6096f3665ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, API, login and hosted archives. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:00:00 -0000 https://status.buttondown.com/incident/758751#c0fa52ea4f68c9e4ba2a20c21311120bdfa0c9aab955127fe06bd6096f3665ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, API, login and hosted archives. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:00:00 -0000 https://status.buttondown.com/incident/758751#c0fa52ea4f68c9e4ba2a20c21311120bdfa0c9aab955127fe06bd6096f3665ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, API, login and hosted archives. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:00:00 -0000 https://status.buttondown.com/incident/758751#c0fa52ea4f68c9e4ba2a20c21311120bdfa0c9aab955127fe06bd6096f3665ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, API, login and hosted archives. https://status.buttondown.com/incident/758751 Wed, 05 Nov 2025 21:00:00 -0000 https://status.buttondown.com/incident/758751#c0fa52ea4f68c9e4ba2a20c21311120bdfa0c9aab955127fe06bd6096f3665ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, API, login and hosted archives. https://status.buttondown.com/ Wed, 05 Nov 2025 20:51:16 +0000 https://status.buttondown.com/#46c1d8144b38cfe5fe12cef81e8e982dc8916dfd53edb3b1b680b4594d693e71 p95 response time went down https://status.buttondown.com/ Wed, 05 Nov 2025 20:51:09 +0000 https://status.buttondown.com/#5e51f85a2fde7dde02a11cc1e678f9cac6af302365f3b3babd156f527c8ef838 buttondown.com/applied-cartography went down https://status.buttondown.com/ Tue, 04 Nov 2025 05:01:35 +0000 https://status.buttondown.com/#4376161ad08ac7759c8bd9825d67096ee2a85c2001460671f84cefd1910bbb0c External event backlog recovered https://status.buttondown.com/ Tue, 04 Nov 2025 04:01:16 +0000 https://status.buttondown.com/#4376161ad08ac7759c8bd9825d67096ee2a85c2001460671f84cefd1910bbb0c External event backlog went down https://status.buttondown.com/ Sun, 02 Nov 2025 04:51:05 +0000 https://status.buttondown.com/#881a29bacc3bb05064f8f11fd6bfebd072151eb58f35b4603a884e4e23d99399 p95 response time recovered https://status.buttondown.com/ Sun, 02 Nov 2025 04:40:46 +0000 https://status.buttondown.com/#881a29bacc3bb05064f8f11fd6bfebd072151eb58f35b4603a884e4e23d99399 p95 response time went down https://status.buttondown.com/ Mon, 27 Oct 2025 21:01:33 +0000 https://status.buttondown.com/#a15f3a1bd3144f12987a61168eb7ba7f9c66438960e497acab374921d1b29018 External event backlog recovered https://status.buttondown.com/ Mon, 27 Oct 2025 19:01:35 +0000 https://status.buttondown.com/#a15f3a1bd3144f12987a61168eb7ba7f9c66438960e497acab374921d1b29018 External event backlog went down https://status.buttondown.com/incident/747590 Mon, 20 Oct 2025 17:58:00 -0000 https://status.buttondown.com/incident/747590#681b1a7ce9cfd137d1eb9072e0821b1a9e8f78f6567f8d2a9a2a754a0264b9a7 We are currently not explicitly impacted by the wide-scale Amazon Web Services outage. A number of our vendors are, including our primary customer service support center and our two hosting infrastructure providers. We will update this ticket if their specific failures bleed over into anything that may impact you. In the meantime, you may experience some delays or intermittent failures. https://status.buttondown.com/ Mon, 20 Oct 2025 09:19:58 +0000 https://status.buttondown.com/#8948eac1102daff33ea12059bf9c74003f51d7c25ffa40922100faad79045f1c buttondown.com recovered https://status.buttondown.com/ Mon, 20 Oct 2025 07:55:51 +0000 https://status.buttondown.com/#8948eac1102daff33ea12059bf9c74003f51d7c25ffa40922100faad79045f1c buttondown.com went down https://status.buttondown.com/ Wed, 15 Oct 2025 15:25:44 +0000 https://status.buttondown.com/#ec912873d25b39b84eea8873f3febcce09d153f72b7d2c5faab6311be96aa7a6 p95 response time recovered https://status.buttondown.com/ Wed, 15 Oct 2025 15:15:40 +0000 https://status.buttondown.com/#ec912873d25b39b84eea8873f3febcce09d153f72b7d2c5faab6311be96aa7a6 p95 response time went down https://status.buttondown.com/ Wed, 15 Oct 2025 05:25:46 +0000 https://status.buttondown.com/#4c68680dfefe0c1ae985c5e0faa5b5a460fe8b4fb11411ef0aa76dc32b0bf2d9 p95 response time recovered https://status.buttondown.com/ Wed, 15 Oct 2025 05:10:42 +0000 https://status.buttondown.com/#4c68680dfefe0c1ae985c5e0faa5b5a460fe8b4fb11411ef0aa76dc32b0bf2d9 p95 response time went down https://status.buttondown.com/ Tue, 14 Oct 2025 03:01:09 +0000 https://status.buttondown.com/#15f831cdf49509c88c9f43d51c91e2d635bb71b690295c1b9ffa48cb7250635a External event backlog recovered https://status.buttondown.com/ Tue, 14 Oct 2025 02:01:05 +0000 https://status.buttondown.com/#15f831cdf49509c88c9f43d51c91e2d635bb71b690295c1b9ffa48cb7250635a External event backlog went down https://status.buttondown.com/ Mon, 13 Oct 2025 21:56:01 +0000 https://status.buttondown.com/#1f7994875c2bf99f1295986bc2e56793d521f4adc8a8b030d0653665f3356a19 p95 response time recovered https://status.buttondown.com/ Mon, 13 Oct 2025 21:45:59 +0000 https://status.buttondown.com/#1f7994875c2bf99f1295986bc2e56793d521f4adc8a8b030d0653665f3356a19 p95 response time went down https://status.buttondown.com/ Mon, 13 Oct 2025 00:21:35 +0000 https://status.buttondown.com/#68cfb68d31cc1027cdf08e1ffce72058782cd4d6b98b6dcf4dc08865c2cda518 Backlog recovered https://status.buttondown.com/ Mon, 13 Oct 2025 00:10:40 +0000 https://status.buttondown.com/#68cfb68d31cc1027cdf08e1ffce72058782cd4d6b98b6dcf4dc08865c2cda518 Backlog went down https://status.buttondown.com/ Wed, 08 Oct 2025 23:52:25 +0000 https://status.buttondown.com/#1427258378916a0abfdd475c5e5aac7e043dd06370925c0973bea2f441cebb31 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Wed, 08 Oct 2025 23:46:30 +0000 https://status.buttondown.com/#1427258378916a0abfdd475c5e5aac7e043dd06370925c0973bea2f441cebb31 buttondown.com/applied-cartography went down https://status.buttondown.com/ Wed, 08 Oct 2025 15:05:54 +0000 https://status.buttondown.com/#9b87f95191707830c7efca65f219aadaa23ac92d433e9fe5b968a3498468a950 p95 response time recovered https://status.buttondown.com/ Wed, 08 Oct 2025 14:15:58 +0000 https://status.buttondown.com/#9b87f95191707830c7efca65f219aadaa23ac92d433e9fe5b968a3498468a950 p95 response time went down https://status.buttondown.com/ Wed, 08 Oct 2025 12:51:14 +0000 https://status.buttondown.com/#a8c118e2e273d7fc7186409a1b75493051f76bf298165c27f78b549dfd2b3a0e p95 response time recovered https://status.buttondown.com/ Wed, 08 Oct 2025 12:30:47 +0000 https://status.buttondown.com/#a8c118e2e273d7fc7186409a1b75493051f76bf298165c27f78b549dfd2b3a0e p95 response time went down https://status.buttondown.com/ Wed, 08 Oct 2025 12:00:53 +0000 https://status.buttondown.com/#6c7de6af243bc7bbb5f2f8f5c7f200dd2f95e904ca2652b17b6d059048709cdd p95 response time recovered https://status.buttondown.com/ Wed, 08 Oct 2025 11:40:46 +0000 https://status.buttondown.com/#6c7de6af243bc7bbb5f2f8f5c7f200dd2f95e904ca2652b17b6d059048709cdd p95 response time went down https://status.buttondown.com/ Wed, 08 Oct 2025 08:40:46 +0000 https://status.buttondown.com/#fc25b3d12b87f518357236e899059078bf3f7b9c2357c4ca3e5a385b31333b84 p95 response time recovered https://status.buttondown.com/ Wed, 08 Oct 2025 08:25:49 +0000 https://status.buttondown.com/#fc25b3d12b87f518357236e899059078bf3f7b9c2357c4ca3e5a385b31333b84 p95 response time went down https://status.buttondown.com/ Tue, 07 Oct 2025 23:25:54 +0000 https://status.buttondown.com/#aafd53a5c979b4246360e08cc7fbc523fc3a593feaa6678ab892cdfd6b807801 p95 response time recovered https://status.buttondown.com/ Tue, 07 Oct 2025 23:05:56 +0000 https://status.buttondown.com/#aafd53a5c979b4246360e08cc7fbc523fc3a593feaa6678ab892cdfd6b807801 p95 response time went down https://status.buttondown.com/ Tue, 07 Oct 2025 22:40:52 +0000 https://status.buttondown.com/#318823e9c2c12df07c800d8ecf45d3b6dec248f8afa08b8e90500c9b0cad9d00 p95 response time recovered https://status.buttondown.com/ Tue, 07 Oct 2025 22:15:51 +0000 https://status.buttondown.com/#318823e9c2c12df07c800d8ecf45d3b6dec248f8afa08b8e90500c9b0cad9d00 p95 response time went down https://status.buttondown.com/ Tue, 07 Oct 2025 22:00:49 +0000 https://status.buttondown.com/#bba773d77d213e5bcf8ad14f7ba1a46128fe50e5f1e27144c187110398f4daef p95 response time recovered https://status.buttondown.com/ Tue, 07 Oct 2025 21:45:50 +0000 https://status.buttondown.com/#bba773d77d213e5bcf8ad14f7ba1a46128fe50e5f1e27144c187110398f4daef p95 response time went down https://status.buttondown.com/ Tue, 07 Oct 2025 20:35:56 +0000 https://status.buttondown.com/#3a3c866998e18c3cce04591d2e7cc31eeed18c5bec2d1d5cf1f53e06f7b4ee9c p95 response time recovered https://status.buttondown.com/ Tue, 07 Oct 2025 20:15:58 +0000 https://status.buttondown.com/#3a3c866998e18c3cce04591d2e7cc31eeed18c5bec2d1d5cf1f53e06f7b4ee9c p95 response time went down https://status.buttondown.com/ Tue, 07 Oct 2025 05:01:34 +0000 https://status.buttondown.com/#6c82c147ff1f420441448cf9a0e6296ab385d186f3284d7f6866f20a504f6a70 External event backlog recovered https://status.buttondown.com/ Tue, 07 Oct 2025 03:01:34 +0000 https://status.buttondown.com/#6c82c147ff1f420441448cf9a0e6296ab385d186f3284d7f6866f20a504f6a70 External event backlog went down https://status.buttondown.com/ Mon, 06 Oct 2025 00:35:38 +0000 https://status.buttondown.com/#e8623ea1c2bcee1d8024ae56f487d03410a17b1ac24f0475b5db99a73f7ba112 p95 response time recovered https://status.buttondown.com/ Mon, 06 Oct 2025 00:15:38 +0000 https://status.buttondown.com/#e8623ea1c2bcee1d8024ae56f487d03410a17b1ac24f0475b5db99a73f7ba112 p95 response time went down https://status.buttondown.com/ Fri, 03 Oct 2025 09:20:58 +0000 https://status.buttondown.com/#300f8e6909700fdb8b843fcb9579e285c0048fd4ad672a1491066be89e308070 p95 response time recovered https://status.buttondown.com/ Fri, 03 Oct 2025 09:10:24 +0000 https://status.buttondown.com/#300f8e6909700fdb8b843fcb9579e285c0048fd4ad672a1491066be89e308070 p95 response time went down https://status.buttondown.com/ Tue, 30 Sep 2025 19:19:42 +0000 https://status.buttondown.com/#c83bf99a2d82384d529fc347ffa9d14bcd02c6151b041915662bb28a7b2db79c buttondown.com/applied-cartography recovered https://status.buttondown.com/ Tue, 30 Sep 2025 19:13:41 +0000 https://status.buttondown.com/#c83bf99a2d82384d529fc347ffa9d14bcd02c6151b041915662bb28a7b2db79c buttondown.com/applied-cartography went down https://status.buttondown.com/ Tue, 30 Sep 2025 17:55:38 +0000 https://status.buttondown.com/#8a841f36a4c30bc12516e0a2cf75a901e9588e7e9e752b5b3b10f28d18dc22ee p95 response time recovered https://status.buttondown.com/ Tue, 30 Sep 2025 17:46:41 +0000 https://status.buttondown.com/#4022ed5fa78a12946b665c1b7aa41f5fea5a51aec42d5f96de02019871c42cd1 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Tue, 30 Sep 2025 17:30:33 +0000 https://status.buttondown.com/#8a841f36a4c30bc12516e0a2cf75a901e9588e7e9e752b5b3b10f28d18dc22ee p95 response time went down https://status.buttondown.com/ Tue, 30 Sep 2025 17:29:40 +0000 https://status.buttondown.com/#4022ed5fa78a12946b665c1b7aa41f5fea5a51aec42d5f96de02019871c42cd1 buttondown.com/applied-cartography went down https://status.buttondown.com/ Tue, 30 Sep 2025 05:01:28 +0000 https://status.buttondown.com/#aef4404a3623ca926e6047b7e76ee98c1fa84e8faa881ed8bd47859b9bff4cae External event backlog recovered https://status.buttondown.com/ Tue, 30 Sep 2025 03:01:44 +0000 https://status.buttondown.com/#aef4404a3623ca926e6047b7e76ee98c1fa84e8faa881ed8bd47859b9bff4cae External event backlog went down https://status.buttondown.com/ Mon, 29 Sep 2025 21:46:00 +0000 https://status.buttondown.com/#73a9c57999b66d22855ee03be92a0c747308930759b37589dbb2e58a57b0889b p95 response time recovered https://status.buttondown.com/ Mon, 29 Sep 2025 21:25:53 +0000 https://status.buttondown.com/#73a9c57999b66d22855ee03be92a0c747308930759b37589dbb2e58a57b0889b p95 response time went down https://status.buttondown.com/ Mon, 29 Sep 2025 20:45:50 +0000 https://status.buttondown.com/#93c0eb4d2f49a3e95e2905c613d7aaa9ade0771625147a1ce5279f7306ac6bdc p95 response time recovered https://status.buttondown.com/ Mon, 29 Sep 2025 20:30:53 +0000 https://status.buttondown.com/#93c0eb4d2f49a3e95e2905c613d7aaa9ade0771625147a1ce5279f7306ac6bdc p95 response time went down https://status.buttondown.com/ Mon, 29 Sep 2025 19:30:49 +0000 https://status.buttondown.com/#b719b1de202db136e6e960e02434d85ba90a97a275ba8bf8618b3a92f77b6e90 p95 response time recovered https://status.buttondown.com/ Mon, 29 Sep 2025 19:04:25 +0000 https://status.buttondown.com/#6361ca7fc9f7d6b82719aef9f3e883b1b7e16c3fc49c03b25634f17e593eb309 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 29 Sep 2025 19:02:00 +0000 https://status.buttondown.com/#6361ca7fc9f7d6b82719aef9f3e883b1b7e16c3fc49c03b25634f17e593eb309 buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 29 Sep 2025 17:49:29 +0000 https://status.buttondown.com/#0c94c224eb3e46f1d4695e6c1b65cc03e37cf3ced611a294c986588ffbdecbdc buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 29 Sep 2025 17:43:01 +0000 https://status.buttondown.com/#0c94c224eb3e46f1d4695e6c1b65cc03e37cf3ced611a294c986588ffbdecbdc buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 29 Sep 2025 13:05:45 +0000 https://status.buttondown.com/#b719b1de202db136e6e960e02434d85ba90a97a275ba8bf8618b3a92f77b6e90 p95 response time went down https://status.buttondown.com/ Mon, 29 Sep 2025 12:45:44 +0000 https://status.buttondown.com/#6b9255412a86764725f71d8ea768c1416546887d390c0ec88ec8a7edd80d22dc p95 response time recovered https://status.buttondown.com/ Mon, 29 Sep 2025 12:15:45 +0000 https://status.buttondown.com/#6b9255412a86764725f71d8ea768c1416546887d390c0ec88ec8a7edd80d22dc p95 response time went down https://status.buttondown.com/ Mon, 29 Sep 2025 11:30:53 +0000 https://status.buttondown.com/#fa4715d993c02921767d93043a230a4b36d67875fe533379bce8f9cd2f216eef p95 response time recovered https://status.buttondown.com/ Mon, 29 Sep 2025 11:10:50 +0000 https://status.buttondown.com/#fa4715d993c02921767d93043a230a4b36d67875fe533379bce8f9cd2f216eef p95 response time went down https://status.buttondown.com/ Mon, 29 Sep 2025 03:21:00 +0000 https://status.buttondown.com/#c9fb831716e8c9b43139b8bda0e1497b1039b4dbfa9d2c5bc5f627ebb1196b4d p95 response time recovered https://status.buttondown.com/ Mon, 29 Sep 2025 03:10:57 +0000 https://status.buttondown.com/#c9fb831716e8c9b43139b8bda0e1497b1039b4dbfa9d2c5bc5f627ebb1196b4d p95 response time went down https://status.buttondown.com/ Sun, 28 Sep 2025 14:15:41 +0000 https://status.buttondown.com/#2d2370fe3ceba0d41c7af95a460a39fc569a1259583195fc420271f1f7b30a70 p95 response time recovered https://status.buttondown.com/ Sun, 28 Sep 2025 14:05:38 +0000 https://status.buttondown.com/#2d2370fe3ceba0d41c7af95a460a39fc569a1259583195fc420271f1f7b30a70 p95 response time went down https://status.buttondown.com/ Tue, 23 Sep 2025 06:00:54 +0000 https://status.buttondown.com/#c83a095c4a21eea76f88b4aa907a0d7d11ce9ec68ebf35ab71f4c47f8f49b5f3 External event backlog recovered https://status.buttondown.com/ Tue, 23 Sep 2025 05:00:54 +0000 https://status.buttondown.com/#c83a095c4a21eea76f88b4aa907a0d7d11ce9ec68ebf35ab71f4c47f8f49b5f3 External event backlog went down https://status.buttondown.com/ Mon, 22 Sep 2025 23:16:01 +0000 https://status.buttondown.com/#677de074dd536573e77e03512cc99945efdc90e802880caee55709e8ec97de0c p95 response time recovered https://status.buttondown.com/ Mon, 22 Sep 2025 23:05:57 +0000 https://status.buttondown.com/#677de074dd536573e77e03512cc99945efdc90e802880caee55709e8ec97de0c p95 response time went down https://status.buttondown.com/ Wed, 17 Sep 2025 14:16:24 +0000 https://status.buttondown.com/#b1ace72039aec8ca639a9174caaa5d56daad4c210908c1239d888a32d49e71be buttondown.com/applied-cartography recovered https://status.buttondown.com/ Wed, 17 Sep 2025 14:10:25 +0000 https://status.buttondown.com/#b1ace72039aec8ca639a9174caaa5d56daad4c210908c1239d888a32d49e71be buttondown.com/applied-cartography went down https://status.buttondown.com/ Tue, 16 Sep 2025 10:35:30 +0000 https://status.buttondown.com/#46b08f8be2c9767ab5cca7f90d68cd5480963577acb69cd97a1aa5252437b15b p95 response time recovered https://status.buttondown.com/ Tue, 16 Sep 2025 10:20:32 +0000 https://status.buttondown.com/#46b08f8be2c9767ab5cca7f90d68cd5480963577acb69cd97a1aa5252437b15b p95 response time went down https://status.buttondown.com/ Tue, 16 Sep 2025 03:45:33 +0000 https://status.buttondown.com/#024d417cdb63802f18b81a2ab3dfa3c0092f854f3589ce22521147759ea448f1 p95 response time recovered https://status.buttondown.com/ Tue, 16 Sep 2025 03:35:29 +0000 https://status.buttondown.com/#024d417cdb63802f18b81a2ab3dfa3c0092f854f3589ce22521147759ea448f1 p95 response time went down https://status.buttondown.com/ Tue, 16 Sep 2025 03:00:53 +0000 https://status.buttondown.com/#dc005091c2503fa9aecdd2de9861a618f72abf4ad43d4b86644b8d6a4dcabfa7 External event backlog recovered https://status.buttondown.com/ Tue, 16 Sep 2025 02:00:52 +0000 https://status.buttondown.com/#dc005091c2503fa9aecdd2de9861a618f72abf4ad43d4b86644b8d6a4dcabfa7 External event backlog went down https://status.buttondown.com/ Mon, 15 Sep 2025 21:40:31 +0000 https://status.buttondown.com/#b30a95e19778d00152d83b2da32c70fb904e13f7bd9af944a99a20603d43194e p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 21:25:24 +0000 https://status.buttondown.com/#b30a95e19778d00152d83b2da32c70fb904e13f7bd9af944a99a20603d43194e p95 response time went down https://status.buttondown.com/ Mon, 15 Sep 2025 20:55:32 +0000 https://status.buttondown.com/#b7f733d1d1ad5e778f2b927f5e5574810bee754bd84e06df010a21b6371df83b p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 20:35:27 +0000 https://status.buttondown.com/#b7f733d1d1ad5e778f2b927f5e5574810bee754bd84e06df010a21b6371df83b p95 response time went down https://status.buttondown.com/ Mon, 15 Sep 2025 17:50:23 +0000 https://status.buttondown.com/#a94fb3aeff7c8a432631d57b2096bc503a0754d64268e32b04ea05b3f6bc5a32 p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 17:35:54 +0000 https://status.buttondown.com/#a94fb3aeff7c8a432631d57b2096bc503a0754d64268e32b04ea05b3f6bc5a32 p95 response time went down https://status.buttondown.com/ Mon, 15 Sep 2025 14:50:33 +0000 https://status.buttondown.com/#667e42c8741c3b56b53b9824759dc3dc52de6a8dcb2ae14bf4b54df6dd46666c p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 14:25:30 +0000 https://status.buttondown.com/#667e42c8741c3b56b53b9824759dc3dc52de6a8dcb2ae14bf4b54df6dd46666c p95 response time went down https://status.buttondown.com/ Mon, 15 Sep 2025 13:45:36 +0000 https://status.buttondown.com/#413973aefaa2a3332df095b2500c14c374dd5043f394469805b79198f6b8583c p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 13:30:28 +0000 https://status.buttondown.com/#413973aefaa2a3332df095b2500c14c374dd5043f394469805b79198f6b8583c p95 response time went down https://status.buttondown.com/ Mon, 15 Sep 2025 10:40:30 +0000 https://status.buttondown.com/#900423046bddd84bf0c91acb853459712a6c13d9769d33130e4b188bbe49f133 p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 10:25:26 +0000 https://status.buttondown.com/#900423046bddd84bf0c91acb853459712a6c13d9769d33130e4b188bbe49f133 p95 response time went down https://status.buttondown.com/ Mon, 15 Sep 2025 09:25:34 +0000 https://status.buttondown.com/#df2934d2388e777bec4aea8be431c2a9ffb4eada1adbfe7dea345988425a6a1d p95 response time recovered https://status.buttondown.com/ Mon, 15 Sep 2025 09:15:27 +0000 https://status.buttondown.com/#df2934d2388e777bec4aea8be431c2a9ffb4eada1adbfe7dea345988425a6a1d p95 response time went down https://status.buttondown.com/ Sun, 14 Sep 2025 15:04:52 +0000 https://status.buttondown.com/#bdf98c61e0030b635fe89b714587225ef12618e83a8047147fa04f631140ef49 p95 response time recovered https://status.buttondown.com/ Sun, 14 Sep 2025 14:50:23 +0000 https://status.buttondown.com/#bdf98c61e0030b635fe89b714587225ef12618e83a8047147fa04f631140ef49 p95 response time went down https://status.buttondown.com/ Tue, 09 Sep 2025 17:00:45 +0000 https://status.buttondown.com/#c5564b276dc5c4e08266095a52f54feab6eacea2d19c13964cd107412427d13f External event backlog recovered https://status.buttondown.com/ Tue, 09 Sep 2025 15:01:06 +0000 https://status.buttondown.com/#c5564b276dc5c4e08266095a52f54feab6eacea2d19c13964cd107412427d13f External event backlog went down https://status.buttondown.com/ Mon, 08 Sep 2025 00:21:07 +0000 https://status.buttondown.com/#9f998932bf03fb4d7d702a4f4b9945d02341f104736b054fc019a17e20347d76 Backlog recovered https://status.buttondown.com/ Mon, 08 Sep 2025 00:11:04 +0000 https://status.buttondown.com/#9f998932bf03fb4d7d702a4f4b9945d02341f104736b054fc019a17e20347d76 Backlog went down https://status.buttondown.com/incident/710664 Fri, 22 Aug 2025 17:10:00 -0000 https://status.buttondown.com/incident/710664#383c4890852310650fce0f8e21b9dea2ae95178a38442bf58b29d96c96a0f28a The team has identified the issue and reverted the commit. The Dashboard is back up and working as expected. https://status.buttondown.com/incident/710664 Fri, 22 Aug 2025 16:40:00 -0000 https://status.buttondown.com/incident/710664#22001581118154589aa5f276696c078d8a33d734cc6e15d0f4d32fb9d560e782 We're currently investigating reports that the Dashboard is unavailable to some customers. https://status.buttondown.com/incident/708185 Mon, 18 Aug 2025 05:30:00 -0000 https://status.buttondown.com/incident/708185#0826b45e3e4690cb90fbfeaac3a8c03b6c6912dc981510cf67085f5e4f7dcd19 Resolved. https://status.buttondown.com/incident/708185 Sun, 17 Aug 2025 18:00:00 -0000 https://status.buttondown.com/incident/708185#79a4bf1b5fa24308cc4930be48c7df6be0206e6793d1e096ff36d2825ecaa894 On Sunday August 17th, 2025, between 2:15pm and 1:30am Eastern Time, sending emails through Buttondown's markdown and fancy mode editors often failed. This did not affect emails sent through the Buttondown CLI, Automations, RSS-to-Email, or the Buttondown API. This manifested as a perpetual "Loading…" screen after pressing "Publish," and the email not sending. This was due to a bug fix that was rolled out in the afternoon. Changes are not frequently made to Buttondown's code on weekends, but this change was degrading some customers' ability to write in the editor, so we verified that the fix was valid and rolled out the change. However, this change introduced a bug: immediately after an email's status was changed to `about_to_send` by an author pressing "Publish," our editor would mistakenly detect that it should be set back to `draft`. Therefore, most emails were put into `about_to_send` and then subsequently pulled back to `draft`. Unfortunately, because this change was made on the weekend and our support operates Monday-Friday, we did not notice customers flagging the issue to support until the evening. We identified the issue and rolled out a fix around 1:30am Eastern Time. We take such incidents, especially to major core functionality, extremely seriously. We apologize for anyone who ran into this issue, especially with time-sensitive emails that were delayed. Going forward, we're preventing an incident like this by adding a monitor that pages the team if an unusually low number of emails has been sent recently, and by adding end-to-end tests that automatically test email sending from the editor. If you have any further questions, we'd be happy to answer: support@buttondown.com https://status.buttondown.com/incident/623243 Mon, 21 Jul 2025 13:19:00 -0000 https://status.buttondown.com/incident/623243#bd0d70d55a97d3d86f58baf65ff2364ec5ca6e7903f0fa0699ac4551a40911a9 We've addressed the root cause of the issue, and our systems are now fully operational. We experienced a sudden increase in load that overwhelmed our previous process model, making it difficult for the system to recover gracefully once under stress. To resolve this, we’ve shifted our load balancing approach and scaled up, which allows us to run multiple concurrent processes per instance. This not only improves overall capacity and responsiveness but also lets us restart individual processes without placing stress on the entire system - increasing resilience moving forward. https://status.buttondown.com/incident/623243 Mon, 21 Jul 2025 13:19:00 -0000 https://status.buttondown.com/incident/623243#bd0d70d55a97d3d86f58baf65ff2364ec5ca6e7903f0fa0699ac4551a40911a9 We've addressed the root cause of the issue, and our systems are now fully operational. We experienced a sudden increase in load that overwhelmed our previous process model, making it difficult for the system to recover gracefully once under stress. To resolve this, we’ve shifted our load balancing approach and scaled up, which allows us to run multiple concurrent processes per instance. This not only improves overall capacity and responsiveness but also lets us restart individual processes without placing stress on the entire system - increasing resilience moving forward. https://status.buttondown.com/incident/623243 Mon, 21 Jul 2025 12:07:00 -0000 https://status.buttondown.com/incident/623243#18fdbf45b099bbc0934cd1d0832c2d8777f3f42ab44235f54a1dae4efcb97226 The service has been restored, but there may still be intermittent errors and delays in loading the application. We're continuing to investigate the root cause and monitor the status of the application. https://status.buttondown.com/incident/623243 Mon, 21 Jul 2025 12:07:00 -0000 https://status.buttondown.com/incident/623243#18fdbf45b099bbc0934cd1d0832c2d8777f3f42ab44235f54a1dae4efcb97226 The service has been restored, but there may still be intermittent errors and delays in loading the application. We're continuing to investigate the root cause and monitor the status of the application. https://status.buttondown.com/incident/623243 Mon, 21 Jul 2025 11:35:00 -0000 https://status.buttondown.com/incident/623243#3bec73a4b52584a3f72d4bc92aa4c200a861b1a9637398bceaeac1aa44f873ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, login and hosted archives. https://status.buttondown.com/incident/623243 Mon, 21 Jul 2025 11:35:00 -0000 https://status.buttondown.com/incident/623243#3bec73a4b52584a3f72d4bc92aa4c200a861b1a9637398bceaeac1aa44f873ee We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, login and hosted archives. https://status.buttondown.com/incident/621474 Thu, 17 Jul 2025 17:42:00 -0000 https://status.buttondown.com/incident/621474#8897b6d9dd258d8df6abe97c13d86ef1020e1a6463a0aed3a388cd4f6c4ebcce ## What was happening For the past few days, we’ve been experiencing `thundering herd`-esque downtime every time we deploy at the top of the hour. The top of the hour nuance is not actually that uncommon: we serve a lot of RSS traffic. (A fun fact is that 80% of our page views come from RSS readers and scrapers and the vast majority of them are fairly naive and run the equivalent of an hourly cron to ping every single RSS feed in which they are interested.) After investigation, we were able to identify the reason why this started happening recently, even though the above traffic pattern and our general CI/CD posture has remained unchanged. Part of our firewall system involves checking incoming IP addresses against a deny list culled from a variety of trusted sources. In order to make the firewall as performant as possible, we aggressively cache that list so that we're not pulling it every time a subscription attempt is made. However, we recently changed the logic to expand the purview of the database that held those IP addresses to also store aggregate-level data about IPs for telemetry purposes. At a high level, the logic looked something like this: ``` @cache def get_problematic_ip_addresses(): ip_address_models = IPAddress.objects.all() return { ip.ip_address for ip in ip_address_models if ip.do_not_honor } ``` And that logic remained the same! _But_ that backing `IPAddress` model went from a few hundred records to a few hundred _thousand_, replete with a JSON payload for each IP. And because we were caching this, it meant that even with rolling deploys, every single time a new server would come online, it would be aggressively unresponsive as it tried to pull and then collate every single IP address within the 30-second time span of a request. We’ve fixed this trivially: ``` @cache def get_problematic_ip_addresses(): ip_address_models = IPAddress.objects.filter(do_not_honor=True) return { ip.ip_address for ip in ip_address_models } ``` Going forward, we’ll be paying much closer attention to the actual timeline of the deploy process. It was easy to chalk this up to luck of the draw to a certain extent, but such “luck” is scarce and still came at the cost of severely degraded performance. By logging and alerting on startup time and deviations thereof, we’ll be able to more actively identify aberrations of this nature in the future. https://status.buttondown.com/incident/621474 Thu, 17 Jul 2025 17:42:00 -0000 https://status.buttondown.com/incident/621474#8897b6d9dd258d8df6abe97c13d86ef1020e1a6463a0aed3a388cd4f6c4ebcce ## What was happening For the past few days, we’ve been experiencing `thundering herd`-esque downtime every time we deploy at the top of the hour. The top of the hour nuance is not actually that uncommon: we serve a lot of RSS traffic. (A fun fact is that 80% of our page views come from RSS readers and scrapers and the vast majority of them are fairly naive and run the equivalent of an hourly cron to ping every single RSS feed in which they are interested.) After investigation, we were able to identify the reason why this started happening recently, even though the above traffic pattern and our general CI/CD posture has remained unchanged. Part of our firewall system involves checking incoming IP addresses against a deny list culled from a variety of trusted sources. In order to make the firewall as performant as possible, we aggressively cache that list so that we're not pulling it every time a subscription attempt is made. However, we recently changed the logic to expand the purview of the database that held those IP addresses to also store aggregate-level data about IPs for telemetry purposes. At a high level, the logic looked something like this: ``` @cache def get_problematic_ip_addresses(): ip_address_models = IPAddress.objects.all() return { ip.ip_address for ip in ip_address_models if ip.do_not_honor } ``` And that logic remained the same! _But_ that backing `IPAddress` model went from a few hundred records to a few hundred _thousand_, replete with a JSON payload for each IP. And because we were caching this, it meant that even with rolling deploys, every single time a new server would come online, it would be aggressively unresponsive as it tried to pull and then collate every single IP address within the 30-second time span of a request. We’ve fixed this trivially: ``` @cache def get_problematic_ip_addresses(): ip_address_models = IPAddress.objects.filter(do_not_honor=True) return { ip.ip_address for ip in ip_address_models } ``` Going forward, we’ll be paying much closer attention to the actual timeline of the deploy process. It was easy to chalk this up to luck of the draw to a certain extent, but such “luck” is scarce and still came at the cost of severely degraded performance. By logging and alerting on startup time and deviations thereof, we’ll be able to more actively identify aberrations of this nature in the future. https://status.buttondown.com/incident/621474 Thu, 17 Jul 2025 16:10:00 -0000 https://status.buttondown.com/incident/621474#5fbeb353a7c7d198be4af136c1e87fb07b7367b9191e1269e19d3dde33e9a655 We've identified the issue, and see service being restored. We're continuing to monitor as there may be intermittent errors. https://status.buttondown.com/incident/621474 Thu, 17 Jul 2025 16:10:00 -0000 https://status.buttondown.com/incident/621474#5fbeb353a7c7d198be4af136c1e87fb07b7367b9191e1269e19d3dde33e9a655 We've identified the issue, and see service being restored. We're continuing to monitor as there may be intermittent errors. https://status.buttondown.com/incident/621474 Thu, 17 Jul 2025 15:50:00 -0000 https://status.buttondown.com/incident/621474#33a46e67d146d1bf1652eed822c57b0249fbe0a4d8ca0b593bdc5e0661710edf We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, login and hosted archives. https://status.buttondown.com/incident/621474 Thu, 17 Jul 2025 15:50:00 -0000 https://status.buttondown.com/incident/621474#33a46e67d146d1bf1652eed822c57b0249fbe0a4d8ca0b593bdc5e0661710edf We're currently investigating reports that the web app is unavailable. This is impacting the Dashboard, login and hosted archives. https://status.buttondown.com/incident/620933 Wed, 16 Jul 2025 19:23:00 -0000 https://status.buttondown.com/incident/620933#8035bf51d92b8674b8c32decb664797b00c5c5e53b6e53d40a6a2cc181eeafcc The issue is resolved, and archives are performing as expected. https://status.buttondown.com/incident/620933 Wed, 16 Jul 2025 18:40:00 -0000 https://status.buttondown.com/incident/620933#e3db4a06e553da476faf09fdcc1dc90e64b20a3ff1cc5247817823cf99a2f166 We're monitoring reports of degraded archive performance caused by an incident [with our hosting vendor](https://status.heroku.com/incidents/2860). This is causing archives to error intermittently. https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:45:00 -0000 https://status.buttondown.com/incident/607839#29cc2cacaf77e405a152885eb4093083a30a07a1b9516cdf5a9f605e1f9b973e We've identified that our hosting provider was incorrectly identifying incoming traffic as being from bots. This issue is now resolved, and all services have been restored. https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:45:00 -0000 https://status.buttondown.com/incident/607839#29cc2cacaf77e405a152885eb4093083a30a07a1b9516cdf5a9f605e1f9b973e We've identified that our hosting provider was incorrectly identifying incoming traffic as being from bots. This issue is now resolved, and all services have been restored. https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:25:00 -0000 https://status.buttondown.com/incident/607839#79ed7de228dac5f8a4c0f9b789320d5658dc4971b183b7479809e9d26b4c6b0a We're investigating reports of Vercel error messages on our marketing site, documentation site, and while logging into the Buttondown app. The API, and the outbound email backlog are *not* impacted by this incident. https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:25:00 -0000 https://status.buttondown.com/incident/607839#79ed7de228dac5f8a4c0f9b789320d5658dc4971b183b7479809e9d26b4c6b0a We're investigating reports of Vercel error messages on our marketing site, documentation site, and while logging into the Buttondown app. The API, and the outbound email backlog are *not* impacted by this incident. https://status.buttondown.com/incident/600179 Wed, 11 Jun 2025 05:30:00 -0000 https://status.buttondown.com/incident/600179#9d8199bd755964dddfef56f87f2b7d5806168efa693502fe3259c47409a53fae The incident is now resolved, and [Heroku](https://status.heroku.com/incidents/2822) is back online. (HugOps to their team!) All services are operational, and Buttondown emails are being sent without delay. https://status.buttondown.com/incident/600179 Tue, 10 Jun 2025 22:00:00 -0000 https://status.buttondown.com/incident/600179#86c93dd0ea9349806db5bd0c400fcff492d45a86012eba2ecf76d94bed421034 We're seeing improvements for this ongoing incident: - Previously delayed emails have now been sent - Our backlog is clearing in a timely manner - New custom hosting domains are connecting as expected But as Heroku is [still reporting this as an active outage](https://status.heroku.com/incidents/2822), we're continuing to monitor. https://status.buttondown.com/incident/600179 Tue, 10 Jun 2025 14:32:00 -0000 https://status.buttondown.com/incident/600179#de7febb84dbfb21ab7e5a25bb9402b6bf28ffb71ad0e8f8bf854a37b17c09a90 Our upstream hosting provider, Heroku, is experiencing an [ongoing outage](https://status.heroku.com/incidents/2822). As a result: - We are unable to scale up our servers to match fluctuations in the number of emails our system is sending. **This means that emails may sit in queue and take longer to send out — but they will eventually send without any intervention necessary.** - We are unable to register new custom domains for archives. All other areas of Buttondown are **unaffected**. - Emails **will eventually send without any intervention**, but may take longer to do so. **If you're waiting on an email to go out, please do not attempt to send it a second time.** - Custom domains that are already set up will work as usual. - Custom domain setup for archives will work again [after Heroku's incident is closed](https://status.heroku.com/incidents/2822). https://status.buttondown.com/incident/547115 Wed, 16 Apr 2025 21:00:00 -0000 https://status.buttondown.com/incident/547115#e3fb3a06a6859995dcb01aa8c5cd27177493f842cf9175822db6b73d8c5e3acf # TL;DR Bad configuration on one of our self-hosted SMTP servers caused a crash that proved difficult to recover from, leaving lots of emails “stuck” in varying degrees – and their being stuck manifested in a slew of unpleasant ways. We’ve fixed the configuration, are investing (literally, right at this very moment) in better tooling and alerting, and are architecting a way to prevent this from ever happening again. ## The gory details Buttondown uses a number of providers to actually send the emails written by authors to their subscribers. In addition to using explicit vendors, we run and maintain our own fleet of servers dedicated to this purpose. We're going to refer to these servers as postal servers, as that's a reference to the great open source project which we rely on. On Wednesday morning, we received some automated alerts from our checker system indicating that our backlog of emails was higher than it should be. After digging in a little bit, we realized the reason it was so high was because each individual email was taking a huge amount of time to send out for one specific server. After a couple more minutes, this server got to the point where all it was doing was trying for a minute and then timing out. (Software engineers reading this might already be getting some ideas of what had happened.) We logged into that server and quickly discovered that the issue was with the database storing messages that were pending delivery. While our initial instinct was thinking that the problem was the overall volume of messages being sent to this particular server, we discovered that the volume was actually secondary to just the overall connection count. We were trying to connect to this database from too many worker threads and it was not set up to recover gracefully or even notify downstream connections what was happening. Once we discovered this, the first order solution was pretty simple. We cycled the database, scaled down the number of workers, and got the connections into a pretty manageable state. The problem we were now left with was that of recovery. We had around 70,000 messages stuck in purgatory. They were technically pending, but some of them, just due to the database connection, were actually correctly sent. Some of them were marked as sent but not actually sent, and so on. We basically entered a fog of war situation where our sources of truth were no longer valid. Our SOP in these cases is to err on the side of caution. Caution in this case means hazmatting that specific server, spinning down all of the workers, leaving all of those messages as pending, and then traffic over to another server or vendor to make sure we don't exacerbate the problem nor accidentally act upon incorrect information. This is exactly what we did. We shifted over traffic, the queue drained, and we resent any emails that we were very, very confident hadn't been sent. We cleared out the problematic server and resumed traffic. ## How we’re fixing it If you've read along this far, you're probably wondering what we're going to do to make this better. The first step, one that is essentially complete by the time we publish this postmortem, is a classic one. Add much more monitoring and alerting. We were over-reliant on the integration and high-level metrics for these servers, which works well when problems are obvious and well-formed, but doesn't work well when they're a little bit more out of the mainstream. To be specific, we already had alerting on pending or stuck messages at a per-server basis. But in order to actually fire those alerts, you needed to have an active connection to the database, which we couldn't have in this scenario. The second one is a little bit broader, which is that we need to do a much, much better job of proactively pushing information about these kinds of spending patterns to you, the author — one of the worst feelings is sending an email and being confused because it's marked as sent, but you haven't seen it in your inbox. We're going to start erring on the side of oversharing about the state of these things so you can proactively poke around within the dashboard and understand what might be causing delays in us getting your emails to your readers. ## Customer impact Over the course of the afternoon, approximately 13,000 subscribers across 40 authors experienced some combination of the following: - Multiple hour delays before receiving a message - Not receiving an email at all (though we’ve redriven these.) - Multiple sends of the same email ## Zooming out To be blunt, we've had too many incidents lately. We've invested a lot in fixing bugs and stability at an object level over the past six months. But we've done a poor job of investing in stability at an end-to-end infrastructural level. The past few weeks have driven that point home. Our most important job as a tool is to reliably send your writing to your subscribers. We have not sufficiently invested in the very boring but very important kinds of observability that we needed to, and we're shifting a lot of our roadmap over the next six months to make sure that our ability to diagnose and resolve these issues is much, much stronger than it has been. If you're still at the end having read through all of this, I know it's not because of rabid curiosity but likely because of frustration because you've trusted us with a job and we haven't been up to the task. But we take this stuff seriously and we're pouring everything we have into it. https://status.buttondown.com/incident/543429 Fri, 11 Apr 2025 01:33:00 -0000 https://status.buttondown.com/incident/543429#528fb50f837c45b8067e1a81db3daf8deb8fc22fbaddf84f6f1be26d2e74da12 From 8:38pm EDT to 8:50pm EDT, we were serving 503s for around 75% of our incoming requests. This was purely due to a high burst of traffic that our scaling mitigated (albeit not quickly enough!) We're going to look into the problematic routes and harden their performance. https://status.buttondown.com/incident/543429 Fri, 11 Apr 2025 01:33:00 -0000 https://status.buttondown.com/incident/543429#528fb50f837c45b8067e1a81db3daf8deb8fc22fbaddf84f6f1be26d2e74da12 From 8:38pm EDT to 8:50pm EDT, we were serving 503s for around 75% of our incoming requests. This was purely due to a high burst of traffic that our scaling mitigated (albeit not quickly enough!) We're going to look into the problematic routes and harden their performance. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 18:20:00 -0000 https://status.buttondown.com/incident/538844#c89c920252787b2fc61b8db6be13e52394b29dd7e23de5e32631afaa50858a7a The change has been reverted, and subscribers have been restored to their correct status. There is no action required from authors. There was no impact to subscriber-facing features. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 18:20:00 -0000 https://status.buttondown.com/incident/538844#c89c920252787b2fc61b8db6be13e52394b29dd7e23de5e32631afaa50858a7a The change has been reverted, and subscribers have been restored to their correct status. There is no action required from authors. There was no impact to subscriber-facing features. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:45:00 -0000 https://status.buttondown.com/incident/538844#08ed25b66dbfa6df50c5c1c54d1e36932e151ef19138b9925705cd66626eae3b The change is still being reverted, but we expect that this will be complete shortly. Please continue to refrain from sending until the fix is complete. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:45:00 -0000 https://status.buttondown.com/incident/538844#08ed25b66dbfa6df50c5c1c54d1e36932e151ef19138b9925705cd66626eae3b The change is still being reverted, but we expect that this will be complete shortly. Please continue to refrain from sending until the fix is complete. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:15:00 -0000 https://status.buttondown.com/incident/538844#0256ffd8a94bc10f3d2bcfa6b293498a26e979eabd2d27e800b825f11412b279 The Buttondown dashboard and API are mistakenly marking subscribers as undeliverable. We're in the process of reverting the change. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:15:00 -0000 https://status.buttondown.com/incident/538844#0256ffd8a94bc10f3d2bcfa6b293498a26e979eabd2d27e800b825f11412b279 The Buttondown dashboard and API are mistakenly marking subscribers as undeliverable. We're in the process of reverting the change. https://status.buttondown.com/incident/536193 Sat, 29 Mar 2025 00:18:00 -0000 https://status.buttondown.com/incident/536193#71531a41abee5355cf7f27b03549fb8ef5e593e946661fd272550ff8bebd40a1 We've processed the backlog. https://status.buttondown.com/incident/536193 Fri, 28 Mar 2025 20:26:00 -0000 https://status.buttondown.com/incident/536193#2cd84cb381e65f6540b2f3eec09c3fc70c9311a4c4f7d670974a31ee1a582ff9 We've got a very large backlog of webhooks/automations that need processing. We're scaling up (a lot!) in order to do so; no actions required on your end. https://status.buttondown.com/incident/533622 Tue, 25 Mar 2025 18:51:00 -0000 https://status.buttondown.com/incident/533622#f0a2e8f3b366b9f490abeb29a18b135ec590afac850eefdd52d4fba558728f51 We've confirmed and resent the majority of our affected emails, and are in touch with authors with whom we haven't automatically resent on their behalf. https://status.buttondown.com/incident/533622 Tue, 25 Mar 2025 02:55:00 -0000 https://status.buttondown.com/incident/533622#c1a9b5d10aa84fd801d1c4fa194eb80916c30ebfbd1f63e3df20fd9dcfc8da5a Postmark has reported that [this incident](https://status.postmarkapp.com/notices/bt3ky3r8zlaapqlo-increased-gmail-spam-reports) is now resolved. Our team is continuing to test and monitor. https://status.buttondown.com/incident/533622 Mon, 24 Mar 2025 23:23:00 -0000 https://status.buttondown.com/incident/533622#668b308b22ba35b08a2942c55ac65d1b11b6c880266c1987d9eca2c43ae9c0c3 Postmark has implemented a fix for [the deliverability issue](https://status.postmarkapp.com/notices/bt3ky3r8zlaapqlo-increased-gmail-spam-reports) that caused their IPs to be flagged by Gmail and other providers. We're actively monitoring and will post an update once this is resolved. If you're sending from a custom domain, continue waiting to send large emails until this is fully resolved. https://status.buttondown.com/incident/533622 Mon, 24 Mar 2025 15:32:00 -0000 https://status.buttondown.com/incident/533622#1a14e0101392e76371081483300cafbf8b1fd6a2fb6807803e6a804e07de07b1 We're tracking [Postmark's incident](https://status.postmarkapp.com/notices/bt3ky3r8zlaapqlo-increased-gmail-spam-reports) that is causing a lot of their IPs to be flagged by Gmail and other providers. If you're sending from a custom domain, this likely impacts you; refrain from sending large emails unless you absolutely need to. https://status.buttondown.com/incident/528333 Fri, 14 Mar 2025 18:31:00 -0000 https://status.buttondown.com/incident/528333#44aadb7b6deca2ce55d74475f840c68d53981a725d51944d2a2f469b8dc5e9cb Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera. We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems _did_ catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution quickly we didn't backtest the performance of this solution — in particular, it added a very serious lag to particularly long emails, causing some of them to time out when you went to go and send them (either via the UI or the API.) Normally, when something like this happens we just roll back to stem the bleeding; because this was also a bit of a security/fraud issue, we opted to roll forward and try and fix it live. We are all set now, but apologies if you had issues sending over the last 24 hours; we take that critical path seriously, and hope you understand (and forgive) the interruption. https://status.buttondown.com/incident/528333 Fri, 14 Mar 2025 18:31:00 -0000 https://status.buttondown.com/incident/528333#44aadb7b6deca2ce55d74475f840c68d53981a725d51944d2a2f469b8dc5e9cb Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera. We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems _did_ catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution quickly we didn't backtest the performance of this solution — in particular, it added a very serious lag to particularly long emails, causing some of them to time out when you went to go and send them (either via the UI or the API.) Normally, when something like this happens we just roll back to stem the bleeding; because this was also a bit of a security/fraud issue, we opted to roll forward and try and fix it live. We are all set now, but apologies if you had issues sending over the last 24 hours; we take that critical path seriously, and hope you understand (and forgive) the interruption. https://status.buttondown.com/incident/504518 Fri, 31 Jan 2025 20:52:00 -0000 https://status.buttondown.com/incident/504518#11206448c278e2dfdd6e83e111e726658aa27676e44f27a81af78c7e566804c1 # Post mortem ## What broke? `workerscheduler`, our process for running asynchronous jobs that are scheduled for some date in the future, was hard down for ~six hours. This meant, amongst other things: 1. Outbound emails were down 2. Cron was down 3. Other stuff, but those two dwarf everything else ## Why did it break? At a very high level, our asynchronous worker schedulers work something like this (none of this is bespoke, it's standard RQ): 1. To enqueue a job, serialize the method name, the arguments you want to pass to that method, and a timestamp — store that in Redis. 2. To find a job that should be worked on, pull all of the potential jobs. Sort those jobs by timestamp and start running the first one that is ready. Now, that _the arguments you want to pass_ thing is a bit of a landmine. Consider the following job that we might enqueue: ```python class Email: id: str subject: str body: str @job('five_minutes') def send_email(email: Email, recipients: list[str]): for recipient in recipients: send_email_to_recipient(email, recipient) ``` Enqueuing this means serializing a Python object that looks like: ``` { "method_name": "path.to.module.send_email", "arguments": [ { "class": "path.to.module.email", "id": "1", "subject": "Hi there!", "body": "How are you doing?" }, [ "penelope@buttondown.com", "telemachus@buttondown.com" ] ] } ``` (This is a simplified example, but it's directionally accurate.) Now, the tricky thing about this is that emails can get... large. Our `Email` object stores over sixty columns (four of which are some variation of "the fully rendered body for an email in different formats"), and some authors write _extremely length emails_. Some emails, in memory, are >5MB! Now, with the above toy example — we're sending a single email to a list of two recipients. In reality, Buttondown _batches_ the total recipient list for an email based on a number of factors: an author with 30,000 subscribers might have their email batched into groups of 100 subscribers each (for a total of 300 batches.) And this is where we get into tricky territory. 300 jobs, all containing a 5MB email... now we've suddenly added 1.5GB of memory to Redis, and also now we force the workerscheduler to deserialize 1.5GB of data just to figure out a single job to run. This is, in fact, exactly what happened: the Heroku dyno running the workerscheduler has a memory cap of 512MB, and once we got into this state it kept on trying to read the entire list of jobs, OOM-ing, restarting, ad infinitum. ## Okay, so don't store the entire thing in memory! Right! This is not rocket science. The correct approach is to do something like this: ```python class Email: id: str subject: str body: str @job('five_minutes') def send_email(email_id: str, recipients: list[str]): email = fetch_email_from_db(email_id) for recipient in recipients: send_email_to_recipient(email, recipient) ``` Which then means, in exchange for a slight performance hit (because now we're hitting the database to hydrate the email), all we have to serialize is this: ``` { "method_name": "path.to.module.send_email", "arguments": [ "1", [ "penelope@buttondown.com", "telemachus@buttondown.com" ] ] } ``` We do this almost everywhere... except one place — rate limiting logic per-domain. We happened to trigger _that one place_, and then we were stuck. ## How did we stop the bleeding? By clearing out all of the problematically large jobs from the safety (and large memort size) of my laptop: ```python import django_rq from rq.job import Job STRING_OF_JOB_TO_REMOVE = "send_email_to" QUEUE_NAME = "five_minutes" queue = django_rq.get_queue(QUEUE_NAME) jids = queue.scheduled_job_registry.get_job_ids() jobs = Job.fetch_many(jids, connection=django_rq.get_connection(QUEUE_NAME)) for job in jobs: print(job.description) if STRING_OF_JOB_TO_REMOVE in job.description: queue.scheduled_job_registry.remove(jid) print("Removing!") ``` Once we did that (and re-ran a bunch of stuff to get things flowing again), we were back to normal (albeit with a big backlog!) ## Why did it take so long to notice/fix? The shortest answer is: almost all of our observability runs on crons. If crons are broken, then we're flying blind. But that leads us to... ## How do we make sure this never happens again? - At an object level, we fixed that one problematic code path. We now load emails by ID there instead of serializing the entire object. - At a meta level, we've added a lot of observability through Better Stack so we're not dependent on our own rails. Most notably, we now get paged if no crons have been executed in five minutes: this would have immediately caught the problem. - At an observability level, I've added some internal tooling around analyzing the backlog. It didn't take us too long to diagnose the issue (~thirty minutes), but that's still not great. https://status.buttondown.com/incident/504518 Thu, 30 Jan 2025 16:21:00 -0000 https://status.buttondown.com/incident/504518#aa34ca823374a3f659f84a3b1fc518acc0f467308a34509469baedcda6009371 We've finished redriving the stuck backlog items! We'll follow up with a postmortem later today. https://status.buttondown.com/incident/504518 Thu, 30 Jan 2025 15:14:00 -0000 https://status.buttondown.com/incident/504518#0ab48241fbb0aee39aac6658bc52e965070d7443c96f8ae68c965bcd6296e8bc We have identified the cause and are working on a fix. In the meantime, background work that was stuck has started to be processed again (for example, emails stuck in "About to Send" are now sending). **Importantly, do not send your email a second time. Emails that were sent previously and are waiting will be sent out eventually.** https://status.buttondown.com/incident/504518 Thu, 30 Jan 2025 09:22:00 -0000 https://status.buttondown.com/incident/504518#94e69cf57038bdf7c85dedc45debf01b6e07748d268783def1fba7e7c11d3415 Degradations to our background processing system are causing email sends, scheduled emails, and other background work to be delayed. https://status.buttondown.com/incident/502703 Mon, 27 Jan 2025 18:52:00 -0000 https://status.buttondown.com/incident/502703#7aee67838f20605854cc51e04d1007d0504aa9a6f5eec4f7f2a40847473cfd1b We have identified the root cause of this issue, and a fix has been deployed. https://status.buttondown.com/incident/502703 Mon, 27 Jan 2025 18:30:00 -0000 https://status.buttondown.com/incident/502703#e5e59d38c0d84a4a75f102281b623695873bfc42df78e6136b09f0591daf3293 We are currently investigating reports that some users are unable to send drafts or newsletters https://status.buttondown.com/incident/500977 Fri, 24 Jan 2025 01:15:00 -0000 https://status.buttondown.com/incident/500977#f070dac04ab80b40069a5ae02b4cfbe3dfb1197c08644d94f96772de829ffaff This issue has been identified, and the fix has been shipped. https://status.buttondown.com/incident/500977 Thu, 23 Jan 2025 22:30:00 -0000 https://status.buttondown.com/incident/500977#87c9274e9548356fc1e92d92f0c65dcf97a805bd4b5c74395605de58adeb73b6 We are currently investigating reports that the author-facing dashboard is failing to load for some customers https://status.buttondown.com/incident/497693 Sat, 18 Jan 2025 02:26:00 -0000 https://status.buttondown.com/incident/497693#471a6fa81847d6fe06be8632c985720da105b768175f840304dcbe71fa3e64d8 As written above, we've since recovered from the incident. https://status.buttondown.com/incident/497693 Sat, 18 Jan 2025 02:21:00 -0000 https://status.buttondown.com/incident/497693#3dfd39ee6bd5379934661b7548dd367d25583889ab8e2e009fba26b7181c0e27 From approximately 12:47pm EST to 2:15pm EST, most API requests coming from the author-facing app failed with a 400 error. This occurred because we rolled out a change to our routing and formatting of those requests; amongst other benign changes, we inadvertently changed the formatting of the HTTP method (GET, POST, etc.) of outgoing requests from uppercase to lowercase (get, post, etc.). This change was tested and passed both our continuous integration and manual testing; however, Heroku (our cloud infrastructure provider) does _not_ support this formatting for HTTP methods, and rejected all inbound requests. Once we discovered the issue, we rolled back the deployment and rolled forward a change to correctly capitalize those methods. https://status.buttondown.com/incident/461653 Thu, 14 Nov 2024 20:15:00 -0000 https://status.buttondown.com/incident/461653#48140a0b276496d4fcc9e25332accdb36dc7192c102279116ffa108375f8ade6 We're tracking reports of broken links for folks who have click tracking enabled. We've temporarily switched off click tracking for affected newsletters and are waiting to hear more from our upstream vendor.